Tiny-vLLM: High-Performance C++ LLM Inference Engine
Show HN feature reveals Tiny-vLLM, a lightweight C++ and CUDA inference engine designed to outperform Python-based alter…
🏷️ LLM inference
12 articles about 'LLM inference'
Optimizing Local LLM Inference on Mac: A Framework Benchmark
Developers evaluate local LLM frameworks for Mac, prioritizing inference speed and memory efficiency over raw model size…
Cloudflare Builds LLM Infrastructure on Its Edge Network
Cloudflare unveils disaggregated prefill architecture and custom Infire inference engine to run large language models ef…
Arm Unveils NPU Architecture for On-Device LLM
Arm Holdings announces a new neural processing unit architecture optimized for running large language models directly on…
MIT Sparse Attention Cuts LLM Inference Costs by 60%
MIT researchers unveil a new sparse attention mechanism that dramatically reduces LLM inference costs while preserving m…
Google Gemma 4 Uses Speculative Decoding for 3x Speed
Google's new Gemma 4 open-weight models leverage speculative decoding to deliver up to 3x faster inference with no quali…
What Can You Actually Do With an RTX 5060 Ti 16GB?
A practical breakdown of AI workloads the RTX 5060 Ti 16GB can handle, from local LLMs to voice recognition and agent fr…
Optimize LLM Inference With vLLM and TensorRT-LLM
A practical guide to dramatically boosting LLM inference speed using vLLM and NVIDIA TensorRT-LLM frameworks.
MoE Architecture Cuts LLM Inference Costs by Up to 60%
A new study reveals Mixture-of-Experts models activate only a fraction of parameters during inference, slashing compute …
GLM5.1 Cloud Deployment: Ollama vs China Telecom
Developers debate the best cloud platforms for running Zhipu AI's GLM5.1, raising questions about reliability, speed, an…
LLM Inference Hardware Crisis Is Worse Than You Think
A new paper from Google DeepMind and Turing Award winner David Patterson reveals the staggering hardware costs of LLM in…
Batch or Split? How to Send Multiple Questions to LLMs Faster
When you have multiple unrelated questions for an LLM, splitting them into parallel requests almost always beats batchin…
1