Quantization Powers On-Device AI Transformers
New quantization techniques enable large transformer models to run efficiently on mobile devices, reducing latency and e…
🏷️ quantization
6 articles about 'quantization'
Compress LLMs with FP8, GPTQ & SmoothQuant
New tutorial demonstrates compressing instruction-tuned LLMs using llmcompressor. Compare FP8, GPTQ, and SmoothQuant for…
Apple Publishes Sub-1-Bit LLM Compression Research
Apple's ML team reveals techniques to compress large language models below 1-bit precision, enabling powerful AI on iPho…
Cut LLM Inference Costs With Quantization & Distillation
A practical guide to reducing LLM inference costs by up to 80% using quantization and distillation techniques without sa…
Microsoft Research Unveils BitNet b2 for LLMs
Microsoft Research introduces BitNet b2, pushing extreme quantization to slash LLM memory and compute costs while preser…
The Math Behind Local LLM VRAM Requirements
A practical guide to calculating exact GPU memory needs before deploying large language models locally.
1