Local LLM Hardware VRAM & Quantization Calculator
Size exact model weights, GQA KV-cache overhead, and estimated decode throughput (tok/s) across Apple Unified Memory, NVIDIA RTX 4090/5090, H100, and Exo distributed clusters before provisioning hardware.
1. Configure Model & Context
FITS_NATIVELY (20.1 tok/s)
Download a starter CLAUDE.md ($0), grab the $4.99 Drop-In Config Pack, or unlock the 9-file Local + LiteLLM fallback router .ZIP ($9.99).
Live Local Inference & Agent Runtime Telemetry (GitHub Baseline)
Includes live Machined Obsidian & Emerald/Amber SVG Shields
Significant-Gravitas/AutoGPT
187,620
Autonomous multi-step AI agent orchestration runtime with local & hybrid model execution.
ollama/ollama
183,243
Get up and running with OpenAI gpt-oss, DeepSeek-R1, Gemma 3 and other models.
ggml-org/llama.cpp
131,250
LLM inference in C/C++ with zero dependencies across Metal, CUDA, and Vulkan.
vllm-project/vllm
93,851
A high-throughput and memory-efficient inference and serving engine for LLMs.
modelcontextprotocol/servers
91,329
Official Model Context Protocol (MCP) reference servers for local filesystem, Git, and DB tool execution.
BerriAI/litellm
61,434
Unified LLM proxy and cost-routing gateway across 100+ local (Ollama/vLLM) and frontier APIs.
exo-explore/exo
47,901
Run frontier AI locally with distributed Thunderbolt RDMA across consumer devices.
