offline-llm-engineering
/Case StudyOffline LLM inference pipeline benchmarking Llama 3.2 3B, Phi-4 Mini, and Mistral 7B on a consumer Intel Mac — zero cloud dependency. Covers CLI streaming, automated benchmarks, Pydantic structured outputs, and a 180-run model comparison study.
- Category
- ML Research
- Type
- Local LLM inference & benchmarking
- Core stack
- Python · Ollama · Pydantic
- Status
- Code on GitHub
300+ inference calls. Real numbers. No cherry-picking.
/What this is
Everything runs offline. No API keys. No data leaves the machine.
- CLI tool: stream responses from any of three models, measure TTFT and latency in real time.
- Inference benchmarking: automated 60-run benchmark across models and temperatures, exported to CSV.
- Structured outputs: JSON schema enforcement with Pydantic validation and retry logic.
- Model comparison study: 180 runs across 30 standardized prompts in 6 categories with quality analysis.
/Why local inference matters
- Privacy: HIPAA, GDPR, and SOC 2 often prohibit sending sensitive data to third-party APIs.
- Latency: cloud round-trips add 200–600ms before the first token; local inference removes the network.
- Cost at scale: per-token billing vs fixed GPU capital; breakeven often at 3–6 months at high volume.
- Edge deployment: factory, medical, and field systems need self-contained on-device models.
/Phase 5 — Full comparison study (180 runs)
30 prompts × 3 models × 2 temperatures · Intel MacBook Pro · 6-Core i7 · CPU-only
| Model | Temp | Avg TTFT | Avg latency | Words/sec | Verdict |
|---|---|---|---|---|---|
| Llama 3.2 3B | 0.0 | 11.76s | 38.23s | 1.3 | Best for CPU |
| Llama 3.2 3B | 0.7 | 12.45s | 38.42s | 1.1 | Consistent |
| Phi-4 Mini | 0.0 | 12.65s | 52.07s | 1.2 | Good at T=0 |
| Phi-4 Mini | 0.7 | 24.83s | 120.70s | 0.7 | Degrades at T=0.7 |
| Mistral 7B | 0.0 | 23.79s | 71.69s | 0.6 | Slow, needs GPU |
| Mistral 7B | 0.7 | 34.26s | 106.10s | 0.4 | Unusable on CPU |
/Cold vs warm inference (Llama 3.2 3B)
| State | TTFT | Total latency |
|---|---|---|
| Cold (loading from disk) | 8.66s | 12.37s |
| Warm (resident in RAM) | 0.41s | 4.08s |
| Speedup | 21× | 3× |
/Key findings
05 pointsLlama 3.2 3B is the only model suitable for interactive CPU deployment — ~38s average latency, best instruction compliance.
Phi-4 Mini has a temperature cliff
At T=0.7 latency triples and training data leakage can appear. Use T=0.0 with conciseness instructions.
Mistral 7B needs a GPU
0.4 words/sec at T=0.7 on CPU due to memory bandwidth limits.
Cold start costs 21× more than warm inference
Production systems must pre-warm models on startup.
Quality and speed are orthogonal
Speed benchmarks alone cannot substitute for quality evaluation.
/Models
| Model | Parameters | Creator | Disk | Notes |
|---|---|---|---|---|
| llama3.2:3b | 3B | Meta | 2.0 GB | Best overall for CPU |
| phi4-mini | 3.8B | Microsoft | 2.5 GB | Use T=0.0 only |
| mistral:7b | 7B | Mistral AI | 4.4 GB | Needs GPU for interactive use |
/Concepts demonstrated
| Concept | Where |
|---|---|
| Local vs cloud inference tradeoffs | Architecture, Phase 1 |
| GGUF quantization | Model setup, technical report |
| Time to first token (TTFT) | CLI, Phases 2–3 |
| Cold start vs warm inference | Phase 2, 21× speedup |
| Pydantic schema validation | Phase 4 |
| Training data leakage | Phase 5 Phi-4 finding |
| Prompt injection defense | comparison_study.py CONCISE_SUFFIX |