Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
SeaWolf-AI 
posted an update 3 days ago
Post
2994
POCKET now speaks Gemma 4 — a 26B model that loads in every app, and runs on your PC with no GPU

We're adding a Gemma-4 sibling to POCKET: POCKET-26B, built from Google's Gemma-4-26B-A4B (Apache-2.0). Our flagship POCKET-35B is a Qwen-family MoE and needs a recent llama.cpp; POCKET-26B trades a little size for the thing people kept asking for — it just loads, everywhere, today: Ollama, LM Studio, PocketPal, MLX, any stock llama.cpp. No fork, no bleeding-edge runtime, no CUDA, no cloud.

It's a sparse Mixture-of-Experts (25.2B total, ~4B active per token), so the work per token stays small — a real 26B that generates on a CPU with no graphics card.

Two things make it stand out:

1) Universal compatibility. Gemma 4 is a standard, widely-supported architecture, so POCKET-26B runs on the tools you already have — no waiting for your app to add a new model type.

2) Quality that survives compression. Measured GPQA-Diamond (198 q, greedy):
• Full base: 67.7%
• POCKET-26B Q4_K_M (17 GB): 67.7% — lossless
• POCKET-26B Q2_K (11 GB): 67.2% — near-lossless, at 11 GB

Live, on a CPU-only box (our demo Space — POCKET-26B vs Bonsai-27B, same machine, same stock llama.cpp): POCKET-26B ≈ 19 tok/s vs Bonsai ≈ 6 tok/s → about 3× faster generation, no GPU. (Honest notes: shared CPU box, sequential race; a dedicated machine is faster.)

Where it fits in the family:
• POCKET-35B (Qwen MoE) — bigger, top-tier, needs a recent llama.cpp.
• POCKET-26B (Gemma 4) — loads in any app, quality-robust when compressed. The demo runs the Q4_K_M build; Q2_K (11 GB) is the smallest footprint. For a true ≤8 GB phone, the 5 GB POCKET-KR (Qwen) is still the pick.

Try it and grab it:
🖥️ Live demo (Gemma4-based, answering on a CPU, no GPU): FINAL-Bench/POCKET-26B-CPU
📦 POCKET-26B-GGUF (Q4_K_M 17 GB · Q2_K 11 GB): FINAL-Bench/POCKET-26B-GGUF
📚 POCKET collection: https://huggingface.co/collections/FINAL-Bench/pocket-models
dipankarsarkar

Q2_K losing only 0.5 points on GPQA (67.7 to 67.2) is the surprising number here, not the 3x speed.

Sparse MoE experts are smaller matrices than a dense FFN of the same total size, so there's less redundancy per expert to absorb quantization error. In theory that should make MoE quantize worse than dense at the same bit width, not survive it near-losslessly.

Is that near-lossless result specific to Gemma 4's expert size, or does POCKET-35B's Qwen MoE show the same pattern at Q2_K?

·
SeaWolf-AI

Good instinct — and you're right that a single small expert has less internal redundancy than a big dense FFN. What rescues it isn't per-expert redundancy though; it's the system around the experts.

Why Q2_K holds up:

Only a small slice is active per token, and the router + shared/attention pathways carry much of the signal — so error in any one expert gets diluted.
There's a lot of cross-expert redundancy (experts learn overlapping functions), so the ensemble stays robust even when each expert individually isn't.
The Q2_K here is importance-calibrated, not naive round-to-nearest — standard llama.cpp practice, but it matters a lot at 2-bit.
On your real question — Gemma-4-specific? Partly yes, and your intuition shows up at the extreme. Gemma-4's experts are fewer and larger, so they take 2-bit gracefully. Qwen-family MoEs with many tiny experts (our POCKET-35B) get more fragile as you push toward the ~1-bit end, and it bites unevenly across tasks — non-English first. So "does an MoE survive?" really is a function of expert geometry, not a universal property.

One honest caveat: near-lossless on GPQA ≠ near-lossless on everything — reasoning survived at Q2 here, but some axes are more bit-sensitive. Great question; this is exactly the knob that sets how far you can compress. 🙌
dipankarsarkar

Non-English first is the detail that matters more than the average GPQA number. That's a failure mode, not a scalar loss, exactly the axis one aggregate score hides.

Do you have per-language numbers for POCKET-35B at Q2_K, or is "bites non-English first" a qualitative pattern from testers so far? If it's measured, which languages hold up longest before the expert-count/size tradeoff shows?
In this post