RVA CYBER Field note · Local AI

Model review

Nanbeige, Bonsai, and Gemma

Three local models. Three different bets.

RVA Cyber Research July 28, 2026 Updated assessment

Verdict: Keep Gemma in production. Nanbeige remains the compact-agent watchlist candidate. Ternary Bonsai 27B is the stronger reason to schedule an isolated bakeoff—but a 7.17 GB checkpoint with a custom low-bit runtime is not a drop-in replacement for a proven Gemma worker.

The decision

Keep Gemma. Put Bonsai next in line.

Nanbeige4.2-3B reports exceptional agentic results for a compact model. Ternary Bonsai 27B makes a different and potentially more important claim: preserve most of a 27B model’s measured capability while compressing its text weights into a 7.17 GB deployed checkpoint.

Neither release answers our production question yet. Our local model summarizes bounded source packets, classifies material, drafts text, and explains source-contained code. It cannot browse, use tools, alter records, send messages, or make final operational decisions. Gemma 3 4B IT QAT already performs that work reliably through a mature local runtime.

Bonsai has earned priority in the next controlled bakeoff. It has not earned a production migration.

What this review is—and is not

This is a production-fit assessment, not a direct local benchmark of Nanbeige or Ternary Bonsai. We reviewed both projects’ model cards, technical reports, runtime instructions, licenses, Gemma’s official documentation, and our own operational experience with Gemma. We did not download or execute either candidate. Their published results remain their authors’ results until independent teams reproduce them.

That limit matters. Nanbeige’s paper compares the model with Gemma 4 variants, not the Gemma 3 4B IT QAT model in our local workflow. Bonsai’s paper compares its ternary Qwen3.6-27B derivative with the full-precision base and other 27B- to 31B-class builds. Neither project publishes a controlled comparison against our Gemma checkpoint on our hardware and tasks.

Our local test platform

Our reference host is an x86 Linux workstation with an Intel Core i7-13700 processor (16 cores, 24 threads), 62 GiB of usable system memory, an NVIDIA RTX A2000 with 12 GB of VRAM, and a 1 TB NVMe SSD. It runs Ubuntu 24.04.4 LTS and serves Gemma through Ollama with CUDA acceleration.

The 4.0 GB gemma3:4b-it-qat package runs fully on the GPU, using about 4.6 GiB of VRAM. A post-load smoke test measured generation at approximately 66.8 tokens per second. That figure describes one short internal test on this host—not end-to-end application latency or a controlled comparison with Nanbeige or Bonsai.

The side-by-side reveals three different bets

Scroll table horizontally

Decision factor Nanbeige4.2-3B Ternary Bonsai 27B Gemma 3 4B IT QAT
Design goal Compact general agent for tool use, office work, coding, reasoning, and research Preserve 27B-class reasoning and tool behavior in deeply compressed ternary weights General local model for text generation, image understanding, summarization, question answering, and reasoning
Model class 4B total parameters; 3B outside embeddings About 27.3B language parameters plus an optional vision tower 4B parameters
Published resident weights Varies by runtime and quantization; no configuration has been tested by RVA Cyber 7.17 GB for the shipped ternary text checkpoint; 5.9 GB is the representation’s ideal size 2.6 GB for Google’s int4 QAT weights, excluding runtime cache; 4.0 GB in the Ollama package we use
Maximum context 256K tokens 262K tokens; practical memory depends on KV-cache settings 128K tokens
Input modes Text Text; optional separate vision projector Text and image
Published agent evidence Extensive and unusually strong for its size; some results depend on author-built scaffolds Authors report 74.01 across two tool-use benchmarks versus 80.00 for the full-precision base Not positioned or evaluated as a purpose-built tool agent
License Apache 2.0 Apache 2.0 Gemma Terms, including redistribution conditions and use restrictions
Production runtime GGUF and local paths exist, but the model’s own instructions still rely on custom code or Nanbeige branches for several runtimes The optimized 7.17 GB path uses PrismML’s custom low-bit kernels and llama.cpp fork Official quantization-aware-trained checkpoint with native Ollama support
Our production evidence Not yet tested in our environment Not yet tested in our environment Passed our bounded four-task bakeoff and remains the current local worker

The table compares published model attributes and deployment fit. It does not convert unlike parameter classes, hardware, or benchmark suites into a synthetic score.

Nanbeige bets on agent design

Nanbeige4.2-3B uses a Looped Transformer that passes tokens through the same layer stack twice. The authors report that this adds effective capacity without adding parameters. Their selected two-pass design retains about 75% of a standard Transformer’s token efficiency in their experiments. The released model has 4 billion total parameters, 3 billion of them outside the embedding layers.

The reported results are striking. The model card places Nanbeige ahead of larger Qwen3.5 and Gemma 4 variants across many general-agent, code-agent, and reasoning benchmarks. It reports 63.6 on SWE-bench Verified, 65.0 on Claw-Gym, and 74.7 on Pinch-Bench-V2. In a separate OpenClaw evaluation, Nanbeige beats the two Qwen models on all 6 listed daily-task, office-task, and deep-research benchmarks.

Those results justify attention. They do not justify a direct victory over our Gemma deployment.

Fact

The comparisons are not model-for-model

The main table compares Nanbeige with Gemma 4-E4B and Gemma 4 12B. The OpenClaw table does not include Gemma. We use Gemma 3 4B IT QAT.

Fact

The evaluation favors agentic behavior

All reported evaluations use thinking mode with prior reasoning preserved. Several office tasks use the authors’ in-house scaffold. Code benchmarks use named external scaffolds.

Fact

The training boundary is visible

The paper says agentic reinforcement learning was restricted to relatively easy tasks with shorter trajectories and higher pass@8 scores because those tasks trained more stably at this scale.

Interpretation

The model may be excellent without being our replacement

Nanbeige’s evidence is strongest where a model plans, calls tools, and works across long trajectories. We deliberately deny those powers to our local offload model.

The correct response to promising vendor-authored evidence is neither dismissal nor adoption. It is a harder test.

Bonsai bets on representation

Ternary Bonsai 27B starts with a much larger model: Qwen3.6-27B. Prism ML says it preserves the architecture while converting the language model’s embeddings, attention projections, MLP projections, and output head to ternary weights. Each low-bit weight takes one of three values: −1, 0, or +1, with a shared FP16 scale for each group of 128 weights.

The representation’s ideal cost is 1.71 bits per weight, or about 5.9 GB. That is not the file users deploy today. Current kernels store each ternary value in a 2-bit slot, bringing the published text checkpoint to 7.17 GB. The distinction is disclosed in the model card and whitepaper; evaluations and capacity planning should use the deployed figure.

The authors report an 80.49 average across 15 thinking-mode benchmarks for the ternary model, compared with 85.07 for the 54 GB full-precision base. They describe that as retaining 94.6%—rounded to 95%—of the baseline average. The strongest retained categories are math at 93.40 versus 95.33 and coding at 85.96 versus 88.74. Instruction following falls from 78.47 to 71.77, agentic tool use from 80.00 to 74.01, and vision from 72.61 to 65.19.

That is a technically meaningful claim. It is not a direct comparison with Gemma 3 4B IT QAT, and “95% of intelligence” is not an independent standard. It is a ratio derived from Prism ML’s own aggregate benchmark score, using its chosen suite, infrastructure, and scoring.

Fact

The deployed checkpoint is 7.17 GB

The 5.9 GB headline is the ideal ternary representation. Today’s GGUF file uses 2-bit slots and is roughly 2.8 times Google’s published 2.6 GB int4 weight footprint for Gemma 3 4B.

Fact

Long context still carries a cache bill

Prism ML reports an 8.4 GB peak at 4K context and a 12.8 GB peak at the full 262K window with its 4-bit KV cache. Those are vendor measurements, not our hardware results.

Fact

The speed evidence is platform-specific

The published table reports 26.2 tokens per second on an Apple M5 Pro, 18.0 on an M4 Pro, and 98.0 on an H100. It does not report throughput on an x86 Linux host with an RTX A2000 12GB like ours.

Interpretation

Fit is proven; production fit is not

The model should fit comfortably in our available memory. That does not establish latency, instruction fidelity, output stability, or operational value in our bounded workflow.

Bonsai’s own limitations section says long-horizon, multi-file, run-test-repair coding is not yet a strong target of this release. That candor matters. The release is best understood as a striking compression result with promising retained behavior—not a finished universal local agent.

Our local worker has a narrow job

Local AI earns its place in our architecture by reducing cost and latency without expanding authority. The local model receives bounded, low-risk, source-contained tasks. A more capable cloud model or a human reviews anything operationally significant.

A genealogy workflow illustrates the boundary. The local model can turn a supplied packet of records into a first-pass summary, classify passages by topic, or draft a research note. It cannot browse for relatives, alter the family tree, contact an archive, or turn a weak inference into a final claim. Similar boundaries apply to document triage, draft writing, and source-contained code explanation.

The local model may

  • Summarize a supplied source packet
  • Classify or route bounded material
  • Draft text for review
  • Explain code contained in the prompt

The local model may not

  • Browse, use tools, or call external systems
  • Read durable private memory by default
  • Send messages or change records
  • Make final security, payment, or configuration decisions

That architecture changes the selection criteria. Tool-call accuracy matters less than instruction fidelity, stable formatting, predictable latency, and competent reasoning over the material already in hand. Nanbeige may prove better on those dimensions. Its current publication does not establish that.

Gemma still wins the operational layer

Gemma 3 4B IT QAT is ordinary in the way production systems need to be ordinary. Google publishes an official quantization-aware-trained checkpoint. Ollama supports it natively. The model accepts text and images, offers a 128K context window, and supports more than 140 languages. We can install it, pin it, test it, run it, and remove it without maintaining a vendor fork.

Google reports that its quantization-aware training reduced the perplexity loss from Q4_0 quantization by 54% compared with ordinary post-training quantization in its llama.cpp evaluation. The 4B checkpoint reduces weight memory from 8 GB in BF16 to 2.6 GB in int4, excluding the runtime’s key-value cache. That optimization targets the exact constraint local deployments face.

Nanbeige’s deployment path is improving quickly. Its model card exposes quantizations, and its instructions include llama.cpp and Ollama. Yet the same instructions require remote custom model code for Transformers and Nanbeige branches for SGLang, vLLM, llama.cpp, or the full native Ollama build. The release notes also state that LM Studio’s bundled llama-server does not yet support the architecture without replacing the backend.

Bonsai has a similar boundary for a different reason. The model’s own ternary quickstart clones Prism ML’s llama.cpp fork, which contains the custom Q2_0_g128 hybrid-attention kernels. Hugging Face’s generated one-click examples point to the 53.8 GB F16 variant; they do not demonstrate that the 7.17 GB ternary path is a stock Ollama deployment. The optimized result therefore arrives with a vendor runtime branch that must be reviewed, built, pinned, monitored, and updated.

This is not a criticism of models released days ago. It is a production fact. Every custom branch becomes another component to inspect, patch, regression-test, and roll back. Compression should reduce operational weight, not merely move it into the runtime.

Both candidates hold one clean operational advantage: Apache 2.0 is simpler than the Gemma Terms. Google permits use and redistribution, but its agreement imposes redistribution conditions and incorporates a prohibited-use policy. Organizations that distribute models or derivatives should evaluate that difference with counsel.

A replacement must win our work, not someone else’s leaderboard

Bonsai moves to the front of our candidate queue because its 27B-class starting point creates a plausible path to a large quality gain. Nanbeige remains on the queue because its compact agent results are unusually strong. Neither candidate enters production until it can be tested through a controlled, recoverable runtime.

  1. 01
    Controlled runtime. Prefer a current upstream Ollama or llama.cpp release. If a vendor fork is required for a research bakeoff, pin it, inspect the diff, deny network access during inference, and keep it outside the production path.
  2. 02
    Four real tasks. Test executive summarization, strict routing and classification, source-contained code review, and precise instruction following with identical source packets.
  3. 03
    Operational measurements. Record first-token latency, output speed, peak memory, prompt processing, prompt adherence, schema validity, error rate, and restart behavior.
  4. 04
    Security review. Inspect model code, templates, custom kernels, runtime changes, network behavior, file access, tool formatting, and dependency provenance.
  5. 05
    A decisive result. We are satisfied with Gemma. A marginal quality gain does not justify a new runtime branch or a slower worker. A candidate must win enough real work to pay for the complexity it adds.

Gemma already cleared an earlier four-task bakeoff against our former local model. It produced stronger synthesis and code/security analysis, followed strict instructions, and met our latency bar. That is internal operational evidence, not a universal benchmark. It is also the evidence relevant to this decision.

Bonsai is the better reason to test. Gemma is still the better reason not to rush.

The next decision belongs to the bakeoff. Until a candidate can enter that test safely and win the work we actually assign, changing models would replace a proven system with a promising claim.

Primary sources