Tiny LLM Smartphone: The Future of On-Device AI
PrismML believes powerful reasoning models don’t need to be massive. The Caltech-founded AI lab is developing tiny LLM smartphone technology that could change how we all use AI — by running advanced models directly on your PC or phone instead of in the cloud.
On Thursday, PrismML released Bonsai 2 27B, its latest compressed model. It shrinks Qwen3.8 27B, a widely used open source model from Alibaba, down to just 5.9 GB. That’s small enough to fit on a PC and possibly a high-end smartphone — a 9x to 10x reduction in memory versus the original.
Who Is Behind PrismML?
PrismML was founded by a group of Caltech researchers and is led by CEO Babak Hassibi, a Caltech professor and expert in compression technologies. The startup also counts Ion Stoica as an adviser. Stoica is a co-founder of Databricks and director of Berkeley’s Sky Computing Lab, which has birthed technologies and startups from Letta to SGLang.
The company is backed by investors Khosla Ventures, Cerberus Capital, and Caltech. It has raised a $22.25 million seed round.
What Makes Bonsai 2 Different?
PrismML is not the only company working on LLM compression. Multiverse Computing, founded by a professor from Spain’s Donostia International Physics Center, is another player — and has raised significant funding.
But Hassibi says PrismML’s compression tech is unique because its models lose virtually no performance compared with the originals. Bonsai 2 matches 98% of Qwen’s aggregate benchmark scores, up from the first Bonsai released in March, which matched 95%.
That original model has already been downloaded over 11 million times, and PrismML’s even smaller models have been downloaded another 2.6 million times, according to the company.
How Does the Compression Work?
PrismML achieves this by shrinking the “weights” that make up a model — the information a model learns and stores during training. Normally, each weight requires 16 bits. PrismML’s approach, called “ternary” weights, simplifies that down to three values: +1, −1, or 0.
With far smaller values to store for each weight, the model takes up dramatically less space. For a deeper dive on the compression technique, the project’s Hugging Face page offers more detail.
Can Tiny LLMs Match Full-Size Models?
Whether PrismML could ever reach 100% benchmark performance parity remains to be seen. Compression will likely always have some impact, Hassibi says.
Still, perfect benchmark parity is fairly academic. LLMs are not so accurate in their uncompressed form — and benchmarks not so perfectly reflective of actual tasks — that a 2% degradation would likely meaningfully affect how a model performs in real use. The surrounding software, or the harness a model runs inside of, also matters a lot when it comes to accuracy.
What’s Next for PrismML?
The startup’s next goal is to apply this compression technique to even bigger models. “The next models that we will release, hopefully in the next couple of months, will be in the several-hundred-billion-parameter range, and I expect it will be easier to retain the intelligence there,” Hassibi told TechCrunch.
As model size grows, he added, “There is more room to be able to compress them without losing the intelligence. So I would just say, as a general trend, for larger models, it’s easier to get to 100%.”
Why On-Device AI Matters
Stoica says he’s excited for this tech because it makes it possible for advanced models to run on users’ devices. “You are going to have intelligence at your fingertips, and it’s going to be free because it’s going to run on the device you already bought,” he said. “It’s also going to be private, because you’re not going to send it to the cloud.”
That combination of cost savings and privacy could be a major shift for how people interact with AI daily. Instead of relying on cloud servers, a tiny LLM smartphone could handle reasoning tasks locally — no internet connection required.
PrismML’s progress from 95% to 98% benchmark parity in just a few months shows rapid improvement. If the company can maintain that trajectory with larger models, on-device AI could become far more capable than many expect.
For now, Bonsai 2 27B represents a meaningful step toward putting genuinely useful AI in your pocket — without the cloud, without the cost, and without sacrificing privacy.
