Gemma 4: Google's Truly Open Model That Runs on a Phone

A genuinely open model, and a tiny one

Last week Google did something most large tech companies have avoided: it released a large language model that is truly free and open source under the Apache 2.0 license. Not "open-ish," not research-only, not "please don't make money with it." Actually free. That model is Gemma 4.

The easy assumption is that a free model comes with a catch, usually that you need a small data center to run it. Gemma 4's twist is the opposite: it is suspiciously small. The large model fits on a consumer GPU, and the edge model is small enough to run on a phone or a Raspberry Pi, while reaching intelligence levels normally associated with models that need data-center GPUs. On paper, that should not be possible.

Why the license matters

Plenty of labs ship "open-weight" models, but the terms vary a lot:

• Meta's Llama models are quasi-open, under a custom license that keeps leverage over anyone who builds a real business on them.

• OpenAI's GPT-OSS models are Apache 2.0, but larger and weaker than Gemma.

• Beyond that, the open scene leans on Mistral and Chinese models like Qwen, GLM, Kimi, and DeepSeek.

Gemma 4 stands out by being American-made, Apache 2.0 licensed, genuinely capable, and, above all, tiny.

The size claim, in numbers

The 31-billion-parameter version of Gemma 4 scores in the same ballpark as models like Kimi K2.5 thinking. The practical difference is enormous:

• Gemma 4: about a 20 GB download, roughly 10 tokens per second on a single RTX 4090.

• Kimi K2.5: a 600 GB-plus download, at least 256 GB of RAM, aggressive quantization, and multiple H100s just to get it running.

Kimi is still the stronger model, but running it locally is out of reach for almost everyone. Gemma 4 you can actually run on your own machine.

How Google shrank it

The key insight is that Google did not just shrink the model, it went after the real bottleneck in local inference: memory. Running a large model locally is usually not limited by your CPU, it is limited by memory bandwidth. Every token the model generates requires reading through a large amount of weights in VRAM (the memory on your GPU). What matters is less how big the model is and more how expensive it is to read.

TurboQuant: smarter compression

Alongside Gemma 4, Google quietly published a research note on TurboQuant, a new approach to quantization (the process of compressing model weights so they take up less space). Normally quantization is a straight trade-off: a smaller model, but worse quality. TurboQuant improves that trade-off in two steps:

• It converts data from a standard Cartesian (x, y, z) coordinate system into polar coordinates (a radius and an angle). Because the angles follow a predictable pattern, the model can skip the usual normalization steps and store information more efficiently.

• It then applies the Johnson-Lindenstrauss transform to compress high-dimensional data down to single sign bits (+1 or -1) while preserving the distances between data points.

The underlying math is genuinely dense, but the outcome is a much better size-versus-quality trade than plain quantization.

The real trick: per-layer embeddings

TurboQuant is impressive, but it is not actually the secret behind Gemma 4's smallest models. You will notice some Gemma models carry an "E" in the name, like E2B and E4B. That E stands for effective parameters, and it comes from a technique called per-layer embeddings.

In a normal transformer, each token gets one embedding at the start, and the model carries that information through every layer, even though most of it is not needed at any given layer. Per-layer embeddings give each layer its own small, custom version of the token, so information is introduced exactly where it is useful instead of all at once. The result is a model that is small, smart, and efficient.

What it is like to use

Running Gemma 4 locally with Ollama on an RTX 4090, it comes across as a solid all-around model, and a strong base for fine-tuning on your own data with tools like Unsloth. It is not yet good enough to replace high-end coding assistants, but as a free, local, general-purpose model it is remarkable.

How we use Gemma at Birdhouse

This is not a spectator sport for us. As an AI-native agency, Birdhouse already runs Gemma models in our own workflows. Because Gemma 4 is genuinely open and small enough to run on hardware we control, it lets us do three things that a closed frontier API makes hard:

• Keep client data private. Sensitive data can stay on our own infrastructure instead of being sent to a third-party API.

• Fine-tune for a domain. We adapt Gemma to a client's specific data and voice with tools like Unsloth, rather than prompting a general model and hoping.

• Control cost at scale. For high-volume, well-scoped tasks, a small local model is dramatically cheaper than paying per token for a frontier model.

For work that genuinely needs the frontier, we still reach for the largest models. But a capable open model like Gemma now covers a surprising amount of our day-to-day work, and Gemma 4 pushes that line further than before.

Why this matters

Gemma 4 is a signal about where AI is heading. Capability is no longer only a function of raw scale and giant GPU clusters. Efficiency, especially around memory, matters just as much. A genuinely open, genuinely small model that runs on consumer hardware means more privacy, lower cost, local inference, and easy fine-tuning. For teams building with AI, that is a meaningful expansion of what is possible without a data center, and it is exactly the kind of shift we build our practice around.