Building Mila on Mila

Running Gemma 4 12B in Docker Behind an OpenAI-Compatible API

Sat, 18 Jul 2026 00:00:00 +0000

This week Mila added a feature I think is really cool! Mila now builds and runs the same way on three surfaces — Windows/MSVC (where I develop), Clang on WSL/Linux, and a single Docker container — and from that container you can drive Mila as the “brain” behind any OpenAI- or Anthropic-compatible harness.

One container, one command, a Gemma 4 12B model as a brain

The dev container is a completely known build environment — clone, one step, a running Mila. Beyond the interactive Chat CLI, it now stands up the Mila Inference Server (MIS): a small FastAPI wire adaptor that serves the runtime over an OpenAI / Anthropic / Mila-native HTTP API.

Flash-Decoding for MQA in CUDA: One KV Head, Split-K, and an FP4 Matvec at 90% of DRAM Peak

Thu, 16 Jul 2026 00:00:00 +0000

Part 1 — Hand-Writing Gemma 4 FlashAttention + an FP8 GEMM Bonus — took ten sessions to drag prefill from 1.95x behind llama.cpp to 1.14x. This part took one session to drag decode from 1.30x behind to 1.03x: Gemma 4 12B FP4 on the same RTX 4070, 38.65 → 49.09 tok/s at a 32K context, against llama.cpp’s measured 50.3–50.7.

It didn’t take one session because decode is easier. It took one session because Part 1 paid the tuition — the parity-oracle ladder, the compute-sanitizer discipline, the online-softmax kernel DNA, and above all the reflex of profiling before believing — and this campaign got to spend the education.

Writing a FlashAttention Prefill Kernel for Gemma 4: Head Dim 512, Tensor Cores and an FP8 GEMM

Wed, 15 Jul 2026 00:00:00 +0000

Adding FlashAttention to Mila’s GQA started as a memory project. It was the last big lever in optimizing memory utilization across the Gemma 4 pipeline — with weight tying, FP8 KV, and bounded sliding-window KV already in place, the attention path was the last thing still materializing large intermediates. A classical attention writes a [chunk × attended_len] score matrix to global memory, and its workspace has to be sized to the full context width; at 64K that’s ~1 GB per buffer of pure throwaway. FlashAttention’s online softmax removes it entirely, and that reclaim — ~1-2 GB at long context — is what took Gemma 4 12B from a VRAM cliff at 32K to running 64K on a 12 GB RTX 4070. Speed was the second prize, and it’s the one that turned into a ten-session campaign.

Bringing Up Gemma 4 in C++: Sliding-Window Attention, Dual RoPE and Token-for-Token Parity

Tue, 23 Jun 2026 00:00:00 +0000

Where We Left Off

Alpha.2 ended with Mila producing token-for-token matches against HuggingFace for Llama 3.2 1B, and the alphas pushed that to Llama 3.1 8B with FP8 and FP4 weight quantization. The framework had a working Grouped-Query Attention op, a two-phase prefill/decode KV cache, and a clean OperationTraits dispatch path. The Llama family was, by then, a homogeneous machine: every layer looked like every other layer.

Gemma 4 broke that assumption on contact.

Closing a Local Agentic Loop: Mila Inference Server and Codex CLI

Fri, 15 May 2026 00:00:00 +0000

Date: 2026-05-15 Status: Proven / Experimental Component: Mila/Inference/Server

Overview

This document describes the design and implementation of a protocol translation layer that enables Codex CLI to drive an agentic coding loop against a locally-hosted Llama 3.2 3B Instruct model served by the Mila Inference Server (MIS).

The result is a fully local, self-hosted agentic coding assistant: no OpenAI API key, no cloud dependency, no model modification required.

Architecture

Codex CLI <--> MIS /v1/responses <--> Llama 3.2 3B Instruct
 (OpenAI (protocol bridge) (local GPU)
 Responses API)

Three parties interact; none required modification:

Driving a Local C++ Inference Server from Claude Code and Codex CLI

Thu, 14 May 2026 00:00:00 +0000

Mila Discussions — Inference Server Series

Introduction

One of the goals of the Mila Inference Server (MIS) is to make a locally-running Llama model feel, from the outside, indistinguishable from a cloud API. If MIS does its job correctly, you should be able to point Claude Code or Codex CLI at http://localhost:8000 and have them work as if they were talking to Anthropic or OpenAI’s servers — except your data never leaves your machine and your GPU is doing the work.

The Eight-Headed Bug That Hid Behind a System Prompt

Fri, 24 Apr 2026 00:00:00 +0000

When I added system prompt support to Mila’s chat CLI, something unexpected happened. Prompts that had previously produced perfect responses started generating fluent, coherent, and completely context-oblivious text. Ask for a story about rabbits living on the moon eating green carrots — get a charming tale about Hungarian meadow rabbits instead. The model wasn’t broken. It was just ignoring everything I said.

This is the story of how a single innocent constant — kPrefillChunkSize — quietly corrupted eight different stages of the GQA prefill pipeline, and why it took a system prompt to wake the beast.

From GPT-2 to Llama 3.2 in C++: RMSNorm, SwiGLU, Grouped Query Attention and RoPE

Sat, 04 Apr 2026 00:00:00 +0000

A show-and-tell of what it actually takes to build a modern transformer from scratch in C++23 and CUDA.

Where We Left Off

Alpha.1 ended cleanly. GPT-2 inference validated token-for-token against HuggingFace using greedy decoding. The full stack — BPE tokenizer, embeddings, multi-head attention, MLP, KV cache, two-phase prefill/decode — all confirmed correct. The methodology was established: if Mila’s greedy output matches HuggingFace’s greedy output exactly, the implementation is correct.

Alpha.2 began with a deceptively simple goal: do the same thing for Llama 3.2 1B.

The Lobotomized Attention Head Bug: One Head Does All the Work, the Others Stare Into the Void

Thu, 26 Feb 2026 00:00:00 +0000

🧠 The Lobotomized Attention Head Bug — A Transformer Debugging Rite of Passage

I just finished building the fast prefill → decode inference path in my Mila DNN library. Everything seemed fine — the model produced coherent text, KV caching worked, and decode mode looked solid.

But… my transformer’s residuals were way off compared to 🤗 HF GPT-2, and the hidden states just felt wrong. Not exploding, not NaN’ing — just wrong.