Local LLMs at the metal

Set up your project

cmake_minimum_required(VERSION 4.0)
project(MyApp LANGUAGES CXX CUDA)

set(CMAKE_CXX_STANDARD 23)
set(CMAKE_CXX_SCAN_FOR_MODULES ON)
set(CMAKE_CUDA_ARCHITECTURES native)

include(FetchContent)
FetchContent_Declare(
    Mila
    GIT_REPOSITORY https://github.com/ToddThomson/Mila.git
    GIT_TAG        v0.20.0
)
FetchContent_MakeAvailable(Mila)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE Mila::Mila)

Load a model and generate

// A module import, not a header.
import Mila;

Mila::initialize();

// The store is the only source -- loading never downloads.
auto installed = ModelStore{}.locate( "gemma-4-12b-it-fp4" );
auto tokenizer = BpeTokenizer::loadGemma( installed->tokenizer_path );

GemmaModelConfig config( 4096 );
config.withFP4Quantization();

// Device and precision are template arguments: the type IS the configuration.
auto model = GemmaModel<DeviceType::Cuda, TensorDataType::BF16>::load(
    installed->weights_path, config );

// The model owns the decode loop and pushes each token to your callback.
auto status = model->generate( prompt_tokens, [&]( int32_t id ) {
    std::cout << tokenizer->decode( std::span( &id, 1 ) ) << std::flush;
}, params );

Build it

cmake -S . -B build -G Ninja
cmake --build build

About five minutes.

Run it

./build/myapp "Why is the sky blue?"

Mila 0.20.0
Loading gemma-4-12b-it-fp4 ...

The sky appears blue because of a phenomenon called Rayleigh
scattering, where the atmosphere's gases and particles scatter
shorter blue wavelengths of sunlight more effectively than longer
red wavelengths. This scattering occurs in all directions, causing
the entire sky to take on a blue hue.

[stop]

A complete example is in Samples/QuickStart/Cpp.

Install Mila

pip install mila-llm

Install a model

import mila
mila.initialize("warning")
mila.ModelStore().pull("gemma-4-12b-it-fp4", mila.default_hub_owner())

Downloads Gemma 4 12B, 6.8 GB. About twenty minutes. Openly licensed, so it needs no HuggingFace account.

Talk to it

python quickstart.py "Why is the sky blue?"

Loading gemma-4-12b-it-fp4 ...

The sky appears blue because of a phenomenon called Rayleigh
scattering, where the Earth's atmosphere scatters shorter
wavelengths of light more effectively. Because blue and violet
light waves are shorter, they scatter more across the atmosphere,
making the sky appear blue to our eyes.

The same program as the C++ tab, in Python.

Start the environment

docker run --gpus all -it -v mila-store:/models toddthomson/mila-llm:0.20.0-devel

Downloads the image, 7.8 GB. You land in a shell, in a Mila tree that is already built.

Install a model

mila install gemma-4-12b-it-fp4

Downloads Gemma 4 12B, 6.8 GB. About twenty minutes. Later runs start with it already there.

Write your own program against Mila

cd ~/myapp        # a CMake project, already configured
vim main.cpp      # import Mila;
cmake --build build

./build/myapp "Why is the sky blue?"

Your first build takes about ten minutes. After that you are only rebuilding your own code. What you get is a Linux binary; to build for Windows, use the C++ tab.

Clone and build

git clone https://github.com/toddthomson/mila.git
cd mila
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build

The build takes about seven minutes. Use Ninja — C++23 modules need it. On Windows, open the folder in Visual Studio 2026 and pick a preset, which supplies the compiler these commands expect.

Run the chat app

./build/mila-chat

/model list --online      what is published
/model install gemma-4-12b-it-fp4

A first run opens on whatever your store holds, and /model install adds to it from inside the session.