nanoMFM

What is this?

nanoMFM is an educational codebase that teaches the modeling fundamentals behind modern generative and multimodal models. It strips each idea down to its essentials so you can read every line, fill in the gaps, and train a working model yourself, without getting lost in engineering overhead.

Designed to be minimal. Relatively small datasets (TinyStories, MNIST, CIFAR-10, MultimodalCLEVR, and The Cauldron / MMStar for the VLM) and small models keep compute modest. Everything was tested on 1–4 V100 GPUs. The fundamentals you learn here translate directly when scaled up.

The Modules

Five self-contained modules, building from a single-stream language model up to multimodal any-to-any generation and vision-language understanding.

Suggested order: for the first three, we recommend nanoGPT → nanoMaskGIT → nano4M. The last two (nanoFlow and nanoVLM) are more independent. In general you can start wherever you like, since every module is self-contained.

01

nanoGPT

Autoregressive transformer for language & image generation

Start with the foundation: a GPT-style decoder-only transformer trained next-token prediction on TinyStories and MNIST. Implement attention and the transformer trunk.

📓 Notebook </> gpt.py

02

nanoMaskGIT

Generation via unimodal masked modeling

Move from left-to-right to parallel, iterative decoding. Inspired by MaskGIT, you generate images (and text) by progressively unmasking tokens, the masked-modeling recipe that also powers recent diffusion-style language models like LLaDA.

📓 Notebook </> maskgit.py

03

nano4M

Multimodal any-to-any generation

Generalize masked modeling across modalities. Following 4M, you train a single model that maps any subset of modalities to any other (RGB, depth, surface normals, and scene descriptions) on the MultimodalCLEVR dataset.

📓 Notebook </> fourm.py

＋ 4M Tutorial (pretrained)

04

nanoFlow

Continuous-time generative modeling

Switch from discrete tokens to continuous data with Rectified Flow / Flow Matching and a DiT-style denoiser. Learn the velocity field that transports noise to data on MNIST and CIFAR-10.

📓 Notebook </> rectified_flow.py

05

nanoVLM

Vision-language model for image understanding & VQA

Bring it together into a vision-language model. Following LLaVA, you connect a vision encoder to a language model through a lightweight projector and train for captioning and visual question answering on The Cauldron, evaluating on MMStar.

📓 Notebook </> vision_language_model.py

Get Started

All exercise scripts and their corresponding notebooks are provided in our GitHub repository. Follow the setup and usage instructions at the link below to get started.

Citation

If you find nanoMFM useful, please consider citing it.

@misc{bachmann2026nanomfm,
  author    = {Bachmann, Roman and Gao, Zhitong and Khattak, Muhammad Uzair and Ye, Mingqiao and Zamir, Amir},
  title     = {nanoMFM: Educational Multimodal Foundation Models},
  publisher = {GitHub},
  year      = {2026},
  url       = {https://github.com/EPFL-VILAB/nanoMFM}
}

Acknowledgments