Independent research / 2026
Kimi K3 from Scratch.
A language model, from its mechanisms up
A paper is a compressed specification, not an implementation. I rebuilt a reduced-scale Kimi K3 language backbone in PyTorch to understand how its mechanisms interact and what it takes to train them reliably. What began with mathematical derivations and component tests grew into a complete 6B-token pretraining run, followed by an independent next-token evaluation.
PyTorchCUDAHybrid attentionSparse MoEMuon & AdamWLLM pretraining
The question
How can a model combine recurrent memory with attention, route computation through sparse experts, and still be trained and evaluated as a coherent system? The project treats the paper as a sequence of mechanisms to reproduce and validate, rather than a list of features to imitate.
Inside the model
The 13-layer, approximately 294.7M-parameter configuration alternates Kimi Delta Attention with Gated Multi-head Latent Attention and uses Attention Residuals, SiTU-GLU and Stable LatentMoE. About 100.4M parameters are active per token. I implemented readable recurrent and chunkwise KDA references and compared them with a fused GPU path.
- Recurrent and chunkwise KDA with numerical equivalence checks
- Gated MLA and attention residual mixing
- Sparse experts with Quantile Balancing and top-2 routing
Training and evidence
To test the implementation beyond unit tests, I built a training stack with Muon and AdamW, sharded data, checkpoint/resume, monitoring and validation. The 294.7M-parameter model completed 6,000,017,408 training-token positions across 183,106 optimizer updates on an H100 NVL. A separate next-token evaluation on 65,536 validation tokens yielded a perplexity of 21.14 (3.0510 nats/token mean NLL). These figures describe this specific model and evaluation protocol, not a comparison with frontier language models.
What this project taught me
Architecture reproduction is inseparable from numerical behavior, data, optimizer design and observability. Passing a forward test is only the beginning: a training run needs reproducible state, meaningful measurements and careful distinctions between implementation correctness and model quality. Greedy generation also revealed repetition, a reminder that completing pretraining is not the same as building a reliable conversational assistant.
Reduced-scale, language-only reconstruction of the Kimi K3 backbone, not the original full-scale model or an instruction-tuned assistant. The optimized fused KDA CUDA kernel uses the upstream Flash-Linear-Attention implementation. Pretrained weights are publicly available on Hugging Face; full training checkpoints and historical artifacts are not distributed. Reported next-token metrics are specific to this evaluation setup.