nanochat

nanochat

A compact full-stack ChatGPT-style LLM pipeline you can train from scratch on a single GPU.

Visit website

What is nanochat?

Trying to train a language model from scratch usually means digging through scattered scripts, wiring up tokenization, pretraining, and evaluation before you can even see whether an idea works. nanochat is Andrej Karpathy's minimal repository that lays these pieces out in a single, readable pipeline: a GPT model, training loop, and evaluation all live in one place, built around PyTorch. You supply the data and a machine, and it handles the run so you can focus on the modeling decision in front of you.

The point of nanochat is scale you can actually follow. It is meant to be a complete but compact path from raw text to a model you can talk to, rather than a maze of framework code. Tokenizing the corpus, pretraining the transformer, and then fine-tuning and evaluating the result all sit in the same project, which makes it a practical base for experiments and for learning how each stage affects the next. If you have a CUDA-capable GPU, that is the main prerequisite; PyTorch and some standard packages do the rest.

That structure is also what makes it useful as a teaching and research starting point. Instead of hand-editing one big script at a time and losing track of what changed, you keep a lean codebase where a modification is visible from end to end. When you are comparing settings or trying a new training idea, the small surface area means results come from your change rather than from hidden plumbing. FindAIGo lists projects like this for people who want to understand the machinery, not just call an API.

The companion autoresearch workflow, which Karpathy spun out into a separate repository, pushes the idea further: you describe the research direction in a Markdown file, and an AI agent edits the training script and runs short experiments, keeping changes that improve validation bits per byte. That pairing is the interesting part — a compact training project that is small enough for an agent to reason over, so overnight runs can test many variations while you sleep. For anyone who has wanted to iterate on LLM training without a cluster or a team, nanochat is a grounded place to start.

加载中 / Loading

Discussion

Loading comments…