Developer Introduces Mini-AGI, a Continual-Learning Model Trained on 8GB VRAM

The post says the model is still running over the first of a 7.8 billion-character corpus selected for training. The weights have not been released and may take about two weeks at the current reading speed. The author published a sample from the training run and a scaling-law graph, which he described as very promising.

Mini-AGI combines two main ideas. One is a mixture-of-experts architecture with many experts that are added to and pruned from the model during training, with only a small subset active at any moment. The author says this can make the number of parameters bounded mainly by disk space, because experts are loaded and unloaded only when needed. The other is batch-one training on a single continuous stream of data. If it works, the author says, it avoids storing large randomized batches and their gradients, reducing the VRAM requirement.

The author said he used Claude while brainstorming and building the project, and that it would have been impossible for him to do something like this without AI. He also acknowledged that the name Mini-AGI is pretentious and asked readers to look at the Readme before criticizing. The model reads continuous interleaved passages from the dataset, each 32K characters long, as a single stream, in a way the author compared to how a person reads.

The project setup is simple enough to clone, run and observe, according to the post. The author said the work grew out of dissatisfaction with the inability to train even moderately large models, in the 1B+ parameter range, on consumer hardware; inference and fine-tuning are possible, but he wanted full control over what the model sees during training. The Hacker News post had 33 points and four comments at the time of publication.