Nested Learning Experiment

A continual-learning paper is hard to trust until you run the same optimizer, the same epochs, and the same seed on both models. We compared a plain MLP with a two-speed memory model on MNIST.

The problem

Continual learning claims usually change two things at once. A new model, and a new training setup, and then a paragraph about why the model remembered. The person it frustrates is the reader who cannot tell the architecture from the schedule. It was worth running one comparison: same optimizer, same epochs, same seed, two models.

The split is MNIST in order. Digits 0–4 first. Digits 5–9 second. If the extra memory does anything, it should show up as the old digits surviving the second phase.

How I built it

Phase 1 is digits 0–4. The plain MLP reached 99.30 percent. The continuum memory system reached 99.18. On the digits they were both trained on, they are effectively the same model. That is the baseline the second phase has to beat, and it is not a dramatic gap.

Phase 2 is digits 5–9, after the first phase, without going back. The old digits fell to 0.00 for both models. The new digits landed at 97.88 for the MLP and 97.51 for the memory system. The memory system did not keep phase 1, and it did not win phase 2 by a margin I would write home about.

I wrote the model-architecture and dataset sections, the first experiment write-up, and the confusion-matrix note. PyTorch trains both. The confusion matrix is there so a 0.00 is visible as a wipe, not as a rounded average that hides a few surviving digits.

Challenges and what I learned

The second phase wiped the first. Both models fell to 0.00 on the old digits, so the extra memory did not keep what phase 1 had learned. I went in expecting the two-speed model to be the one that remembered. The matched setup is why I cannot blame the optimizer, the epochs, or the seed for that miss.