What is the problem being addressed?
Modern AI models are trained once and then deployed — but the real world keeps changing. When you take a neural network that already knows how to do one task and try to teach it a second task, something damaging happens: the new training overwrites the weights the model used for the first task. This is called catastrophic forgetting, and it is one of the most fundamental unsolved problems in deep learning. A model that forgets is not a lifelong learner, it is just a very expensive single-use tool. In our experiments, a standard dense transformer lost 7% of its accuracy after learning just two additional tasks in sequence. There was no mechanism to protect what it already knew. The goal of this project is to build a system that can learn new tasks continuously, one after another, without ever forgetting what it learned before.
What is your project idea and how will it work?
Our project builds a three-part continual learning pipeline that addresses catastrophic forgetting through architectural restructuring, reinforcement learning, and mathematical gradient protection working together.
The first component is Mixture of Experts (MoE). We convert the dense feed-forward layers of a transformer into a set of 8 smaller sub-networks called experts. Each token only activates 2 of them, chosen by a learned router that uses a task identity tag to make task-aware routing decisions. This gives us modular control — we can protect, clone, or freeze individual experts without touching the others. We compare two configurations: 4 experts (deeper, better at retaining old tasks) and 8 experts (shallower, better at adapting to new tasks).
The second component is a PPO reinforcement learning agent that makes adaptation decisions at every task boundary. For each expert, it chooses one of three actions — Share (serve both tasks with gradient protection), Clone (give the new task its own private copy of the expert), or Freeze (lock the expert entirely). The agent observes gradient statistics per expert and learns from a reward that balances new task accuracy against forgetting on old tasks. Unlike a fixed rule, the PPO agent improves its decisions with each task transition.
The third component is Orthogonal Gradient Projection (OGP). When two tasks share an expert, we intercept every weight update and remove the component that points in the direction of old task gradients — using QR decomposition to build an orthonormal basis of the old task's gradient subspace. What remains after projection is mathematically perpendicular to old task knowledge and cannot overwrite it. We verified this empirically: cosine similarity after projection equals 0.0 exactly, and between 41% and 74% of conflicting gradient was blocked across our runs.
The full pipeline runs as follows: a transformer is pre-trained on WikiText-2, its neurons are profiled and assigned to expert buckets by a PPO agent, and then the model is fine-tuned sequentially across tasks — PIQA and Math7k — with the RL adaptation agent and OGP active at every task boundary. We measure forgetting explicitly using Backward Transfer (BWT), which re-evaluates old task accuracy after each new task is learned. Our current results show that the Phase 1 MoE pipeline achieves a BWT of +2% on PIQA after Math training, meaning the model retained and slightly improved its old task performance. The 4-expert comparison and further RL tuning are ongoing.
