Note on the quadratic penalties in elastic weight consolidation

Catastrophic forgetting is an undesired phenomenon which occurs when neural networks are trained on different tasks sequentially. Elastic weight consolidation (EWC; ref. 1), published in PNAS, is a novel algorithm designed to safeguard against this. Despite its satisfying simplicity, EWC is remarkably effective.

Motivated by Bayesian inference, EWC adds quadratic penalties to the loss function when learning a new task. The purpose of penalties is to approximate the loss surface from previous tasks. The authors derive the penalty for the two-task case and then extrapolate to handling multiple tasks. I believe, however, that the penalties for multiple tasks are applied inconsistently.

In ref. 1 a separate penalty is maintained for each task T, centered at θT∗, the value of θ obtained after training on task T. When these penalties are combined (assuming λT=1), the aggregate penalty is anchored atμT=(FA+FB…+FT)−1(FAθA∗+FBθB∗…+FTθT∗).From the third task onward this is inconsistent with Bayesian inference. In the Bayesian paradigm the posterior p(θ |DA,DB) encapsulates the agent’s experience in both tasks A and B, thus rendering the previous posterior p(θ |DA) irrelevant. Analogously, as θB∗ was obtained while incorporating the penalty around θA∗, once we have θB∗, θA∗ is not needed anymore.

The correct form of penalties can be obtained by recursive application of the two-task derivation (2). It turns out that a single penalty is sufficient; its center is the latest optimum θT∗ and its weights are given by the sum of diagonal Fisher information matrices from previous tasks FA+FB+…+FT. This single penalty version is akin to Bayesian online learning (3).

If the agent has to revisit data from past training episodes, multiple penalties should be maintained similarly to expectation propagation (4, 5). The aggregate penalty should be centered at θT∗ rather than μT. The anchor point for task T’s penalty should therefore beθ∼T=FT−1((FA+FB+…FT)θT∗−FAθ∼A−…−FSθ∼S)rather than θT∗, where tasks A…S precede task T, and θ∼A…θ∼S are the respective penalty centers for these tasks.

To illustrate the behavior and effect of different penalties I applied EWC to a sequence of linear regression tasks. The tasks define quadratic losses each with a diagonal Hessian (Fig. 1, Left). As all simplifying assumptions of EWC hold exactly, one should expect it to match exact Bayesian inference: Reach the global minimum of the combined loss and do so irrespective of the order in which tasks were presented. Fig. 1, Center shows that although a quadratic penalty could model task B perfectly the penalty is placed around the wrong anchor point. Table 1 shows that this leads to suboptimal performance and an unwanted sensitivity to task ordering. By contrast, EWC using the corrected penalties around θ∼T models the losses perfectly (Fig. 1, Right) and achieves optimal performance in an order-agnostic fashion (Table 1).

• Download figure
• Open in new tab
• Download powerpoint

Fig. 1.

(Left) Elliptical level sets of quadratic loss functions for tasks A, B, and C also used in Table 1. (Center) When learning task C via EWC, losses for tasks A and B are replaced by quadratic penalties around θA* and θB*. (Right) Losses are approximated perfectly by the correct quadratic penalties around θ∼A=θA∗ and θ∼B.

I expect the negative impact of using incorrect penalties to be negligible until the network’s capacity begins to saturate. Furthermore, optimizing λ can compensate for the effects to some degree. As a result, it is possible that using incorrect penalties would result in no observable performance degradation in practice. However, since the correct penalties are just as easy to compute, and are clearly superior in some cases, I see no reason against adopting them.