Sequentially Acquiring Concept Knowledge to Guide Continual Learning

Teaching continual learner's what to prioritize next, guided by what they already know.

CoLLAs 2026

Shivanand Kundargi¹ Kowshik Thopalli² Tejas Gokhale¹

¹ University of Maryland, Baltimore County ² Lawrence Livermore National Laboratory

Overview of SACK: a concept extractor reads the model's current knowledge, a concept-based scoring step compares it against the concepts of incoming classes, and the resulting class-wise weights guide training on the next experience.

The goal of continual learning (CL) is to adapt to new data (plasticity) while retaining knowledge acquired from old data (stability). Existing methods focus on balancing stability and plasticity to mitigate catastrophic forgetting, but the impact of the order and nature of samples a network trains on remains under-explored. We argue a CL algorithm should be able to rank incoming samples by their relationship to prior data and their effect on learning.

We propose SACK (Sequentially Acquiring Concept Knowledge), a plug-and-play technique that dissects a model's categorical knowledge into fine-grained concepts, computes the relationship between previously-learned and new concepts at each experience, and uses this relationship to prioritize new samples. Across regularization, replay, and prompt-based CL methods in both class- and task-incremental settings, SACK improves accuracy, reduces forgetting, handles long-tailed distributions, sharpens semantic grounding, and yields better-calibrated, more robust models, with negligible computational overhead.

Highlights

        Concept-based curriculum — assigns importance weights to incoming classes from the semantic overlap between new-class concepts and the concepts the model has already activated.
Model-, method-, setting-agnostic — drops into any CL approach with no architectural change, on ResNets and Vision Transformers alike.
Extensive evaluation — CIFAR-100, fine-grained CUB-200, and distribution-shifted ImageNet-R, in class-incremental and task-incremental settings.
Broad secondary benefits — better long-tail performance, sharper semantic grounding, and improved calibration, beyond accuracy and forgetting.

      

Method Overview

Concept extraction

Concept-based scoring

SACK first asks what the model already knows: paired image and text encoders compare the model's neuron activations against a growing vocabulary of natural-language concepts, surfacing the concepts each neuron has come to represent (left). It then asks how the next experience relates to that knowledge, the concepts describing each incoming class are compared against the model's activated concepts, and the resulting similarities are aggregated into a single importance weight per class (right).

Gains Across the Whole Stream

Accuracy after every experience, not just at the end of the stream, for LWF, iCaRL, and CODA-Prompt with and without SACK on a CIFAR-100 task-incremental stream.

A single final-accuracy number doesn't really convey about how a method behaves after each experience. In the figure above we check the performance of SACK on each experience for LWF, iCaRL, and CODA-Prompt on a CIFAR-100 task-incremental stream. We observe that at each experience, integration of SACK with any baseline method yields better performance than the baseline method alone. The same pattern recurs on the fine-grained and distribution-shifted benchmarks, and in the class-incremental setting, detailed in the paper.

Where SACK helps most

Gains scale with how much of the network the base method actually updates. Prompt-based methods learn only a small set of prompt parameters, so concept-based prioritization has comparatively little to act on; replay- and regularization-based methods that update the full backbone see the clearest, most consistent improvements, a pattern we observed across every backbone method tested.

Learning to Look in the Right Place

Grad-CAM attention maps on CUB-200 bird images across early, middle, and late experiences, comparing DER with DER trained under the SACK curriculum.

Grad-CAM attention on fine-grained bird images, tracked across early, middle, and late experiences of a CUB-200 stream. In each pair, the left map is a replay-based baseline (DER); the right is the same method trained under the SACK curriculum. As the stream progresses, the baseline's attention increasingly drifts toward background texture, while the SACK-guided model keeps anchoring on the head, beak, and other parts that actually distinguish one species from another, a visual sign that the model is still reasoning about the concepts it was meant to learn, rather than leaning on incidental shortcuts.

Robust to the Choices Behind the Scenes

Three robustness checks: sweeping the concept-selection threshold, injecting noisy concepts, and shuffling class order, all in the class-incremental CIFAR-100 setting.

Sweeping the concept-selection threshold (left) shows the SACK-guided curve holding its advantage over the baseline across a wide operating range. Deliberately injecting noisy concepts into the vocabulary (center) leaves the accuracy trajectory essentially unchanged, since the thresholding step already filters out irrelevant signal. And reshuffling which classes appear in which experience (right) still produces a SACK-guided curve that tracks above its baseline across permutations.

The Idea Doesn't Depend on a Single Prioritization Schema

Comparing several ways to aggregate concept similarity into a class weight, all tested against the iCaRL baseline in both class- and task-incremental settings.

The concept based scoring is at the heart of SACK and is a design choice, concept similarities can be aggregated with a simple maximum, an average, or a softer softmax-style blend. Every variant tested improves over the baseline in both settings.

Three further checks: comparing curriculum interpolation schedules, tracking confidence calibration, and evaluating performance under long-tailed class distributions.

(a) shows that the direction matters: starting from what the model already knows and gradually exploring outward outperforms starting from uniform sampling or from a deliberately inverted ranking among other variants of SACK. (b) Confidence calibration, measured on early data as the model keeps learning new experiences, holds steady or improves under SACK across every logit-based baseline. (c) And on an artificially long-tailed version of CIFAR-100, where classes arrive with uneven numbers of examples, the SACK- guided curriculum still comes out ahead across regularization-, replay-, and prompt-based methods alike.

Holding Up Under Further Adversarial Conditions

Four more checks, each run in a setting not already shown above, tell the same story from a different angle.

Accuracy across the experience stream under contrast loss, frost, and JPEG compression corruptions at several severities, comparing iCaRL with and without SACK.

Image corruptions

Under contrast loss, frost, and JPEG compression at several severities, the SACK-guided model keeps a visible edge at mild-to-moderate corruption.

Sweeping the concept-selection threshold across four additional continual learning baselines: LwF, CODA-Prompt, DER, and DER++.

Threshold, across baselines

SACK is not sensitive to the concept-selection threshold across various backbone methods.

Class-order permutation robustness across four continual learning baselines: LwF, CODA-Prompt, DER, and DER++.

Class order, across baselines

Across independently shuffled class orders and four different backbone methods, the integration of SACK yields better performance than the baseline method alone.

Class-incremental results on a 300-class subset of the naturally long-tailed iNaturalist dataset, comparing iCaRL with and without SACK.

A larger, messier benchmark

On a large, naturally long-tailed slice of iNaturalist, the same advantage persists.

BibTeX

Proceedings details (volume, pages) are TBD and will be updated once available.

TBD