A recurring claim in efficient-AI work is that external retrieval can substitute for model parameters: a small model that looks facts up need not memorize them, and so can rival a much larger model on knowledge-dependent tasks. We test this claim quantitatively and, critically, under imperfect retrieval. On a controlled knowledge-dependent classification task, a retrieval-augmented model with 561 parameters outperforms a 196,353-parameter model (350x larger) by 8.6 percentage points when retrieval is perfect. We then degrade retrieval by injecting wrong-fact errors at rates from 0% to 50% and locate the crossover: the small retrieval model's advantage persists until retrieval error reaches approximately 17%, beyond which the larger model's memorized capacity wins. All results are averaged over 8 random seeds with standard deviations reported. We additionally report a null result: across four memory-budget sizes, the choice of replay-buffer consolidation strategy (class-balanced vs. recency vs. random eviction) produced no statistically meaningful difference on our continual-learning task, correcting a hypothesis from our earlier reports.
Two questions motivate this report. First, how much can retrieval genuinely substitute for parameters, and how robust is that substitution to the imperfections of real retrieval systems — which routinely fetch irrelevant or wrong context? A result that holds only under perfect retrieval would be of limited practical value. Second, in our earlier reports (TR-01, TR-03) we hypothesized that how a bounded replay buffer decides what to keep (its consolidation policy) should matter for resisting catastrophic forgetting, especially under tight memory budgets. An earlier single-seed experiment was inconclusive; here we test it properly.
We construct a knowledge-dependent binary classification task. Each input pairs a query vector with a key that indexes one of 500 hidden 'rule' vectors; the correct label depends on the interaction between the query and the rule for that key. A model with no retrieval must therefore memorize all 500 rules in its weights, which requires capacity. A retrieval-augmented model is instead supplied the relevant rule vector as additional input features, so it need not memorize. We compare three systems at matched conditions: (A) a small network (8,433 parameters) with no retrieval; (B) a large network (196,353 parameters) with no retrieval; and (C) a small network (561 parameters) with retrieval. To model imperfect retrieval, condition C fetches the wrong rule vector with probability p, swept from 0 to 0.5. Every point is averaged over 8 random seeds; we report means and standard deviations. Training uses early stopping.
| Retrieval error | C: small+retrieval | vs. big model |
|---|---|---|
| 0% | 84.5% ± 1.0 | +8.6 pts |
| 5% | 81.5% ± 1.3 | +5.5 pts |
| 10% | 79.7% ± 1.2 | +3.8 pts |
| ~17% | (crossover) | 0.0 pts |
| 20% | 74.4% ± 0.9 | -1.5 pts |
| 30% | 70.4% ± 1.3 | -5.5 pts |
| 40% | 66.1% ± 0.6 | -9.8 pts |
| 50% | 61.8% ± 2.3 | -14.2 pts |
Table 1 — Retrieval-augmented small model (561 params) vs. the large model (196,353 params) as retrieval error rate increases. The crossover falls at roughly 17% retrieval error.
The substitution of retrieval for parameters is real but conditional. It is not a free lunch that holds regardless of retrieval quality: it holds while retrieval is accurate enough — here, above roughly 83% correct. This gives a concrete design rule. A system choosing between a large memorizing model and a small retrieval-augmented one should estimate its retrieval accuracy on the target domain; above the crossover, the small model is both cheaper and better, and below it, capacity is worth paying for. The exact crossover will depend on task and retrieval difficulty, but the existence of a crossover — and the method for locating it — is the transferable result. For narrow domains where a curated knowledge base can keep retrieval accuracy high, the small-model regime is the right one, which is consistent with the design premise of the Aethelas memory stack.
We separately tested whether the policy governing which examples a bounded replay buffer keeps affects continual-learning performance. Using real image data (four sequential binary digit tasks) and a bounded buffer, we compared three eviction policies — drop-oldest (recency), drop-random, and a class-balanced policy that keeps the buffer evenly split across classes — across memory budgets of 8, 12, 20, and 40 items, each over 8 seeds. At every budget the three policies fall within roughly one standard deviation of one another (all near 80%, standard deviations 2.4-4.2 points). The class-balanced policy we hypothesized would help under tight budgets led by only 0.7 points at the smallest budget and trailed slightly at larger ones — none of it distinguishable from noise. We report this as a null result: for this task family, the choice of consolidation policy did not matter, and our earlier hypothesis that it would matter under memory pressure is not supported. We think the most likely explanation is that the binary digit tasks are individually easy enough that even a poorly-curated handful of replayed examples suffices to prevent forgetting, leaving no room for a smarter policy to help. Whether consolidation policy matters on harder, higher-class-count tasks remains open.
Retrieval can substitute for a large parameter count on knowledge-dependent tasks — a 561-parameter model beat a 350x-larger one by 8.6 points — but only while retrieval accuracy stays high; the advantage inverts past roughly 17% retrieval error. The practical takeaway is to measure retrieval accuracy before choosing the small-model regime. Separately, and honestly, replay-buffer consolidation policy showed no measurable effect on our continual-learning task across budgets, correcting an earlier hypothesis. Both the positive result and the null result are reported with multi-seed variance; neither is peer reviewed, and both are offered as preliminary, reproducible findings.