The AI Co-Pilot · Part 3 of 7
The Knowledge That Compounds
In short
- A bigger context window is not memory: it costs every call, dilutes attention and has no structure.
- Four memory types (user, feedback, project, reference) as small files with a one-line-per-entry index, loaded selectively.
- One feedback memory on the feed-in sign convention turned a draft-breaking error into a structural prevention.
- 4memory types, each with its own job
- 100+entries in the memory index
- 3versions of the quality metric, each after a real failure
Why a long context window is not the answer
The easy assumption is that a bigger context window solves memory. Paste more history in, job done. It doesn’t work, for three reasons.
Cost. Every token in the window is paid for on every call, whether the task needs it or not.
Attention. Models have trouble with material buried in the middle of a long context, and the attention budget is finite everywhere. Stuffing the window isn’t the same as making it usable.
Structure. This is the one that broke me. A flat dump of old conversations is a transcript, not a knowledge base. Finding a lesson from three weeks ago means scrolling through two hundred messages of unrelated work, and the assistant fares no better than you.
What you need is a filing cabinet: a small index at the top, organised by topic, pointing at self-contained notes the assistant opens as required.
Four types of memory
The categories do real work. Each answers a different question.
User memories answer who I’m working with. Background, role, preferences, how much needs explaining. Mine says British English, direct, don’t over-explain the basics. A future session knows not to open with “Great question!” or explain what a battery’s state of charge is.
Feedback memories answer what I’ve told it about how to work. Each has the rule, a Why line with the incident behind it, and a How to apply line. The why matters. A rule without context goes brittle the moment the situation doesn’t match exactly. One example: never send a set-mode call and a set-temperature call to a heating zone as separate actions; combine them. The second overrides the first before it takes effect and the unit flicks off. That memory sat quietly for months and stopped the same mistake being reinvented in at least three new scripts.
Project memories answer what’s going on. Decisions, current state, deadlines, what’s blocked on what. These decay fastest. One from three months ago may be wrong now. That’s fine, provided wrong memories are updated or removed rather than left to pile up.
Reference memories answer where things live: which system holds the retailer billing data, which dashboard is the source of truth for what. They are short and purely factual. Their job is to stop the assistant asking where the billing data is for the fiftieth time. The total disk footprint is trivial. The total cognitive footprint across a project is enormous.
The index matters as much as the notes
The index is loaded at the start of every session. It isn’t a memory. It’s a map to them: one line per entry, organised by topic, each under about a hundred and fifty characters, with a hook saying what the note holds.
- battery grid-charge automation — live 8 April, three helpers, 14:30 cutoff
- feed-in sign convention — critical: negative means earning. Got an earlier draft entirely wrong.
- Home Assistant API — endpoints and the dashboard save pattern
The hooks are the load-bearing part. The assistant reads the index, decides which notes the task needs and opens only those. It needs to know what’s available and reach for the right ones.
My index now runs to a little over a hundred entries. Early on it was heavy on user memory, calibrating tone and preferences. Now user memory is stable and feedback keeps growing as new failure modes surface.
A worked example: the sign convention that broke a draft
The same error featured in the last post. It is a better illustration of memory than of review, because the fix is almost entirely about memory.
The data stores feed-in values, the rate at which the grid buys a household’s excess solar, with an accounting sign. Negative means earning. Positive means cost. That is how Amber, the retailer I was with then, presents it, and it is consistent in the billing data.
My first knowledge base described the field as an “all-in feed-in tariff in cents per kilowatt-hour” and left the sign out. An overnight drafting agent took positive numbers as earnings, because that is what “tariff” suggests, and wrote a post saying the household had earned a certain amount over three months. The magnitude was right and the sign wrong, so the whole piece was inverted. The same framing was in the knowledge base for a second batch of overnight drafts, and it went into several at once. That is what turned a one-off error into a systemic one.
The fix was a single feedback memory: the field uses accounting sign, negative means earning, negate before presenting from the homeowner’s view. It had the incident as its why, and applied to any analysis, chart or draft touching feed-in values. Once it had a line in the index, every session touching billing data read it. The rerun the next evening started with it in context, and none of the drafts reproduced the error. The assistant hadn’t got smarter in a day. It had a different baseline.
The same pattern has repeated with smaller errors. An early dashboard read the battery power sign the wrong way round. Once the memory existed, no later dashboard repeated it. An early session assumed a configuration file could be edited in place when it lived on a separate filesystem. That memory has saved several wasted hours since.
The point of the worked example is not that the error was particularly embarrassing, although it was. The memory converted a one-off incident into a structural prevention. Without it, I’d be relying on reminding the assistant myself, every time, forever.
How memories get written
A memory is a short block with a name, a one-line description, a type and a body. The index gets updated whenever one is added. That’s the protocol.
The hard part is when to write one. Save too eagerly and the index becomes unreadable noise. Save too sparingly and lessons vanish into the transcript.
My test is whether the learning is non-obvious from the project files. If it can be rediscovered by reading the code or configuration, it doesn’t need a memory. If it’s why a decision was made, or what not to do because of a past incident, nothing else captures it, so it goes in.
The other trigger is my feedback. When I correct the assistant (“no, do X, because Y”) or confirm a non-obvious choice worked, that’s a candidate. Confirmations are easy to miss because they sound like praise. They’re arguably more valuable than corrections, because they validate a judgement call that would otherwise stay implicit.
A memory that turns out wrong is worse than none, because it actively misleads. When new evidence contradicts one, I update or delete it. It’s a text edit.
Why this beats one long project file
The alternative is a single long instruction file the assistant reads at the start of every session. I have one, and use both.
The project file is the canonical description of the project: rules and patterns true for every task, the incident log, the file inventory. The memory system holds personal context and decision history that a given session may or may not need. How I communicate isn’t project knowledge, so it lives there. Feedback about specific past incidents is indexed and loaded on demand, not pasted in full every time.
The two overlap on purpose. A critical lesson can live in both, as a hard rule in the project file and a fuller feedback memory. Belt and braces, where forgetting would be expensive. For most things one is enough.
A single giant file fails the way a long context window fails. The assistant either reads all of it every time, which is expensive and dilutes attention, or reads none of it closely and misses the important bits. A small index plus discrete notes lets it decide what’s relevant and leave the rest on disk.
What compounding looks like
Three months in, I can point at where it has paid off.
New sessions start calibrated. I don’t re-explain British English or that I don’t need solar basics. Week one took several exchanges to calibrate. This week took none.
Failures don’t repeat. The sign convention is the headline example. A charting library silently accepts an integer in one config field and then misbehaves. That’s now a memory, and no session has repeated it. YAML patterns that broke first time now work first time.
Quality has a floor. The first draft of a blog post is no longer a blank page. It is a blank page plus an index pointing at the quality metric, the editorial gate, the known-bad framings and the sign conventions that matter. The fifth post’s first draft is measurably better than the first post’s, and not because I wrote a better prompt.
The metric evolves. It has gone through three versions, each after a real failure. The change itself is a memory, so later tasks can’t relapse into the old shape.
None of that came from a clever prompt. It came from writing things down in a structured place the assistant can find.
Boring on purpose
The memory system is a folder of text files and an index. No embeddings, no vector database, no fuzzy retrieval, no semantic graph. The assistant reads the list and picks the files to open.
I have looked at the cleverer approaches and may end up using some. For now dullness is a feature. I can read the index in a minute and check any memory against reality. I can edit or delete anything I don’t like. Nothing is hidden behind a retrieval score. Every entry is something I decided was worth keeping, in prose a human can read.
If it fails, say by acting on a stale memory or missing the obvious connection, I can see why, because it is all plain text. On a personal project where I’m the only operator, that is worth more than a cleverer system I can’t inspect. The knowledge compounds because the machinery is simple enough to trust and plain enough to audit. That is the whole trick.
The point
The first session on any kind of task is exploration. The fifth shouldn’t be. The difference is whether the lessons from sessions two to four are still in the room, and organised well enough for the assistant to find.
Four categories, a short index, discrete notes, clear triggers for writing, and updates when things change do that work. It isn’t the only way. It has worked for me on a project with more workstreams than I can hold in my head, run in eight hours a week with no team and no illusions about remembering what I learned three weeks ago.
The filing cabinet beats the transcript. The feedback memory beats repeating yourself. And the fifth draft starts somewhere the first couldn’t have reached, however good the prompt.