The first time an agent changes code without reading the tests, you correct it. The second time, you start to wonder where the first correction went.
It went somewhere, probably. Into the chat transcript. Into a polite apology. Into a sentence about checking assumptions next time. But unless something outside the model changes, the lesson has not really become part of the system. A week later, the same mistake can return with the same confidence.
So we create places for failure to live: prompts, skill files, checklists, project notes. Inspect the tests before changing code. Check spreadsheet headers before writing formulas. Verify the range before trusting the total.
This helps. It also creates the next problem. Once memory is easy to write down, we start over-recording it. Every failure wants to become a rule. Every rule sounds wise in isolation. “Always run the full test suite” may be the right lesson after one bad coding change, and a wasteful habit on the next small edit where one targeted test would have told the truth.
That is where memory becomes dangerous. A system that remembers everything is not wiser. It is cluttered, cautious in the wrong places, and increasingly governed by old accidents.
This is why I ended up taking SkillOpt more seriously than I expected. On the surface, it sounds familiar: let an agent attempt tasks, record where it failed, ask a model to reflect on those failures, and use the reflection to edit a skill document. My first reaction was skepticism. Is this really learning a skill, or just prompt optimization with a better name for the agent era?
I still think that question is fair. SkillOpt is not a clean break from earlier text-optimization work. TextGrad had already made the basic move feel natural: if prompts, critiques, tool instructions, and plans are text, then feedback can revise them. Anyone who has debugged an agent has done a manual version of this. Watch it fail, add a warning, adjust the instructions, try again.
But editing text is the easy part. The hard part is deciding which edits survive.
That is the part SkillOpt makes concrete. It does not merely ask the model to append advice after a failed run. It treats the skill file as something closer to an operating manual the agent will read next time. A failure can propose a bounded change to that manual. The changed version is tested inside a validation loop. If it improves behavior there, it can be retained. If it sounds plausible but causes regressions, it gets rejected. The memory is not just appended. It is earned.
That is the real distinction: not reflection, but selection.
Models are good at producing reflections that sound intelligent. “Verify intermediate results.” “Check assumptions.” “Handle edge cases.” These are not wrong, exactly. They are just cheap. They become valuable only when they change behavior in a way that holds up beyond the incident that produced them.
Spreadsheet work makes this concrete. When an agent gets a sheet wrong, the fix is rarely “be more careful.” The real lesson might be: inspect the visible layout before choosing a range; preserve formulas instead of overwriting them; recalculate and compare the answer against the table’s structure. Those are habits worth keeping if they prevent future errors. But even there, too much remembered caution can become ritual.
Coding has the same shape, but the danger is easier to feel. A useful memory might say that this repository’s generated files should not be edited, or that a migration test catches the failure mode better than the broad suite. A bad memory says “always do everything,” and slowly turns the agent into a machine for performing ceremonies around uncertainty.
This is also where the limits of SkillOpt matter. If a skill is just a few Markdown paragraphs optimized against a fixed benchmark, it can easily become a benchmark-specific cheat sheet. Real team skills are thicker. They include tool permissions, recovery procedures, project conventions, organizational rules, historical quirks, and judgment about when not to act. SkillOpt does not solve all of that. It does not make agents discover new tools or understand the boundaries of a messy organization.
But it points at the right engineering problem. A lot of agent reliability will not live only inside model weights. It will live in ordinary skill files: a few rules, a few warnings, a few exceptions left behind by past failures. We used to call these documentation. In agent systems, they may become a kind of trainable behavioral memory.
The danger is that memory can become superstition. One bad incident becomes a permanent ritual. One benchmark quirk becomes a rule. One missed edge case turns into three paragraphs of nervous prose. A system that remembers everything does not become wiser. It becomes harder to trust.
So the central question is not whether a model can write down what went wrong. Of course it can. The question is whether the system can tell the difference between a real lesson and attractive prose.
SkillOpt does not prove that agents can now learn skills in the full human sense. It shows something smaller and more useful: failure can propose changes, validation can accept or reject them, and an agent’s operating manual can evolve without simply accumulating noise.
When an agent fails, the experience should not vanish. But it should not automatically become doctrine either.
Not every reflection belongs in memory. The ones that do should have earned their place.
Originally published on Medium.
Blog 