Why most AI agents feel forgetful, and a simple, practical pattern for giving them persistent memory without fighting the context window.
Loading…
Playbooks on AI and Web3 product marketing, GTM and distribution. No spam, unsubscribe anytime.
Written by Kevan Shah, Web3 and AI product marketing strategist and founder of Hangryfeed.
Head of BD at Zealy and growth operator with 8+ years experience helping AI and Web3 products raise $26M+, scale to $2.2B monthly DeFi volume, and reach 1.2M+ users.
Imagine you’ve been working with an “AI teammate” for a few weeks.
You’ve shared your company background, your tone of voice, your product roadmap. You’ve worked through naming options, pricing experiments, draft emails, maybe even a few personal preferences—like “never use exclamation marks in investor updates” or “our customers hate the word ‘synergy.’”
Then one morning you open the chat, ask it to write a follow-up sequence, and it responds like you’ve never met.
That’s the persistent memory problem in AI agents: they’re incredibly good at generating text in the moment, but they’re shockingly bad at remembering what matters over time.
Under the hood, most agents live inside a context window—a fixed-size chunk of text the model can “see” at once.
Every message you send, every tool output, every internal note has to fit inside that window. Once it fills up, frameworks start doing “compaction”:
It’s like trying to record a multi-month project on a stack of sticky notes by constantly rewriting the older ones in smaller and smaller handwriting.
That works until it doesn’t. Compaction is lossy by design. Subtle preferences, edge cases, and long-running threads are the first to get erased. The result:
Teams end up treating the agent as disposable chat, not as a persistent collaborator.
One useful shift is to stop treating the context window as the place where all memory must live.
Instead of obsessing over better compaction algorithms and prompt tricks, some builders are taking a more pragmatic approach:
In other words: treat the agent like software, not magic. Give it files and databases it can revisit, just like a human would.
One concrete implementation of this pattern organizes memory into four layers:
NOW.md)MEMORY.md)Together, they form a hybrid memory system that actually survives compaction.
The heart of the design is a compact, stable file called NOW.md.
Think of it as a roughly 200-line snapshot of “what’s going on right now” that the agent treats as canon:
Because NOW.md is small and intentionally curated, it reliably fits into the context window even when a session gets long. When the agent needs to compress older messages, it saves essential state into NOW.md first, and only then lets the rest be summarized or dropped.
This flips the usual order of operations:
NOW.md, then treats compaction as disposable.For a founder or marketer, this is the difference between “sometimes it remembers our brand voice” and “it always knows the handful of rules that define how we sound.”
Beyond the immediate “now,” there’s a second file: MEMORY.md.
This is the agent’s long-term memory, a place to store:
Unlike NOW.md, this file can be long. It doesn’t need to live in the context all the time. The agent pulls pieces of it in when needed.
The interesting part is that the AI manages this file:
This is closer to how a human builds memory: we don’t remember every sentence of every conversation, but we do extract enduring lessons and keep them somewhere accessible.
For operators, this means you can have your AI agent slowly build up an internal “playbook” of your business—without manually curating every line.
Files alone aren’t enough. Sometimes you need to ask:
“What did we talk about regarding X?”
That’s where a vector database such as ChromaDB comes in.
Every chunk of conversation can be embedded into vectors and stored. When the agent needs to recall something, it searches by semantic similarity:
This lets the agent revisit prior discussions without keeping them all in memory, and without relying on brittle keyword search.
Typical uses:
For teams running ongoing campaigns or multi-stage projects, this turns scattered chats into something closer to a searchable institutional memory.
Finally, the system uses an SQLite-based graph to track entities and relationships:
Where the vector database gives semantic memory (“what did we talk about?”), the graph gives structured memory:
That structure matters when you want the agent to reason about your world:
Instead of scanning raw text, the agent can query the graph and then use those results to anchor its responses.
The key insight behind this design is simple:
Persistent memory is a state management problem, not a prompt engineering trick.
By externalizing state into files and databases, the agent gets:
NOW.md survives compaction.MEMORY.md holds long-term lessons across sessions.It doesn’t prove that better compaction algorithms are useless. In some cases, smart summarization of long threads is still valuable. But it does show that you don’t need to rely on compaction as your primary memory mechanism.
You can treat compaction as an optimization, not as the place where your critical business state lives.
If you’re a founder, operator, developer, or marketer trying to build agents that feel like real collaborators, a few practical lessons emerge:
Create a small, human-readable document that holds:
Keep it under a few hundred lines. Make sure every agent run loads it, and that the agent updates it before long contexts get compacted.
Don’t rely on chat history alone. Add a long-term memory file or store where the agent can write:
Prompt the agent periodically to distill recent activity into this file, instead of leaving it buried in old conversations.
Introduce a vector database for:
Train yourself and your team to ask your agent questions like “What have we previously discussed about X?” and have it pull from that store.
Even a lightweight SQLite schema can go a long way:
Give your agent tools to query and update this graph. Over time, you’ll get a rough map of your business that the agent can navigate.
You don’t need the perfect memory architecture on day one. But you do need to make sure that:
Start small: a NOW.md file and a basic memory doc will already make your agent feel dramatically more “aware” of your ongoing relationship.
From there, add retrieval and structure as your use cases deepen.