Memory

We can now retrieve documents with RAG. We can call tools and APIs with function calling. But imagine this conversation with a personal assistant:

Can the assistant answer? Sometimes yes. Sometimes no. The reason is subtle but crucial: the model itself does not remember the earlier conversation. Every inference call starts from the tokens the application sends right now.

How can an AI application remember information across interactions?

People do not remember everything equally. Some information stays in working memory: the sentence you are reading, the phone number you are about to type, the current step in a recipe. Some information becomes long-term memory: your city, your food preferences, your friend's name. Some information gets written down in notebooks, calendars, or databases. Some information is forgotten.

AI applications need the same kind of structure, but the mechanism is different. Human memory is biological. LLM memory is usually application code around a stateless model.

The common misconception is: the LLM remembers. Usually it does not. The application remembers, then writes relevant memories into the next prompt.

Suppose the model can receive at most $N$ tokens in its context window. The conversation has length $L$ tokens. If $L \le N$, the whole conversation can fit. If $L > N$, some tokens must be omitted.

The word forgotten here means forgotten by the current model call. The old messages may still exist in a database, transcript, or log, but the model cannot use them unless the application sends them again.

Memory storage can also be viewed as thresholding. A candidate memory receives an importance score $I$. The system stores it only if it crosses a threshold $\tau$:

A low threshold stores too much. A high threshold forgets useful facts. Good memory design is partly the art of choosing what deserves to persist.

The most important distinction in this lesson is:

Model weights may contain general facts learned during training. Context contains the exact tokens provided right now. External memory is persistent application state, such as a user profile, vector database, conversation summary, or task record.

The human analogy is useful, but imperfect. It helps name different memory types, but an LLM does not literally have a hippocampus, lived experience, or conscious recall.

Procedural memory is the most metaphorical row. A model can have learned routines in its weights, and an application can encode procedures in code. But updating those procedures during a normal chat is not the same thing as storing a user preference in a memory database.

Short-term memory usually means conversation state included in the prompt. The application keeps a message list, then resends recent messages to the model.

This is simple and powerful. It lets the model refer to earlier turns, resolve pronouns, follow corrections, and maintain a coherent local conversation. But it is bounded by the context window. A six-month relationship cannot be represented by resending every previous message forever.

Long-term memory is information stored outside the prompt and retrieved later. For a personal assistant, useful memories might include:

• The user's favourite programming language is Rust.
• The user lives in Vancouver.
• The user is vegetarian.
• The user's dog is called Luna.
• The user prefers concise morning summaries.

On a later request, the memory retriever searches for relevant records and inserts only those records into the prompt. That keeps the prompt smaller and makes memory selective rather than a giant transcript dump.

A memory system should not store every sentence. If it does, memory becomes noisy, expensive, and invasive. Store facts that are stable, useful later, and appropriate to remember.

Practical systems often score candidate memories by importance, stability, sensitivity, recency, and usefulness. A medical detail, address, or identity attribute needs much stricter handling than a preference for short explanations.

Memory retrieval often looks like RAG. The system embeds memories, embeds the current query, compares them with cosine similarity, and retrieves the most relevant records.

If the user asks Can you suggest dinner?, the memory The user is vegetarian should rank highly. If the user asks What should I name this Rust crate?, the programming-language memory is more relevant.

Retrieval should also respect permissions and context. A memory may be relevant but inappropriate to use in a particular situation.

Memory is not append-only. Suppose the user says:

If the store already contains I live in Vancouver, the system needs conflict resolution. It might replace the old memory, keep a version history, add a timestamp, lower confidence in the old record, or ask the user to confirm.

Good memory records often include fields like:

{
  "id": "city.current",
  "text": "The user lives in Toronto.",
  "type": "semantic",
  "source": "user stated directly",
  "created_at": "2026-07-10",
  "supersedes": "city.vancouver.2025",
  "confidence": 0.98
}

Long conversations can be compressed into summaries. Instead of sending 100 messages, the application sends a compact state summary: user goals, decisions, open questions, and preferences.

Summarisation is lossy compression. It reduces tokens, latency, and cost, but it can discard details. A good summary preserves information needed for future turns and leaves out local chatter.

Memory should not grow forever. Forgetting is a feature, not only a failure. Systems may delete or deprioritize memories using expiration times, recency, frequency, importance, manual deletion, or least-recently-used policies.

For example, the user is visiting Seattle this weekendmay expire after the trip. the user is vegetarianshould persist until changed. A user should also be able to inspect, edit, and delete stored memories.

Start with a simple in-memory store. This version is not production-ready, but it shows the core operations: add, retrieve, overwrite, and delete.

class MemoryStore:
    def __init__(self):
        self.memories = {}

    def add(self, key, text, importance=0.5):
        self.memories[key] = {
            "text": text,
            "importance": importance,
        }

    def get(self, key):
        return self.memories.get(key)

    def overwrite(self, key, text, importance=0.5):
        self.memories[key] = {
            "text": text,
            "importance": importance,
        }

    def delete(self, key):
        self.memories.pop(key, None)

store = MemoryStore()
store.add("favourite_language", "The user's favourite programming language is Rust.", 0.94)
store.add("diet", "The user is vegetarian.", 0.9)
store.overwrite("city", "The user lives in Toronto.", 0.88)
print(store.get("favourite_language"))

Semantic memory adds retrieval. A production system would use a vector database, but the shape is:

from sentence_transformers import SentenceTransformer
from numpy import dot
from numpy.linalg import norm

model = SentenceTransformer("all-MiniLM-L6-v2")

memories = [
    "The user's favourite programming language is Rust.",
    "The user lives in Vancouver.",
    "The user is vegetarian.",
    "The user's dog is called Luna.",
]

memory_vectors = model.encode(memories)

def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))

def retrieve(query, k=2):
    query_vector = model.encode([query])[0]
    scored = [
        (cosine(query_vector, memory_vector), memory)
        for memory_vector, memory in zip(memory_vectors, memories)
    ]
    return [memory for score, memory in sorted(scored, reverse=True)[:k]]

print(retrieve("Can you suggest a vegetarian dinner place?"))

A tiny assistant combines extraction, retrieval, prompt building, and updating:

def extract_memory(user_message):
    if "favourite programming language is" in user_message:
        return ("favourite_language", user_message)
    if "I live in" in user_message or "I now live in" in user_message:
        return ("city", user_message)
    if "vegetarian" in user_message:
        return ("diet", "The user is vegetarian.")
    return None

def build_prompt(user_message, store):
    relevant = []
    if "favourite programming language" in user_message:
        relevant.append(store.get("favourite_language"))
    if "dinner" in user_message or "restaurant" in user_message:
        relevant.append(store.get("diet"))
    if "where do I live" in user_message:
        relevant.append(store.get("city"))

    memories = [item["text"] for item in relevant if item]
    return {
        "system": "Answer using relevant memories only when helpful.",
        "memories": memories,
        "user": user_message,
    }

message = "My favourite programming language is Rust."
memory = extract_memory(message)
if memory:
    key, text = memory
    store.overwrite(key, text, importance=0.9)

prompt = build_prompt("What is my favourite programming language?", store)

Modern assistants can implement memory at the product layer. Conceptually, the system identifies potentially useful facts, stores them in a user-controlled memory layer, retrieves relevant memories for future conversations, and gives the user ways to view, update, or delete them.

The details vary by product, but the high-level idea is the same: the model is not quietly changing its weights after every chat. The application is maintaining state and deciding which state to include in future prompts.

Memory should be evaluated like any other system component. Useful metrics include retrieval accuracy, false retrieval rate, memory precision, memory recall, storage growth, latency, cost, update correctness, deletion correctness, and privacy incidents.