Memory in Large Language Models
“Memory” is an overloaded word in large language model systems. It can refer to information encoded in model parameters, tokens supplied in the current request, cached attention tensors, retrieved documents, recurrent state, or records stored by an application. These mechanisms have different lifetimes, costs, failure modes, and privacy implications. Treating them as one feature leads to architectural mistakes. A larger context window does not create cross-session memory. A key–value (KV) cache does not make a model remember a user. Retrieval does not update model parameters. A product that recalls a preference after a week is using storage and orchestration outside the base model unless the model itself has been retrained or edited. ...