In February 2023, while deeply exploring the AI space, I realized that LLMs possess a couple of human abilities: pattern recognition and replication, and following vague instructions by applying learned information. Pattern matching solves problems like writing blogs and drawing pictures. Instruction following solves problems in digital workflows like marketing funnels, auditing jobs, and web research.

But could it have human memory? I thought further.

In ancient times, humans recited songs with their friends in the evening because they didn't have paper to write them down. They had to remember facts about which plants were safe and which were poisonous.

With the invention of paper, humans could store memory and pass it to the next generation easily. This created the world's first knowledge base. The printing press and books simply made knowledge spread across the world more easily.

Digital memory and internet search increased the amount of information we could store and our ability to fetch relevant information.

LLMs made fetching world knowledge good and also presented it well.

Until today, LLMs lacked personalized memory. They were stateless creatures that did a couple of tasks and then died, forgetting everything. But it's changing rapidly. ChatGPT has released features that remember previous conversations and build a knowledge base. Gemini brought in a file search API so you can dump all your documents into it, and all Gemini calls will fetch context from them. Limitless made a pendant that's always on, storing all the audio you hear and speak. Littlebird captures all your screen data all the time and builds memory from it. Supermemory built infrastructure so anyone can dump a memory and fetch it later efficiently.

Now personal information can also be stored and fetched. Top labs are building memory layers that take in all visuals, audio, and digital data from a person, building both short-term and long-term memory.

Queries like "when did I join my first company?" or "What was the name of the restaurant I visited in Hong Kong in February 2023?" or "What is the progress of project X in the last 3 months?"—all such queries will have genuine answers.

With the kind of queries these systems can answer now, I'm sure AI agents have far superior and more useful memory than humans.

But agent memory is different at the same time. An LLM's context window is quite small compared to a human's, but it's very accurate. It can fetch the exact date I joined my first company. But it can't analyze 20 years of experience working in a company and make judgmental decisions (until now).

What's coming up next in memory?

Google recently cracked the catastrophic forgetting problem of fine-tuning. They call it Nested Learning. With this, you can live ingest your entire day with all input types: vision with camera, audio with mics, other digital pipelines to ingest all screens and subscribed digital accounts and RSS feeds. Add your tuned voice agent to it. Now the fine-tuned AI agent is essentially you, talking with all your life's context.

This solves the entire context window size problem, and the AI could gain decision-making abilities like a 20-year experienced person.

What else is left to conquer in the realm of human intelligence? 😂

CTA: I’m open to remote contractual roles in projects that are building context aware AI agents with efficient tool calling. DM me or mailto: [email protected]

@ali abdaal would your productivity application benefit with the personalised context? Happy to contribute!