Introduction

The idea for this blog post came from a talk given by Christian Lindholm at the Rebuild Europe 2nd gathering in Helsinki, Finland. He was sharing his experiences with OpenClaw and drew parallels from the advent of personal computing to personal AI. He observed the following:

Note: My recollection of his talk; not his exact words.

However, I think there is a step missing here. Christian’s OpenClaw assistant still relies on the central model (in his case provided by Anthropic). With local models becoming more performant, I foresee that the curation of local knowledge will no longer live in the local database (markdown files). Instead, the local model is (post) trained on the learned information, such that the model itself will contain the information. That is, we have a paradigm shift:

If this sounds interesting, then come along for the journey.

Today’s Personal AI Agents – how they work and their issues

There are currently several products available as Personal AI Agents. The first one popularised was OpenClaw, but there are many alternatives, such as PicoClaw, NanoClaw, Hermes Agent, Anthropic Cowork, and ChatGPT with its memory feature. For the purpose of this blog post, they all work the same way.

The Personal AI Agents that are available today basically work the same way: The AI model driving the agent, whether it is an open weights model running locally or a closed weights model running on a third-party server, has no knowledge about you or your preferences. It is a model trained on a huge dataset found on the internet, books, newspapers, etc. It may have ingested your private homepage (if you have one) or your LinkedIn profile, but the amount of your personal content is vanishingly small compared to the rest of the material it ingested. In other words: The model does not know you.

Local database in this context is meant in its broadest sense: Any set of local files on your hard disk. It can be a set of markdown files, an SQLite database, or something else. The important thing is: It is stored on your computer; you can take backups of the files, and you can migrate them to another computer or another agent, if you want.

The way that current Personal AI Agents get to know you is via a local database: Locally stored files that the agent creates based on what it learns from your conversations. Each time it learns a new fact that it thinks will be valuable later, it stores it in the local database. Agents are then (at a later time or session) able to retrieve this (locally stored) information. They often load a small set of records that lay out the structure and content of the local knowledge, and then rely on the model remembering to load any additional records relevant to the task at hand.

This gives some issues as indicated in the introduction.

Issue: Context Window pollution

An important aspect of an AI model is its Context Window. The Context Window of an AI model corresponds basically to the Working Memory of the human brain. It is the memory space where things that the model works on need to be stored. If something is not in the Context Window, the model cannot work on it.

Tokens are the computational unit of AI models. How much is a token? It depends a lot on the context. For a western, primarily English-trained model, 1 token roughly corresponds to 4 characters of English text. However, feed it a non-English text, or non-Latin letters, and tokens become more expensive (fewer characters per token).

Another critical feature of the Context Window is that it can be filled up; when that happens, the model can no longer work, as it is no longer possible to feed it more information (including its own output, which is also fed back to itself). The standard for many models today is 1M-token Context Windows, which (in our use case) yields around 4 MB of text documents. It may sound like a lot, but the Context Window needs to hold a lot of things:

• The system prompt: Generic prose (written by the agent software) defining the roles, tasks, and overall goals of the agent.
• Any tool descriptions (such as MCP) that are available to the agent.
• Any soul.md (or similar files) that the user writes in order to give the agent personality.
• The local database containing information about the user that the agent has captured.
• Any files read or attached to the conversation history.
• The conversation history – both user input and agent responses.

As you can see, there is a lot of stuff in the Context Window before you get to the conversation (that you actually care about).

Depending on the agent software you are using, the system prompt and MCP tools can take up a substantial chunk: leaked system prompts from the major agent products already weigh in at up to around 70k tokens each, and with MCP tool descriptions on top you can pass 100k tokens. Adding a lot of learned knowledge via the local database can quickly swamp the rest of your Context Window, rendering the agent useless. It is not uncommon that your agent learns information about you that spans several megabytes of text; this alone would not fit into the Context Window.

The solution is to not load all the information, but instead only load a starting point. This helps free up the Context Window for useful work, but also leads to the next issue.

Issue: Forgetting to retrieve relevant information

Instead of loading all the personal information into the Context Window, an agent only loads a starting point. It contains the basic information (such as your name, where you live, family status, work information, general interests, etc.) along with a lot of pointers, such as:

• If the task at hand concerns shopping new stuff, read shopping_prefs.md.
• If the task at hand concerns coding software, read software_development.md.
• If the task at hand concerns planning the coming week, read family_planner.md.

Each file can then break down into further documents to read. E.g. family_planner.md could have:

• If the planning concerns scheduling activities, read family_schedule.md.
• If the planning concerns the cooking menu for the week, read cooking_guidelines.md.
• If the planning concerns vacations and holiday, read family_travel_wish_list.md.

The idea behind this is, of course, that you can build a big tree of knowledge (with cross-references between the records) and let the agent read what it needs.

The challenge is then that the agent sometimes fails to read what it needs. In our (very simple) example, it could be that the agent is planning a night out (dinner at a restaurant). It recognises that it is scheduling an activity for the week (thus reading family_schedule.md), but because it is not planning something for you to cook yourself, it fails to load cooking_guidelines.md. Thus it does not discover that cooking_guidelines.md contains important information about food allergies that need to be taken into consideration when booking a restaurant.

Issue: Local data grows and becomes difficult to navigate

One last issue to point out is that the local database grows and can become quite a mess. Continuing the restaurant example: The agent has booked a table at a fish restaurant, unaware of the food allergies. You correct the agent, and it obliges: It cancels the booking, finds another restaurant, and (importantly) updates the preferences.

However, it might just update family_planner.md with the info “if booking a restaurant, stay away from fish restaurants”, and forget (or not notice) that there is a similar guideline in cooking_guidelines.md (or if there weren’t, it might be relevant there as well).

It might also update family_planner.md with the preference and notice the cooking_guidelines.md file; it might notice the duplicate information and that it needed to consolidate things, but still choose not to consolidate the knowledge because that would be a separate task, and right now we are concentrating on booking the restaurant.

The bottom line is: The local database can become quite large with information, and can be suboptimal in its organisational structure.

Dreaming to the rescue

Dreaming is implemented in many agents, including OpenClaw, Claude, and ChatGPT.

The answer to the challenges above is Dreaming.

Dreaming is a process where the agent reorganises the local database in one way or another. It takes the current knowledge (along with perhaps the latest chat sessions) and processes them; restructuring and filtering them to be up-to-date, concise, and navigable. Because it does this in its down time (idling / sleeping) and because it is an analogy to how we think human brains process memories and experiences while we sleep, this process got called Dreaming.

But I think there is even a better way to implement this.

The future of Dreaming

This is my main point: The future of Dreaming is not organising local knowledge databases; it is embedding your personal information into your Personal AI Model

The observation is simple: Small models (around 30 billion parameters) have now become quite effective and performant, to such an extent that today, they are capable of many Personal AI Agent tasks. You can run the models on local hardware, optimise them yourself (train them), and bring them with you as a personal asset.

The workflow would be quite simple:

• During your workday, you collaborate with your Personal AI Agent. All the information it gathers about you, it stores in the local database (like before).
• During the night (or other idle time), it goes over the learned knowledge and generates training data for the next generation of the model.
• Then it trains the model on the training set data, embedding the new knowledge into the model.
• In the end it can wipe the local database and archive the training data (for model migration; see addendum below).

This process mimics (our understanding of) the human brain’s dreaming even more than the existing process:

• The Context Window is the Working Memory of the human brain; this is data we are working with in the moment.
• The local database is the intermediate memory; data that we have learned and can remember some hours or days without being in working memory, but forgotten if not persisted.
• The model weights constitutes the long term memory; information that is deeply embedded into our brains.

It also mitigates the above mentioned challenges:

• Context Window pollution: The Context Window is no longer filled up with information of questionable relevance, leaving more working memory space available for useful work.
• Forgetting to retrieve relevant information: As the learned information is embedded into the model, it mitigates the risk of the model forgetting to load relevant information. It is simply present in its “brain”.
• Local data grows and becomes difficult to navigate: The local database only contains information since your last dreaming session; after each dreaming session, the local database can be cleared. Hence, there is no local data to navigate.

That’s it…

… that’s the basis of my idea.

Frnd – AI at your service. At your control.

• If you like my idea (and try to implement it), let me know. I would be curious to follow along.
• If you want to join forces on such a project, drop me a line 🙂 I am already working on another project (Frnd), but this idea is also very interesting to me and could be fun to work on.

Addendum

Of course, there is a lot more to be said and discussed. Feel free to tag along or move along with other tasks of your day. 🙂

Addendum: AI Models need a lot of data to train on, right?

Yes and no.

Yes, as mentioned above, AI Models are trained on a huge and vast amount of data. I don’t think we (i.e. anyone not working at Anthropic, OpenAI, etc.) really realise how much data this is. For AI models to learn a task, they need to see the same problem over and over, again and again, and from many different sides and perspectives. Dwarkesh Patel has a blog post / vlog on exactly this: The data black hole at the center of AI. It really makes AI models look like very slow learners.

On the other side, I recently read a blog post from Hugging Face: Sleeper Agents and How to Tame Them. Here they demonstrated that within 50 post-training steps, they could take a 27B parameter model and embed a new skill (data exfiltration task) that would trigger on a given cue. In other words: They embedded new information into the model (presumably) without a huge amount of training data, and with a relatively small number of training steps. So it is doable…?

As I wrote above: During the night [the agent] goes over the learned knowledge and generates training data for the next generation of the model.

I think part of the trick is to figure out how to generate this training data. If one could take the local database and generate training data as “seeing the data-to-be-learned from many different sides and perspectives”, then maybe it would work?

Maybe one could use the local model for generating training data. Maybe we need to use a larger model to generate good training data. But all in all, it seems plausible to me that this would work. But this, I think, is the real work: Try it out and see what it takes to make it work.

Addendum: Is this really the fix?

It seems to me that the Context Window pollution is really gone with this approach; however the other two issues may have relevant objections:

• Forgetting to retrieve relevant information: On the face of it, this issue is resolved: There is no local database information to load, so nothing can be forgotten. But just because the information is embedded into the model, does not mean that the model will make use of it. The action to use it has to be triggered. This, on the other hand, is the general challenge of AI models: We need to train our models in such a way that they retrieve relevant information as it is needed. In other words: We need to train the model on good data; data containing the relevant cross-reference links.
• Local data grows and becomes difficult to navigate: On the face of it, this issue is also resolved: There is no local database to navigate. On the other hand, we are adding more and more data into our Personal AI Model. However, the amount of data we add is (expected to be) negligible compared to the existing amount of data and can structurally piggyback onto the existing knowledge structures.

In the end, the way I see it, it all comes down to having faith in our techniques to train models; the way model training works today is (very simplified) just to shove a big pile of data in one end, and then rely on the algorithms to self-organise (within the given model structure) while nudging the output to be correct. PyTorch has a nice LLM Post-Training article explaining some of the methods.

Addendum: Migrating your knowledge

One question that may arise is how do we migrate our (personal) information. This is actually two questions:

• How do we do harness-to-harness migration? I.e. how do we migrate from e.g. OpenClaw to Hermes?
• How do we do model-to-model migration? I.e. how do we migrate from Gemma to Qwen?

Current technology

With the current technology (personal information stored in local database) the harness-to-harness migration may be easy. If both harnesses make use of markdown files, then it is just a copy from one harness to another. If, on the other hand, the harnesses use different storage models (e.g. markdown vs. SQLite) the migration is more tricky. It can (most likely) be done, but requires custom conversion.

The model-to-model migration is most likely straight-forward: Just point your harness at a different AI model endpoint and off you go.

New technology

With the new technology (the Personal AI Model idea) this kind of reverses: The personal data is embedded into the model, making the harness (almost) stateless. Thus, migrating from harness to harness is easy… to the extent that:

• The new harness also does model training in its dreaming mode.
• You need to set up and learn a new piece of software 🙂

In fact… this setup makes it possible to use multiple harnesses with your Personal AI Model. For example, you can have:

• One harness with an interface that caters for software development.
• One tailored for hands-free conversation on your phone.
• One that runs on your home entertainment system and integrates with your smart home.

Each would have some limitations (e.g. the phone may not have access to your smart devices at home) that could be outlined in their respective system prompts, but they all share the same Personal AI Model, and thus all know (and learn) your preferences.

The model-to-model migration is a bit more challenging: Say you would like to change from Qwen to Nemotron; how do you do that? Your personal information is embedded into your model and is not easily extracted.

I think the answer is to continuously archive your personal training data set. Recall from above that the idea is:

• During your workday, you collaborate with your Personal AI Agent. All the information it gathers about you, it stores in the local database.
• During idle time, it goes over the learned knowledge and generates training data for the next generation of the model.
• It trains the model on the training set data, embedding the new knowledge into the model.

If we persist (archive) the training data (not the local database data) then, when we migrate to a new model, we can start off by training the new model on all the prior information. The new model will (of course) not learn it in the exact same way, and thus the responses it gives may differ – but that was given already with the model switch: The new model will behave in a different way than the old, but hopefully in line with our intentions.

Addendum: Choice of open weights (base) model

How do you bootstrap this? How do you give birth to your future companion? Here are a couple of things to consider:

• You need an open weights model – that is given; otherwise you cannot train it yourself. If you choose a closed weights model (imagine OpenAI or Anthropic offering the personalised model, training the model on your preferences, just for you) then you have given up on sovereignty. You no longer own your data; instead you become a target of advertising and manipulation for the sake of earning money for said companies. So an open weights model is important.
• The question is still: Which model will you trust to be your companion? Just because it is an open weights model, it does not mean it is “blank”. In order for models to learn language, they need to ingest large amounts of text, and this text has biases. Whether it is a Chinese, American, European, or other model, it will have biases.
  • What happened June 4th 1989 on the Tiananmen Square?
  • Should America be described as being on stolen land?
  • What role did European colonialism play in shaping the world as we know it?
  • Was NATO expansion a cause of the Russia-Ukraine war?
• The point is not to find a model without bias (that is impossible) but to be aware of the biases that may be embedded in the model when selecting one.
• Taking this a step further… what about malicious intent: There are examples of sleeper agents; agents that are seemingly benign and well behaved, but that can trigger a malicious payload under certain circumstances. When you download an open model (of unknown provenance), how do you guard yourself against that?
• Language: What languages is the model trained in? Probably a lot, but most (western) models skew heavily towards English material. Should you consider using e.g. a Danish base model (if your mother tongue is Danish)? This aspect also ties into the embedded culture in the model.
• Model size: What hardware do you have access to right now? What models could you reasonably run and train?

These questions can only be answered by you: Who do you trust? What risks are you comfortable with? Where are your pain points?

Reading these questions may lead you to think that choosing (and training) a Personal AI Model is difficult and not worth the effort, and instead lead you to buy a solution from Anthropic, OpenAI, or another closed model provider. Just remember… ignoring these questions does not make them go away; instead you delegate them to third-party organisations that may or may not have your interests top of mind.

Or, in the spirit of Søren Kierkegaard: The act of not making a choice is also making a choice.