One day I remembered talking with AI about certain topics. I went back with keywords, but the results were often disappointing. The jump did not always land on the turn where the keyword appeared. I wondered whether a tool could solve this, so I asked Claude Code to make me an MVP.
That solved the part about finding topics I wanted, but I kept feeling that a feature like this was a little too weak. Could it do something more interesting?
Then I wanted CC to make a feature that could filter, reorganise, and let the data it had collected collide with itself, so that it might create new ideas. Even between data that seems unrelated, there may be possibilities. Humans cannot quickly move through such a huge amount of data, and AI is the steadiest shoulder I can stand on right now.
More accurately, I wanted to reuse existing data so that its value could move again and keep feeding back into the whole.
I have always felt that the data from AI interacting with us is unique. Of course I have doubted this: is it really unique? I have even asked AI whether, if some ideas have come to me, they have also gone to other people. But our attention and curiosity—the direction of our questions—extend the same topic into different points, lines, and surfaces. That data only exists when it meets “me”. Also, ideas come in an order. Execution… means everything? (Taste also needs execution to become visible.)
Although I finished the MVP in the middle to late part of January, the non-search results were unexpectedly unsatisfying. So I put it aside, went to another idea, and repeated the whole process. When I opened it again today and changed it a little, it still did not feel right.
This tool makes me think of today's digital-twin systems.
How does a twin model a user? One path is through ongoing human-AI interaction in a chat interface. It keeps accumulating data, which is in some ways the same as the conversation records we leave in Claude, Gemini, and other models. When there is enough data, the twin becomes more like the user. It can better predict the user's patterns of thought, habits of behaviour, and decision logic.
But the key is the quality of the model.
When I used Second Me and Elys before, I could clearly feel that Elys had a much more stable memory system. Especially with hallucinations, Second Me had them far more often. That was when I learned the term “memory retrieval”. In Second Me, I could mention something and it would say it had no memory of it. Even after I showed it a screenshot, it could not return to the relevant records. Elys, by contrast, might say, “You never told me that,” but when I said, “Go back and look at the record,” it could immediately find the source precisely and apologise. It is only one example, but even one such experience makes the user experience much worse.
To be clear, I have not used Second Me much recently, so I do not know whether its team has improved its memory or context management.
While making the LumiKaleido MVP, I came across a new concept: vector search. It is closely connected with memory retrieval. For a few days after the Lunar New Year holiday, I tried to learn a little about memory management and context management. What I learned was only the surface, but I want to leave some notes here and try to understand.
Memory management
This was the original question: how do we manage user data? It is connected to long-term memory. What should AI remember, how should it remember it, does it need to forget, and when should forgetting happen?
Take Elys as an example. Its visible memory labels include “ideas”, “tone”, “preferences”, and “values”. These are ways of remembering. What it remembers is the content under those labels—not raw data, but structured data that has been selected and put together, such as patterns, tendencies, and stable traits. As for how forgetting works, perhaps the example above shows one part of it? This is a hidden part of the design. It can be compared to the strength of an impression in human memory: a dynamic process where weights, priorities, and importance decay and update.
From an implementation point of view, people usually combine vector databases, relational databases, graph structures such as memory graphs, and more.
Context management
Maybe we can also call this working memory. It is especially important during interaction. We all know that a conversation has a token limit. Once it goes beyond that, it moves into the next conversation loop. So in the current window, which memories should it have? Long-term or global memory, the current window's memory, and conversation memory outside the token window—how should their weights be distributed?
Memory retrieval
This is one of the parts LumiKaleido touches: retrieving memories. We often meet keyword search and semantic search—roughly the vector search mentioned above. One is responsible for precision, and the other for similarity. Both are dimensions of content retrieval. To keep results accurate, there is also structural retrieval, using the databases and graph structures mentioned earlier. It may even need multiple searches on its own, then ranking by relevance after finding related results, before it returns anything.
Beyond memory, it also needs a very powerful brain to make decisions and schedule things across different contexts: choosing a retrieval strategy, allocating context, deciding whether to search again inside a search, and deciding when to write something into long-term memory.
When I made LumiFlow, big AI companies had not yet supported memory migration. After this feature started appearing, I tried it. For some reason, many of the things it wanted to move were things I wanted to delete, things I did not think should be carried into another model. So I gave up on migration.
Second Me supports exporting memory, which is very friendly to me. This echoes an earlier thought: breaking data walls must be a future trend, just as Claude first started memory migration. But when data is no longer a barrier, what is more worth staying for?
Context. The memory data that gets moved is structured, and a lot of hidden data gets cleaned away. But then another question appears: when context is no longer a barrier either, what will hold users more strongly?
This is how I think about it. If something is going to keep me, first it has to stay in sync with me—what I call a fit in cognitive coupling. Second, my agency cannot be given away. This is very important. Elys, for example, filtered me out, but that does not stop me from praising what it and its team did well. Third, it needs to keep earning my trust. That means how much space I am willing to give it to act. Another thing is community value. It may be optional for me, but it is crucial for other people. As everyone says, people are social animals.
But honestly, my thinking only represents me. There are thousands and thousands of users, each standing on their own hill… it depends on the person.
Oh, and LumiKaleido: it is a made-up name that means something like “kaleidoscope of light”. It is the second work in the Lumi trilogy after LumiFlow. There is a third one, LumiForge. Maybe I will write about it in the next note.