I have thousands of notes.
Obsidian vaults, random Markdown files, half-finished research documents, meeting notes from three years ago.
And for the longest time, they just sat there, searchable by filename, searchable by keyword, but never actually understood.
Then one evening I thought, What if I could just ask my notes a question?
You might be thinking the ChatGPT already does that, but it does not have access to your local workspace.
You need to upload all the files you want to discover, not a cloud service that makes me wonder what happens to my data.
Instead, I thought of using just a small AI model, completely running on my own machine.
I thought that it could handle these situations, which could read my notes and answer questions about them.
I tried it.
It worked.
And I have not stopped using it since.
No GPU required, no monthly subscription, no data leaving my machine.
Here is the honest truth: most knowledge management systems fail not at storing information, but at retrieving it.
You spend time writing the note.
You tag it.
You organize it.
And then, three weeks later, you cannot find it because you cannot remember what you called it or which folder you put it in.Traditional search is keyword-based.
It finds notes that contain the word “containerization”, but it cannot answer “how do I deploy a container in a restricted environment?” by synthesizing your notes on Docker, your notes on firewalls, and that one Markdown file you wrote during a workshop.That synthesis is exactly what a local AI model can do.
The technique is called RAG (Retrieval-Augmented Generation).
The AI does not memorize your notes.
Instead, it reads them at query time, pulls the most relevant chunks, and uses them as context to generate an answer.
Think of it as giving the AI your notes as a reference book every time you ask a question.
A fully local setup with three components:
Everything runs on the local machine.
The notes never leave the computer.
The model does not even need an internet connection once it is downloaded.
This is the part most guides get wrong.
They recommend a 7B parameter model.
You try to run it on your laptop, and it takes 90 seconds to answer a simple question.For a knowledge base assistant that runs comfortably on a CPU, even a mid-range one, the sweet spot is Phi-3 Mini (3.8B parameters from Microsoft) or Gemma 2B (from Google).
Both are surprisingly capable for question-answering tasks, and they run well with 8 GB of RAM.
You can also refer to this article for information about models running on CPU.My recommendation: phi3:mini, it is fast, accurate, and was specifically fine-tuned for instruction-following tasks like Q&A.
It is the model I use for my own notes.
Ollama is the engine that lets you run LLMs locally.
It is a single binary, and the installation is one command.
Once installed, verify it is running:
Ollama starts a background service automatically.
You can check its status:
Now download the model.
For the CPU-friendly, low-resource setup I recommend:
This will download about 2.3 GB.
Grab a coffee.
If you have more RAM available (16 GB+) and want slightly better answers, you can also try:
To verify the model is installed and working, test it immediately:
If you see a coherent response, the model is working.
We can move on.
AnythingLLM is a self-hosted, open-source application that lets you upload documents, point it at a folder, and query them through a clean chat interface.
It handles the chunking, embedding, and retrieval pipeline so you do not have to set up a vector database yourself.
Create a directory to store AnythingLLM’s data (your note indexes will live here):
Now run it:
Verify the container is running:
Now open your browser and navigate to: http://localhost:3001
You will see the AnythingLLM setup wizard.
During setup, AnythingLLM will ask you to choose an LLM provider.
Select Ollama from the list.
For the embedding model, which converts your notes into searchable vectors, stay within Ollama and choose:
nomic-embed-text is tiny (274 MB), fast, and produces excellent embeddings for English-language text.
It is purpose-built for document retrieval, exactly what we need.
Let’s move to connect knowledgebase with AI.
In AnythingLLM, a workspace is a scoped knowledge base.
Think of it like a project folder; you can have one workspace for your work notes, one for personal research, one for a specific book you are reading.
AnythingLLM will split your notes into chunks (typically 500–1000 tokens each) then generate an embedding vector for each chunk using nomic-embed-text and then store everything in a local vector database (LanceDB, stored in ~/.anythingllm).
This process takes a minute or two, depending on how many notes you have.
Now is the time to make the most of the notes by using AI to question it.
Now the satisfying part.
Click on your workspace, type a question in the chat box, and press Enter.
To truly see the power of local AI, ask something that ChatGPT could never possibly know.
For example, if you uploaded your home server notes:
Notice that AnythingLLM shows you which notes it used to generate the answer.
This is the RAG process in action; ChatGPT can tell you what Traefik is, but only your local AI knows that you switched to it because you were tired of managing SSL certificates manually, and that your Pi-hole is at 192.168.1.100.
Try a few more highly personalized queries:
Each answer is grounded in your notes.
The model is not making things up from its training data; it is synthesizing from what you wrote, which means the answers are accurate to your actual knowledge and context.
If you want AnythingLLM to start automatically with your system:
And Ollama is already managed as a systemd service, so it will start on boot automatically.To update the model later if a newer version is released:
Ollama handles versioning for you.
I have been using this setup for a few months now.
Here is what changed for me:Before: I would search my notes, find five partial answers in different files, open each one, manually cross-reference them, and then write a synthesis myself.After: I ask a question.
In 5–8 seconds, I get a synthesized answer with the source notes cited.
I open the sources to verify.
Done.
The model is not perfect – Phi3:mini will occasionally miss nuance or misread a technical term from a note.
But for a free, offline, CPU-only setup, it is genuinely impressive.
What it is excellent at:
What to watch out for:
The entire stack is free, open-source, and runs completely offline.
Once set up, it requires zero maintenance and adds nothing to your monthly cloud bill.
The popular mental model of AI is “Cloud service that reads your data”.
This does not have to be the case and for personal notes, it arguably should not be.A small, open model running locally on your laptop is good enough for synthesizing your own knowledge base.
It will not beat GPT-4 at coding challenges or creative writing.
But for the task of “help me navigate and synthesize the notes I already wrote”, it does the job well.
Your notes are already doing half the work.
A local AI just helps you finish the other half.
—
**📚 Original Source:**
[How I Supercharged My Personal Knowledge Base with Local AI](https://linuxhandbook.com/blog/personal-knowledge-base-with-local-ai/)
About The Author
