Let's start with some simple tasks like web searching, file reading and utilizing the CLI. They are classified as simple tasks because these only involve tool calling, which I'll explain shortly.

Without a doubt, a substantial amount of source code of the agent is generated by AI. Especially the LLM tools, which are mostly boilerplate code with some slight changes to do very basic tasks. As I stumbled upon the idea of LLM skills, a natural question followed: why not let the agent build itself?

Therefore, I defined create_tool skill with a common SKILL.md skill file and a tool_template.py template file. Spoiler: it worked.

Browser automation is essential to make a personal agent more powerful since a large proportion of daily workflow (at least of mine) happens in the browser. Now that the agent is equipped with the skill to create tools for itself, I asked it to build one to control the chrome browser via Playwright. Controlling the browser is obviously much more complicated than just sending macos notifications. In the end, the tool was mostly functional from the start, though I had to manually fix a few bugs.

The architecture of the agent is fairly simple. It consists of the python application, Ollama, a PGvector container and a Searxng container for web search. The communication between the python application and Ollama is nothing more than REST API calls, you can refer to their comprehensive documentation for that.

Raw POST requests to Ollama alone don't give the agent any real-world awareness. It doesn't know the current time, has no idea about my location, let alone search on the internet. However, it's quite easy to mitigate this by defining tools to enable the LLM to interact with the real world.

Ollama and some other popular AI services like OpenAI and Anthropic provide tool calling functionality in a similar way. All you need is to include your tool definition in the POST request — the LLM handles the rest. When and which tool to invoke is purely dependent on what's in the LLM's response.

And to return the tool calling result to the LLM, you only need to append the result to the message list and POST it to Ollama again. So the complete flow will be like the following:

With that, the tool-calling loop is complete — but tools alone aren't enough for complex tasks. What do skills do? Think about a complex task like creating an LLM tool for the agent. It will involve orchestrating multiple tools to achieve the best result. Beyond that, file format, location, and naming conventions all need to be clearly defined for the LLM to follow. Therefore, this is where skills shine. A skill usually only consists of a markdown file containing the recipe of how to tackle certain tasks. For example, my SKILL.md for the create LLM tool skill can be summarized as follows:

You might be wondering: "Wait, how do we feed skills to the agent?". Indeed, there's no entrypoint in the API payload where you can easily supply the skills to the LLM like tools.

As a result, we need to literally tell the LLM that you have those skills. And the system prompt is the perfect spot. According to Ollama's documentation, some behavioral instructions can be supplied to hint the LLM about how to generate ideal responses by inserting a message with the role of system in the dialogue. In there, we can compose a paragraph saying "you have the following skills, where they are located, and what each one does.". Thus, the message flow between the agent and the LLM would be like the following:

Besides the skills, we can also put some brief but useful information into the system prompt to speed up the response generation. For example, the current time with time zone, the city you live in and the underlying OS the agent runs on. Since they add trivial token overhead, we can naturally include them up-front so that LLMs don't have to ask the agent for them through tool calling.

At first, we can start with the web search tool as web pages usually contain so much irrelevant information. Take Stackoverflow as an example, OP's problem and the other people's replies are all we want the LLM to know. Obviously, including the navigator on the left part of the page or the linked posts on the right would be a huge waste of tokens. Luckily, there is an existing library specifically for this job — trafilatura .

With the help of trafilatura, only a few lines of code are needed to extract the most relevant information from a web page.

Ideally we wish the LLM to remember all the skills we provide and all the messages in the conversation. It won't be a problem at the beginning when there are only a few skills and a few dozens of messages. But it's worth planning ahead. We need to be smart and selectively include relevant skills and information when the total amount of them is beyond control. This is the moment we need to consider RAG.

RAG is a technique we can't avoid when discussing AI agents. The foundation of RAG is semantic search. It enables us to accurately search for relevant information to enhance LLM response generation. Compared to old-school lexical search, semantic search focuses on semantic relevancy instead of lexicographical similarity.

The most common way to conduct semantic search is to generate embeddings — high-dimensional vectors — and compare their cosine similarity. Think about the skills once again, when users send a message like "How can I make a tool for car fixing?", create_tool would clearly be irrelevant here. Each skill has a short description, and I use its embedding to decide whether it's relevant enough to include in the system prompt.

Although the similarity score is better when a user asks "Make yourself a tool for browser automation." than to "How can I make a tool for car fixing?", the score difference is not significant enough. That makes it difficult to set a score threshold so that we can eliminate the noise. However, when working with sentence-transformers, there are always surprises. It provides a Cross Encoder to achieve higher accuracy prediction.

Unlike a standard embedding model that encodes each sentence independently, a Cross Encoder takes two sentences simultaneously as input and outputs a relevance score directly. There is no embedding or cosine similarity involved — the model reads both texts together and reasons about their relationship as a whole. This makes it significantly more accurate than similarity-based ranking, at the cost of being slower since you cannot pre-compute and cache the vectors upfront. For skill selection, where precision matters more than speed, it is the right trade-off.

Now let's look back on the previous section and push the web search efficiency to its limit. As depicted in Figure 9 , there is still quite a lot of garbage text in the trafilatura extraction result. The approach is to first split the text into several text chunks, then exclude low-scoring chunks and return the remaining chunks reranked by a Cross Encoder.

This two-stage pipeline — filtering by embedding similarity, then reranking with a Cross Encoder — is a common pattern in production RAG systems, balancing speed and accuracy.