RAG Chat

RAG chat lets you ask natural-language questions and get answers grounded in your own knowledge base. Every answer cites the articles it drew from.

What RAG means

RAG stands for Retrieval-Augmented Generation.

Instead of asking a language model to answer from its training data (which may be outdated, wrong, or hallucinated), RAG:

1. Retrieves relevant passages from your knowledge base
2. Augments the prompt with those passages
3. Generates an answer grounded in what was retrieved

The result: answers that cite your own content, not the model's guesses.

Why this matters

Standard LLMs hallucinate. They confidently state things that aren't true. In security work, a hallucinated answer could lead to a wrong command, a wrong technique, or a wrong recommendation to a client.

RAG reduces this dramatically. The model can only draw on what's in your KB. If the answer isn't there, it says so.

Opening RAG chat

Go to AI in the sidebar.

The chat interface has:

• A message history pane
• An input box at the bottom
• A sources panel that updates per message

Type a question and press Enter. The response streams in token by token.

How a query flows

1. Your question is embedded using nomic-embed-text
2. Semantic search finds the top 5 relevant KB passages
3. Those passages are injected into the prompt as context
4. Llama 3.2 generates an answer based on the context
5. The answer streams back with inline source citations
6. Sources appear in the right panel

Everything happens locally. No query, no passage, no answer leaves your machine.

Example queries

Good questions for RAG chat:

• "What's the difference between Kerberoasting and AS-REP roasting?"
• "How do I check for unconstrained delegation in a domain?"
• "What evidence should I capture for a SQL injection finding?"
• "Walk me through the PTES exploitation phase."
• "What are the OWASP API Top 10 items?"

The model answers using your playbooks, cheat-sheets, and articles as source material.

What happens when the KB doesn't have the answer

The model says so. Example:

"I don't have information about that in the knowledge base. The articles I retrieved don't cover this topic. You may want to add an article or search the web."

This is the correct behavior. Better to admit missing knowledge than to hallucinate.

Source citations

Every claim in the answer is linked to a source. Click a citation to jump to the article.

If an answer has no citations, it's because the model is summarizing general knowledge — be cautious with those responses.

Streaming responses

Responses stream in real time. Tokens appear as they're generated, not all at once.

This means:

• You see the answer forming
• You can stop generation mid-stream if the answer is going the wrong way
• The interface stays responsive

Stopping generation

While a response is streaming, the send button becomes a stop button. Click it to halt generation. The partial answer remains in the history.

Multi-turn conversations

The chat remembers context within a single conversation. Follow-up questions can reference earlier answers:

• You: "What's Kerberoasting?"
• AI: (explains)
• You: "How would I detect it?"
• AI: (answers using the previous context)

Context is limited to the model's window (2048 tokens by default). Long conversations are truncated from the oldest turns.

Clearing the conversation

Click New conversation at the top of the chat to start fresh. The old conversation is cleared from the view but remains in the audit log.

The system prompt

RAG chat uses a system prompt that instructs the model to:

• Only use the retrieved context
• Say when the context doesn't cover something
• Cite sources for every factual claim
• Be concise
• Avoid speculation

You can see the prompt in Settings → AI → System prompt. Editing it is possible if you want to tune the behavior.

Temperature and sampling

By default, temperature is set low (0.2) for factual answers. This makes the model more deterministic and less creative.

You can adjust in Settings → AI:

• Low (0.1–0.3) — factual, consistent
• Medium (0.5–0.7) — balanced
• High (0.8–1.0) — creative, variable

For security work, keep it low. Save higher temperatures for brainstorming.

Context size

The context window determines how much text the model can reason over at once. Larger windows mean more retrieved passages, but slower inference and more memory.

Default: 2048 tokens.

Adjust in .env.local:

OLLAMA_NUM_CTX=4096

Or in Settings → AI. Restart Ollama after changing.

For an 8 GB machine, 2048 is comfortable. For 16 GB, 4096 works.

Retrieval parameters

By default, RAG chat retrieves the top 5 passages from semantic search. Adjustable in Settings → AI:

• Top 3 — faster, less context
• Top 5 — balanced (default)
• Top 10 — more context, slower

More passages give the model more to work with but slow generation proportionally.

Adding to the KB

The best way to improve RAG answers is to add more content to the KB. Every article you write becomes retrievable.

See Using the knowledge base for how to write articles.

Privacy

RAG chat is fully local. The only network activity is between ATHELOS and your local Ollama instance on port 11434. Nothing leaves your machine.

You can verify this: disconnect from the internet and use RAG chat. It works identically.

Performance

Response generation is limited by your hardware.

On an 8 GB laptop with 3B model:

• First token: 500ms–2s
• Generation: 10–30 tokens per second
• Typical answer: 3–10 seconds

On a machine with a GPU:

• First token: 100–500ms
• Generation: 30–60 tokens per second
• Typical answer: 1–4 seconds

The biggest variable is the model size. 3B is fast; 7B is slower but better; 13B is much slower.

When chat isn't available

RAG chat requires:

• Ollama running
• The chat model pulled (llama3.2:3b)
• The embedding model pulled (nomic-embed-text)
• At least one KB article

If any is missing, the chat panel shows a setup message with what to install.

Common issues

Responses are slow

Reduce context size (OLLAMA_NUM_CTX=1024) or switch to a smaller model.

Responses are wrong

Check the sources. If the retrieved passages don't actually answer the question, the model is doing its best with bad context. Improve the KB.

Responses are cut off

The model is hitting the context limit. Reduce the retrieval count or increase OLLAMA_NUM_CTX.

"Offline" indicator

Ollama isn't running or OLLAMA_HOST is wrong. Check with curl http://localhost:11434/api/tags.

Where to go next

• Semantic search — the retrieval layer
• Assistive actions — AI inside the editor
• Overview — local-only AI and hardware