The Complete Guide to Training an AI Agent on Your Knowledge Base

An AI support agent is only as good as the knowledge behind it. Upload a messy PDF dump and you'll get messy answers. Structure your content deliberately, and you'll see resolution rates climb week over week.

This guide covers everything we recommend to Mozo customers training agents on knowledge bases — from content architecture to ongoing maintenance.

How RAG training works (briefly)

Mozo uses retrieval-augmented generation (RAG). When a customer asks a question:

1. The question is converted to a vector embedding
2. Mozo searches your training data for the most relevant chunks
3. Those chunks are sent to the LLM as context
4. The model generates an answer grounded in that context

Your job is to make step 2 reliable — which means clean, chunked, findable content.

See RAG vs Fine-Tuning if you're deciding between approaches.

Content source hierarchy

Not all training sources are equal. Use this priority order:

Rule of thumb: Q&A pairs for precision, crawled articles for coverage, PDFs only when no better format exists.

Writing docs that retrieve well

One topic per article

Don't combine "Billing FAQ" and "API Reference" in one page. Retrieval returns chunks — if your article covers 12 topics, the wrong chunk may surface.

Bad: "Everything about Acme" (4,000 words, 15 topics)

Good: Separate articles for "Refund Policy," "Updating Payment Method," and "Understanding Your Invoice"

Front-load the answer

Put the direct answer in the first paragraph. LLMs weight early content heavily, and chunk boundaries often split mid-article.

Bad:

Refund Policy

At Acme, we believe in customer satisfaction. Our journey began in 2019 when... (300 words later) ...you can request a refund within 30 days.

Good:

Refund Policy

You can request a full refund within 30 days of purchase. Contact support@acme.com with your order number. Refunds process in 5–7 business days.

Use descriptive headings

Headings become chunk metadata. "FAQ" tells retrieval nothing. "How to cancel your subscription" matches customer language.

Include exact phrases customers use

If customers ask "how do I cancel," your doc should contain that phrase — not just "subscription termination procedures."

Q&A pairs: your secret weapon

Q&A pairs in Mozo bypass fuzzy retrieval for close matches. Use them for:

• Pricing and plan comparisons
• Return and refund policies
• SLA commitments
• Security and compliance statements
• Anything where exact wording matters legally

Example pair:

Q: Can I get a refund after 30 days? A: Refunds are available within 30 days of purchase. After 30 days, we can offer account credit at our discretion — email support@acme.com with your order details.

Write 3–5 variant phrasings in the question field if Mozo supports multi-question mapping, or create separate pairs for common variants.

Website crawling best practices

Scope your crawl

Don't crawl your entire domain. Target docs paths:

Include: /docs/, /help/, /support/
Exclude: /blog/, /press/, /careers/, /legal/privacy

Mozo lets you set include/exclude patterns per crawl job. See Training Data docs.

Respect page structure

Crawlers extract main content, but pages heavy on navigation, footers, and sidebar widgets add noise. Clean HTML templates produce cleaner chunks.

Enable auto-retrain (Pro+)

Docs drift. Product updates, pricing changes, and new features invalidate old answers. Weekly auto-retrain on Pro and Business plans re-crawls configured URLs and updates embeddings without manual intervention.

File upload tips

PDFs

• Export from source (Notion, Google Docs) rather than scanning printed docs
• Ensure text is selectable — image-only PDFs require OCR and lose quality
• Split large PDFs (>100 pages) into topic-specific files

CSV

Useful for structured Q&A bulk import. Format: question in column A, answer in column B.

DOCX / TXT

Preferred over PDF when available. Cleaner text extraction, fewer formatting artifacts.

Chunking and retrieval: what you control

Mozo handles chunking automatically, but your content structure influences chunk quality:

• Shorter sections (200–400 words) retrieve more precisely than walls of text
• Tables should have headers; retrieval may split rows across chunks — consider duplicating key table data in prose
• Code blocks are included in chunks; add a prose summary above complex code samples

Testing retrieval quality

After processing training data, test in the Playground with questions designed to stress retrieval:

1. Direct match: A question copied verbatim from a doc heading
2. Paraphrase: Same question in casual customer language
3. Adjacent topic: Question near but not exactly matching a doc (tests precision)
4. Out of scope: Question you deliberately didn't train on (tests hallucination guardrails)
5. Multi-hop: Question requiring info from two articles (tests synthesis)

Log failures. Each one maps to a fix: new Q&A pair, doc rewrite, or exclusion of a noisy source.

Keeping your agent accurate over time

Weekly (5 minutes)

• Scan conversation logs for "I don't know" responses
• Check lead capture volume for escalation spikes

Monthly (30 minutes)

• Review top 10 failed questions → add Q&A pairs or update docs
• Verify crawled page count matches expected docs count
• Remove deprecated sources (old product versions, archived pricing)

On every product launch

• Publish docs before announcing the feature
• Add Q&A pairs for anticipated launch-day questions
• Run playground tests on launch-specific queries

Common training mistakes

Measuring training ROI

Track these metrics monthly:

• Resolution rate — % of conversations without escalation
• Retrieval confidence — conversations where agent cited no relevant context (add training)
• Repeat question rate — same customer asking the same thing (answer wasn't clear)
• Time to update — days between doc change and agent reflecting it

Next steps

• Build your first agent step-by-step
• RAG vs fine-tuning deep dive
• Start training on Mozo — free plan includes file upload and website crawl

Good training isn't a one-time upload. It's an ongoing practice — and the compounding returns are what separate agents that demo well from agents that actually deflect tickets.