✦

Project 2 // _support-agent

RAG Support Agent

A support agent that answers only from your docs — and stays quiet when they don't cover the question.

client

personal product — open source

timeline

Jul 2026 · live

role

solo build · full-stack + AI

scope

RAG architectureingestion pipelineretrieval + groundingeval harnessembeddable widget

## the-problem

A chatbot over your docs is a weekend demo. One you'd put in front of customers isn't, and the gap is what it does when it doesn't know — a naive RAG app answers anyway, fluently and wrongly, because a model's default is plausible text, not an admitted gap.

That single failure mode is what keeps these off real support pages: an invented refund window creates the ticket it was meant to deflect. And retrieved text is attacker-controlled — an "ignore previous instructions" buried in a crawled page is a live injection vector.

## the-approach

▸made refusal control flow, not a prompt instruction — below threshold, the request returns a fixed "I don't know" without reaching the model
▸grounded answers in numbered <source> chunks with required inline [n] citations, so every claim traces to its passage
▸treated retrieved text as untrusted data — sources delimited, instructions inside them ignored
▸chunked on structure, not character count: headings and paragraphs, heading path carried along
▸768-dim embeddings in Postgres (Matryoshka-truncated to fit HNSW); unchanged sources hash-skipped on re-ingest
▸built a 20-case eval harness scoring retrieval, correctness and refusal instead of eyeballing it

// the gate — retrieval decides whether the model runs at all

const result = await retrieve(question);
 
// Telemetry first, so the UI can show the decision even on the
// path where no answer is ever generated.
writer.write({ type: 'data-retrieval', data: {
  scanned: result.chunks.length,
  kept: result.relevant.length,
  topScore: result.topScore,
  threshold,
} });
 
// Honest "I don't know": nothing cleared the threshold, so a fixed
// string goes back. No LLM call — nothing to hallucinate with.
if (!result.hasRelevant) {
  writeStaticText(writer, FALLBACK_MESSAGE);
  return;
}
 
// Only now does the model see anything, and only the chunks that
// cleared the threshold, wrapped as untrusted <source> data.
const llm = streamText({
  model: languageModel(),
  instructions: buildSystemPrompt(),
  messages: [...history, buildUserPrompt(question, result.relevant)],
});

## the-architecture

// the query path — and the gate that can end it early

client

chat UI

Next.js · streaming

inline [n] citations
live retrieval read-out

client

embedded widget

one <script> tag

iframe, no host deps
same backend

ai / llm

embed the question

768-dim

Matryoshka-truncated
fits pgvector's HNSW limit

data

vector search

Postgres + pgvector

top-k cosine, HNSW
chunks + heading paths

backend

ingestion

files · crawled URLs

structure-aware chunking
unchanged sources skipped

backend

the relevance gate

similarity threshold

nothing clears it → stop
decision is streamed to the UI

ai / llm

grounded answer

cited, streamed

sources as untrusted data
tokens + cost logged

backend

"I don't know"

no LLM call

fixed string, not generated
nothing to hallucinate with

clientai / llmdatabackend

## the-results

LLM calls when it can't answer

eval cases scored per run

1 line

to embed on any site

✓the refusal path is structural — on an out-of-scope question the generation step is never reached, a property of the code rather than a promise in a prompt
✓every answer ships with its evidence: inline [n] citations plus a panel showing each chunk's similarity and whether it cleared the gate
✓retrieval is measured — 20 eval cases per run, plus 33 tests over chunking, token budgets, HTML extraction and the crawler

nextreacttypescriptaiscraping

// want something like this built for you?

start-a-project