AI
What we learned building an assistant that answers questions from a municipal code
A retrieval-augmented assistant over a published city code, built as an internal exercise. The hard parts were not the model.
As an internal exercise, we built an assistant that answers questions from a published municipal code using retrieval-augmented generation: find the relevant sections, answer only from them, cite them. Here is what we learned, for agencies considering something similar.
Chunking the code is the first real problem
Municipal codes are hierarchical: titles, chapters, sections, subsections. A section often only makes sense with its parent's definitions. Splitting the text into fixed-size pieces lost that context. We ended up chunking by section and prepending the chapter and title headings to each chunk so the model saw where it sat.
Cross-references break naive retrieval
"Subject to the requirements of Section 12.08.040" is everywhere. When a question depended on a referenced section, retrieval by similarity often missed it. We added a pass that follows references in retrieved sections and pulls those in too.
Definitions matter more than you think
Many wrong answers came from the model using a common-sense meaning of a term the code defines differently. We now always retrieve the definitions chapter alongside any answer and instruct the model to apply defined terms.
The model must refuse
When the code does not address a question, the correct answer is "the code does not address this." Getting the model to say that reliably took explicit instruction and testing. Agencies should require this behavior; a confident wrong answer about a permit requirement is worse than no answer.
Evaluation is not optional
We wrote a set of questions with answers checked against the code by hand and scored every change against it. Without that, "it seems better" is the only measure, and it is not one.
Citations change how people use it
Every answer links to the sections it came from. Testers stopped treating the output as an answer and started treating it as a research shortcut, which is the right relationship.
What this means for an agency
The technology works, and the effort is in the content structure, the retrieval design, the refusal behavior, and the test set. Ask any vendor about those four things.