Text analytics on 311 comments, complaints, and survey responses

Reference article · 5 min read

Agencies collect more free text than anyone has time to read. A 311 system takes in descriptions typed by residents, a satisfaction survey ends with "anything else you want us to know," a proposed ordinance draws written comment, and inspectors leave notes on every visit. Each field exists because a dropdown could not capture what mattered. Together they form a body of text that gets summarized by whoever is assigned, or ignored.

Text analytics groups that text into recurring themes, counts how often each appears, follows it over time, and connects it to a place. Done carefully, it gives a program manager a defensible picture of what residents are writing in about. Done carelessly, it produces a clean chart no one can trace back to the comments underneath, which is worse than no chart.

Know what kind of text you have

The differences matter more than the volume.

• 311 and service request text. Short, often typed on a phone, already attached to a category someone picked at intake. The free text is usually where that category turns out to be wrong.
• Survey open ends. Narrow in topic, but written by a self-selected group. They tell you what respondents care about, not what the population thinks.
• Public comment. Longer and more formal, and sometimes an organized campaign in which many submissions share wording. Counting near-identical text as separate themes distorts everything downstream.
• Inspector and field notes. Written for other staff, full of abbreviations, addresses, and names. These carry the highest chance of personal information you never intended to analyze.

Before any analysis, decide which fields are in scope, how far back you go, and what gets stripped, so the analyzed copy does not become a liability of its own.

How grouping into themes works

Two approaches, and most agencies need both. The first is a fixed scheme. You define the categories you already report on, assign every comment to one or more, and send the rest to an "other" pile you review. It is auditable and stable across quarters, which is what a recurring report requires.

The second is discovery. The system clusters comments by similarity and proposes labels. This finds what your scheme missed: a new fee confusing people, or one intersection generating complaints filed under four categories. It is useful for exploration and unreliable as a reporting backbone, because clusters shift when the data shifts.

Run discovery to learn, fold the real findings into the fixed scheme, then track that. Keep source comments attached to every theme so any number in a report can be opened and read.

Time and place

Themes are most useful as a trend. Plot each by week or month and the questions sharpen: did complaints on a route rise after the schedule change, did they fall after the fix, is the pattern seasonal. Normalize by total volume, since a theme can grow as a share while contacts overall fall.

Location is the other axis. Most service requests carry an address, so themes roll up to council district or service area. That invites the most common misreading in this work: reporting volume reflects who reports, which tracks with familiarity, language, tenure, and trust in the agency. A quiet area is not necessarily a satisfied one. Pair contact data with inspection findings before concluding anything about where problems are.

The pitfalls worth naming

• Invented themes. Tools that generate labels can produce a plausible theme no comment supports, or a summary quoting a sentence nobody wrote. Require every theme to link to its records, and read a sample.
• Volume read as severity. Many parking complaints and a few about a failing retaining wall do not rank that way. Keep severity a separate, human judgment.
• Non-English and mixed-language text. Many tools perform worse outside English, and comments that switch languages mid-sentence confuse them further. Ask how each language is handled, check whether non-English comments pile up in "other," and have bilingual staff review a sample.
• Duplicates and campaigns. Deduplicate near-identical submissions, and report a campaign as a campaign rather than as a spontaneous theme.
• Drift. Changing the scheme or the model mid-year breaks comparability. Version the scheme, note the version on the chart, and reprocess history when it changes.

Records and what reaches the council packet

Comments submitted to an agency are generally public records in most states, with exemptions that vary. Derived artifacts, meaning theme assignments, scores, summaries, and the configuration behind them, are often records themselves and subject to your retention schedule. Text that was low-risk while scattered can become sensitive once aggregated and tied to addresses. These rules differ by state and change over time, so confirm scope and retention with your records officer and counsel rather than assuming. If a summary goes into a council packet, the method should be describable in a paragraph and the counts reproducible by someone else.

How to start, and what to ask

Start with one source, one quarter, and one question a program owner already has. Ask any vendor, and any internal team:

1. Show me a theme, then show me the individual records behind it.
2. What happens to comments in the languages our residents write in, and how was that measured?
3. Is our text used to train anything outside our tenant, and where is it stored?
4. Can we export themes, assignments, and history in an open format if we leave?

In the contract, require that the agency owns the text, the labels, and the scheme; that model or scheme changes are disclosed before they take effect; that an audit log records what was processed and when; and that full export is available on demand at no added charge.

HQT works with state and local agencies on data and analytics systems, including unstructured text from resident-facing services.

Frequently asked questions