Form design that produces clean data: pick lists, validation, free text

Reference article · 6 min read

A year after a new online form goes live, someone asks a simple question: how many of these requests involved a rental property, and how long did they take to close? Whether that question takes a minute or a summer of spreadsheet work was decided long before the question was asked. It was decided when someone chose between a four-value pick list and a comment box.

Form design is data design. Every field on an intake, permit, license, or inspection form is a decision about what the agency will be able to count later, what it will have to read by hand, and what it will never recover. The good news is that the choices are finite and most of them can be made correctly the first time.

Start from the questions the data has to answer

Before laying out a single field, write down the handful of questions the form has to answer: the monthly report to the department head, the annual number in the budget book, the measure a grant requires, the question a council member asks every spring. Each of those questions implies a field with a defined set of values.

Two other decisions belong at the front:

Separate operational fields from reporting fields. Operational fields let staff do the work. Reporting fields let the agency count the work. Reporting fields are the ones quietly cut when a form feels too long, and they are the ones nobody can reconstruct afterward.
Name the unit of analysis. One row per application, per parcel, per site visit, or per violation found? Forms that mix units produce counts that cannot be reconciled. If one submission can contain several inspected items, it needs a repeating section, not five numbered text boxes.

Controlled vocabularies, and the "other" problem

A pick list turns prose into something countable. To hold up over years:

Store a code, display a label. If the stored value is a stable code, the label can be reworded without breaking historical reporting.
Retire values, never delete them. Mark an option inactive so it disappears from the form but still resolves on old records.
Make options mutually exclusive and reasonably complete. Overlapping options push staff to guess, and two people guess differently.
Match the control to reality. If a case can genuinely have two causes, a single-select forces a false choice and the resulting counts understate everything.
Handle "other" deliberately. Pair it with a required short text box, then review those entries on a schedule and promote the recurring ones into the list. A high share of "other" is not a data problem to clean up later. It means the list is wrong.

Required, optional, and conditional logic

Requiring a field that the person filling it out cannot possibly know does not produce data. It produces "N/A", "unknown", "999", and single spaces. Offer explicit values instead: not applicable, unknown at time of filing, declined to state. Absence recorded on purpose is analyzable. Absence disguised as a value is not.

Conditional logic keeps the form short and keeps "required" meaningful. Ask about septic systems only when the applicant indicates septic. It also helps to require fields by stage rather than all at once: a smaller set to submit, a fuller set before a case can be closed.

Formats that quietly break reporting

Addresses. Capture structured components rather than one free line, and validate against an authoritative reference such as your jurisdiction's address points or centerline file. Store the geocode result and which source produced it. Addresses typed as prose cannot be joined to parcels, districts, or service areas.
Dates. Use a date picker, and distinguish the date an event occurred from the date it was entered. Free text dates arrive in every format a keyboard allows.
Numbers and money. Use numeric field types with the unit stated in the label. Any field that might receive "about 2,000 sq ft" should not be a text box.
Names and contact details. Separate name parts, allow generous length and single names, and validate the shape of email and phone without rejecting legitimate variations.

Where free text still belongs

Narrative fields are valuable and should stay: the complainant's description, the inspector's observations, conditions of approval. The working rule is that free text supplements a coded field rather than replacing it. Code the category, then let the narrative explain the specifics.

Tell staff and the public what a narrative field is used for. Content submitted to an agency may be subject to public records disclosure and to redaction requirements, and those rules vary by state and by record type. Confirm the treatment of a given form with your records officer or counsel, and avoid inviting sensitive identifiers into open text.

Validation, accessibility, and testing before launch

Validate at the point of entry with plain inline messages that say what to do, not just what failed. Use hard stops sparingly and soft warnings often, since a hard stop on a field someone cannot satisfy leads to invented answers. Put help text next to the field rather than in a separate guide.

Test with staff and with real applicants, using your most confusing past cases as scripts. Meet accessibility requirements for public-facing digital services, including programmatically associated labels and error messages that assistive technology can read. If the form is offered in more than one language, translated labels must map to the same stored codes.

After launch, monitor field-level quality: the share of "other" and "unknown", the distribution of values, and where people abandon the form.

What to ask vendors, and how to start

Ask during evaluation, and reflect the answers in the contract:

Can agency staff add, reword, and retire pick list values without a change order, and is each change versioned?
What happens to existing records when a value is retired?
Are stored codes separate from display labels in the exported data?
Is conditional logic configurable by staff, or only by the vendor?
What address reference data is used, and can it point at our own address points?
Are audit fields captured, including who entered a value and what it was before?
Can we export all submitted fields, including narrative, in an open format on demand and at termination, with a current data dictionary as a deliverable?

To start, take one form, list the five questions it must answer, map those questions to fields, and fix the three biggest gaps. Repeat on the next form. HQT works with state and local agencies on data modeling, form and workflow design, and reporting built on the fields those forms capture.

Frequently asked questions

How do we fix a form that already collected a year of free text we cannot count?

Do not try to retro-code everything at once. Read a sample of the free text responses and use the patterns you find to build the pick list you should have had, then add that coded field to the live form so the counting starts now. If a specific report requires historical numbers, code only the subset of records that report covers and document that the earlier period was derived differently.

Should every field on a government form be required?

No. Requiring a field that the person cannot answer produces placeholder text, zeros, and single spaces that look like data but are not. A better pattern is to require fewer fields at submission, use conditional logic so people only see fields that apply to them, and offer explicit values such as not applicable or unknown so that a missing answer is recorded on purpose.

How many options should a pick list have, and what do we do about Other?

Keep the list short enough that a person can scan it, use grouping or type-ahead when the list is genuinely long, and make sure the options do not overlap. Always pair Other with a short required text box, then review those entries on a regular schedule. If a large share of submissions choose Other, treat that as evidence the list is missing real categories rather than as cleanup work.

Is it better to validate addresses when they are entered or clean them up later?

Validate at entry. Structured address components checked against your jurisdiction's address points or centerline data let you join submissions to parcels, districts, and service areas immediately. Cleaning addresses later is slower, never fully accurate, and has to be repeated every time new records arrive.

Who should own the list of values on a form, the program office or IT?

The program office should own the meaning of the values because it owns the reporting, and IT or the vendor should implement changes through a documented process that versions each change. The key contract and configuration question is whether staff can add or retire values without a change order, and whether retiring a value leaves historical records intact.