Resident survey data: sampling, weighting, and margin of error

Reference article · 5 min read

A resident satisfaction or community needs survey is often the only systematic evidence an agency has about what people outside the building think. It gets cited in budget hearings, strategic plans, grant applications, and council presentations, and it tends to get cited as though the percentages were exact. They are not exact. Every estimate carries uncertainty, and the size of that uncertainty is set by design decisions made long before a vendor delivers a report.

The items below determine whether your numbers hold up when someone challenges them. They are worth working through twice: once while writing the scope, so you can state the precision you need, and again when results arrive, so you know what you can responsibly say in public.

The sample frame decides who could have been reached

The frame is the list from which people were selected. Coverage error is the gap between that list and the population you mean to describe, and it is the error weighting is least able to repair.

• Address-based frames. Sampling from a comprehensive address list covers households regardless of phone or internet use, which is why it is common for general population work.
• Administrative lists. Voter files, utility accounts, permit records, and program rosters are convenient and systematically incomplete. A utility billing list, for example, tends to underrepresent renters whose utilities are bundled.
• Open web links. A link posted on your site or social accounts produces a self-selected group, not a sample. It can be useful for gathering input. It does not support population estimates.
• Definition of resident. Decide up front how you treat renters, non-English speakers, people in group quarters, and people without a stable mailing address. Whatever you decide belongs in the methodology statement.

Mode changes who answers and what they say

Mail, phone, online, text-to-web, and in-person intercept each reach different people and produce different answers to the same wording. Mixed-mode designs are used to offset these tendencies, but they introduce their own complication: differences between waves may reflect a change in mode rather than a change in opinion.

Two mode decisions have legal and practical weight. Public agencies generally carry obligations around meaningful language access and around accessible electronic content, so translation and accessibility of the instrument are design requirements rather than extras. Specific obligations vary by jurisdiction, funding source, and program, so confirm the applicable requirements with your counsel or program office.

Response rate matters less than nonresponse bias

Response rate is the share of eligible sampled units that completed the survey. There are several accepted formulas that treat unknown eligibility differently, and AAPOR's Standard Definitions is the usual reference. Ask which definition the vendor used, and ask for the raw disposition counts behind it.

A low response rate is not automatically disqualifying. Bias arises only when nonrespondents differ from respondents on the thing being measured. Useful checks include comparing the respondent profile to known population characteristics, comparing early responders to those who answered only after follow-up, and running a targeted follow-up on a subsample of nonrespondents.

Weighting adjusts the sample toward the population

Weighting usually has two parts. Design weights account for unequal probabilities of selection, such as oversampling a neighborhood. Post-stratification or raking then aligns the sample to population benchmarks, commonly drawn from Census Bureau products such as the American Community Survey, on variables like age, sex, race and ethnicity, housing tenure, education, and geography.

Two cautions. First, weighting redistributes the information you collected. It cannot create information about groups you never reached. Second, weighting increases variance. The design effect quantifies that increase, and dividing the completed interviews by the design effect gives the effective sample size, which is what your margin of error should be based on. Ask for both numbers.

Margin of error, and the subgroup problem

For a simple random sample, the 95 percent margin of error for a proportion near 50 percent is roughly plus or minus three percentage points at 1,000 completes, about four and a half points at 500, and about ten points at 100. Cutting the margin in half takes roughly four times the sample.

• The margin belongs to the estimate, not the survey. A citywide margin says nothing about a district-level or Spanish-speaking-respondent estimate computed from a fraction of the completes.
• Comparisons need a wider margin. The uncertainty around a difference between two subgroups, or between this year and last, is larger than the margin on either figure alone.
• Adjust for the design. Multiply the simple random sample margin by the square root of the design effect.
• Sampling error is only one error. The margin says nothing about coverage gaps, nonresponse, question wording, order effects, or translation quality.
• Opt-in samples. Margin of error is not strictly defined for nonprobability samples. Some vendors report a modeled credibility interval instead, which rests on assumptions that should be stated.

What to put in the scope and require on delivery

State the target population, the subgroups you must report separately, and the precision you need at the subgroup level. That single requirement drives sample size, cost, and oversampling more than any other. Then require as deliverables: the questionnaire as fielded in every language, full disposition counts and the response rate definition used, the weighting method with named benchmark sources and the date of those benchmarks, the design effect and effective sample size, margins of error reported per subgroup, a de-identified respondent-level data file including the weight variable, and clear agency ownership of the data.

Ask the vendor how they will detect nonresponse bias, what they will do if a subgroup falls short of the required precision, and how they will keep the instrument comparable if you field it again. Before publishing any figure, confirm it is the weighted estimate, that its own margin is stated beside it, and that any comparison you are drawing survives that margin. HQT works with state and local agencies on survey data pipelines, analysis, and reporting.

Frequently asked questions