Reference checks that produce real information about a vendor

Reference article · 5 min read

A vendor's reference list is a curated document. The agencies on it were chosen because someone at the vendor is confident about what they will say, and they usually know the call is coming. That does not make the list useless. It makes it a starting point rather than a verification step. The information you need is generally one or two questions past the first answer, and some of it comes from agencies the vendor did not name.

Reference checking also carries procedural weight. Depending on the jurisdiction, references may feed a scored evaluation factor, a responsibility determination, or both, and the two are handled differently. Rules vary by state and by local code, so confirm with your procurement counsel how references may be used in your solicitation, whether you may contact agencies that were not listed, and what you must disclose to offerors. Whatever the framework, the notes you take can end up in the file that defends the award. Write them as though a disappointed competitor will read them.

Find the references the vendor did not offer

The listed references are the floor, not the sample.

• Public award records. Search state and local award notices, cooperative purchasing portals, and neighboring agencies for the product name. The instructive projects are often the ones left off the list.
• Governing body minutes. Council, board, and commission agendas often include contract amendments and change orders for software projects, described in plain language and dated. They tell you where to call.
• Peer networks. State CIO and procurement associations, user groups, and regional consortia carry knowledge that never appears in a proposal. One question on a listserv often surfaces a candid conversation.
• Non-renewals. Ask the vendor for public sector clients who did not renew or who ended the engagement early, and for permission to contact them. A refusal is itself information.

Ask about the named project team, not the company

Implementations succeed or fail on individuals. The organization that delivered well elsewhere may not be the people proposed to you, and the reference may be describing staff who have since left.

• Name matching. Compare the key personnel named in the proposal against who actually did the work at the reference site. Ask who their project manager and lead analyst were, by name, and whether those people are still with the vendor.
• Subcontractors. Ask who was an employee of the vendor and who was a subcontractor, and whether that was clear at the time.
• Turnover. Ask how many project managers they went through, what happened at each handoff, and how long the gaps lasted.
• Escalation. Ask who they called when the project manager could not resolve something, and whether calling that person changed anything.

Questions that produce specific answers

Ask an open question, then ask for the example behind the answer. Silence is useful here: after a general reply, wait rather than moving on.

1. What was the go-live date in the original contract, and what was the actual date? What accounted for the difference?
2. What work did you end up doing yourselves that you expected the vendor to do?
3. What were the change orders for, who proposed them, and which ones should have been in the base scope?
4. Which requirement in your solicitation turned out to mean something different in the product?
5. How many passes did data conversion take, and who owned data cleanup?
6. What is still not working the way you expected?
7. What would you write differently if you ran the procurement again?
8. Would your agency award to this vendor again, for this product, knowing what you know now?

The last question is the one to press. A reference who praises the vendor at length and then hesitates on that question has told you something.

Ask about support in the present tense

Implementation memories fade. Current service is observable, and it is what you will live with for years.

• How does a ticket get opened, and does a person respond or a queue?
• How do the response times written into the contract compare with what the agency experiences?
• Have service credits or other remedies ever been claimed, and what happened?
• How are upgrades scheduled, who tests them, and has an upgrade broken configuration or an integration?
• What does the agency pay now compared with the original award, and what drove the change?

Record answers so the file supports the award

Reference information helps only if it survives a protest or an internal challenge.

• Use one question set. Ask every offeror's references the same core questions, in the same order, with the same agency staff on the call.
• Two staff per call. One asks, one writes. Capture near verbatim answers with the date, the reference's name, title, and agency, and how you identified them.
• Separate fact from impression. A missed date is a fact. "They seemed disorganized" is an impression, and should be labeled as one.
• Do not promise confidentiality you cannot deliver. Your notes may be subject to public records law. Say so at the start of the call.
• Tie findings to stated criteria. Score only what the solicitation says will be scored. If a reference raises something that goes to responsibility rather than evaluation, route it through that process and involve counsel before it affects an award.
• Allow a response where your rules require it. Some jurisdictions require giving an offeror an opportunity to address adverse reference information. Confirm what applies to you before relying on a finding.

HQT is a technology and software company working with state and local government. Questions about named project staffing, change order history, and support response are fair to put to any vendor, including us.

Frequently asked questions