Building a QA program your agents will thank you for

Decide what the program is for before you design the form

QA can do three jobs. It can protect the business by catching compliance and accuracy failures. It can improve agents by giving them specific feedback on real contacts. And it can inform the operation by showing which policies, tools, and knowledge articles make contacts harder than they need to be. Most programs claim all three and are designed for none.

The purpose dictates the design. A program built mainly for protection needs coverage of the risky contact types and binary criteria. A program built for coaching needs enough evaluations per agent to show a pattern, with criteria that describe behaviors an agent can practice. A program built for insight needs evaluators to record why the contact happened and what got in the way, regardless of how the agent performed.

Write down the ranking, because evaluator hours are finite and the three jobs compete for them. Then write down what QA is not for. If scores feed straight into discipline or pay with no coaching step in between, agents will treat every evaluation as a threat, and they will be right to. QA results can inform performance management, but the path should run through coaching first, and agents should know how the two connect.

Build a scorecard that measures what the customer experienced

The conventional scorecard is a long list of items, each worth a few points, summed to a hundred. It is easy to build and it fails in a predictable way. Because every line carries similar weight, using the customer's name counts about as much as solving the problem. An agent can earn a high score on a contact that failed the customer completely. Evaluators tire halfway down the form. And agents learn to perform the checklist, which is how you get calls that sound scripted and score well.

A better structure separates the things you are judging instead of blending them into one total.

• Critical items — a short list of pass or fail criteria where a miss means the contact failed regardless of everything else: identity verification, required disclosures, materially wrong information, abusive conduct. Report these as an error rate of their own. Never bury them in a points total.
• Outcome — whether the agent correctly identified the issue and either resolved it or set up the right next step with clear expectations. This carries the most weight, because it is what the customer came for.
• Skills — a handful of observable behaviors that drive outcomes, such as effective questioning, taking ownership, explaining clearly, and documenting accurately. Write them as things you can hear. Not 'showed empathy' but 'acknowledged the customer's stated concern before moving to the fix'.
• Process notes — unscored observations about what made the contact harder: a missing knowledge article, a slow tool, a policy the agent had to apologize for. These never touch the agent's score.

Test every line, then write the definitions

Put each proposed item through three tests. Two evaluators listening to the same contact would agree on it. The agent controls it. The customer would notice it. An item that fails two of the three should be cut.

Then write a definitions document: for every item, what meets the standard, what does not, and two or three real examples of each. A scorecard without definitions is an opinion form. The document is what you hand a new evaluator, cite in a dispute, and update after every calibration session. It is the actual standard. The form is only where you record against it.

Be careful with tone items. Warmth matters, but 'sounded friendly' is where evaluator bias lives, including bias about accents and speaking styles. Anchor tone criteria to observable moments, such as interrupting, unexplained dead air, or failing to acknowledge a stated frustration, and leave general impressions out of the score.

Choose the sample on purpose

The default sample is a few random contacts per agent per month. Do the arithmetic. An agent handles hundreds of contacts in a month, and a few evaluations are a sliver of them. The agent's monthly score is therefore mostly chance, and a swing between months usually means nothing. Treat agent-level scores as conversation starters, not measurements. Team and program averages, built from many more evaluations, are far more stable.

A useful sample mixes two streams and keeps them labeled. The random stream gives every agent the same treatment and gives you an honest program-level trend. The targeted stream goes where risk and learning are concentrated: new hires in their first weeks, contacts about a newly launched product or policy, contact types with compliance exposure, contacts followed by a repeat contact or a poor survey, escalations, and unusually long or short contacts. Targeted reviews skew negative by design, so they feed coaching and process fixes, not the agent's reported score.

Check coverage by channel and by shift. Evaluators tend to work days, so nights and weekends quietly get sampled less, and a 24/7 operation needs those hours stratified into the plan deliberately. If your platform scores every contact automatically, use it to choose what humans review, not to replace them. Automated scoring tends to be dependable on whether something was said and much weaker on whether the answer was right.

Finally, evaluate fast. Feedback within a few days lands as coaching. Feedback on a contact the agent cannot remember lands as an accusation.

Calibrate until a score means the same thing from every evaluator

Calibration is the practice that makes scores fair, and most programs do it too rarely and too casually. Hold sessions weekly while the scorecard is new and at least monthly once it settles. Everyone scores the same contacts independently before the meeting, without seeing each other's results. In the session, compare item by item, not total by total. Two evaluators can reach the same total through different items, which is hidden disagreement.

Have the most senior person reveal their scores last, or the room will simply converge on them. Discuss each split until the group reaches a ruling, then record the ruling and the example in the definitions document. If a calibration session does not change that document, it was a meeting, not a calibration.

Track the spread between evaluators on the same contact, by item. Items with persistent disagreement are badly written and should be rewritten or removed. Audit the evaluators too, by having a second person re-score a sample of completed evaluations.

Invite more than evaluators. Team leads need to coach to the same standard the evaluators score to. If the work is outsourced, the client has to be in the room regularly, because the client's view of a good contact is the standard. And rotate agents in. An agent who has scored a few contacts against the definitions learns the standard faster than any training module can teach it.

Give agents a dispute process that can change a score

A fair appeal is what makes everything else credible. Keep it simple: the agent has a fixed window to dispute an evaluation in writing, naming the item and the reason. Someone other than the original evaluator reviews it against the definitions document. The decision comes back within a fixed time, with reasoning, and the score is corrected if the dispute is upheld.

Then treat disputes as data. Track volume and overturn rate by evaluator and by item. A high overturn rate on one item means the definition is broken. A high overturn rate for one evaluator is a calibration problem. And a program with no disputes at all is not healthy. It usually means agents have concluded that disputing is pointless or risky.

Close the loop: every evaluation ends in coaching or a fix

A score delivered without a conversation changes nothing. Build the loop so that it cannot be skipped. The agent hears the recording before the conversation and assesses it first. Agents are often harder on themselves, and more specific, than the evaluator was. Pick one behavior to work on, not six. Agree on what it sounds like when done well. Then follow up by listening to the agent's next few contacts for that one behavior and saying what you heard.

Coach from trends across several evaluations, not from one bad call, with a single exception: critical errors get a same-day conversation. Use QA to find excellent contacts as well, and praise them specifically. A library of real, well-handled contacts is first-rate training material.

The other half of the loop is the process notes. Aggregate them monthly and send them to whoever owns the knowledge base, the policy, or the tool. When ten agents stumble at the same point, you do not have ten coaching problems. You have one process problem. A QA program that only ever finds fault with agents is ignoring half of what it hears, and agents notice.

Report QA so it stands up next to your other numbers

Report the critical error rate and the skills score separately, since a blended average hides both. Show the distribution of scores, not just the mean. Alongside the results, report the health of the program itself: evaluations completed against plan, evaluator spread from calibration, dispute and overturn rates, and the share of evaluations that led to a delivered coaching conversation within your target time. That last figure tells you whether you run a quality program or a scoring program.

Validate the scorecard against outcomes. Pull contacts that were followed by a repeat contact or a poor survey and look at what QA said about them. If QA passed them comfortably, the form is measuring the wrong things. This is the check most programs skip, and it is the one that keeps QA honest.

What to do Monday morning

None of this needs a new platform. It needs a week on the basics.

• Rank the purposes — state whether protection, coaching, or insight comes first, and how QA connects to performance management.
• Cut the scorecard — run every item through the three tests, split out the critical items, and add an unscored process notes field.
• Start the definitions document — even a rough one, with examples, and make it the reference in every dispute and calibration.
• Schedule calibration — independent scoring first, item-by-item comparison, rulings written down.
• Publish the dispute process — with a window, an independent reviewer, and a response time, and track overturns.
• Measure the loop — count how many evaluations from last month ended in a coaching conversation. Start there.