How to judge a QA partner before the first test case

Escrito por Jesus Dugarte en

...

 

It is easy to see how they run QA in a polished demo. The slides look good, the dashboard is colorful, and the team speaks confidently about coverage. The harder question is what happens on a normal Tuesday, when a release is late, a defect is ambiguous, and three people disagree about whether the build is ready. That is where the operating model shows.

 

QA is not a department that receives builds and returns bug lists. It is an operating system for release confidence. It decides what to test first, what to automate, what to accept as risk, and who gets to say no. A partner with a strong system makes those decisions visible. A weak one hides behind activity metrics.

 

The first thing to examine is discovery. Does the team ask about users, failure modes, release cadence, and the cost of a defect? Or does it jump straight to a test plan template? A QA team that only executes test cases is a ticket queue, and a ticket queue cannot protect a release. Discovery is where a partner proves it understands the business.

 

The second is risk. Not every feature deserves the same depth. A payment flow, an authentication change, and a copy update carry different consequences. A good partner can explain its risk model, show how tests map to it, and adjust when the release changes. If everything is high priority, nothing is.

 

What a healthy QA operating model looks like

 

A healthy model starts before code is finished. Testers review acceptance criteria, challenge edge cases, and help shape the definition of done. That shift-left work prevents rework and catches ambiguity early. Quality is a feedback loop, not a phase, and the loop has to start while the team is still designing.

 

The next layer is environments and test data. A team that cannot reproduce a bug because staging is unstable will spend its week chasing infrastructure. Ask how environments are provisioned, how data is refreshed, and who owns access. A partner that treats this as someone else's problem will slow every release.

 

Automation should be a product, not a pile of scripts. A useful suite has clear ownership, stable tests, fast feedback, and a plan for retiring checks that no longer earn their place. Flaky tests are worse than no tests because they teach the team to ignore red builds. Ask for the flaky rate, not just the number of automated cases.

 

Metrics should describe outcomes. Escaped defects, time to detect, cycle time, and release confidence are more useful than test case counts. Coverage can be a diagnostic, but it is not proof of quality. A partner that leads with "we executed 4,000 cases" is telling you about effort, not risk.

 

How to evaluate a partner in practice

 

Ask for a walkthrough of the last release, not the best demo. Pick a real feature, including one that went badly, and ask how the team planned, what changed, and what it learned. Ask for the last release, not the best demo, because the messy one reveals the operating model.

 

Watch a bug triage session. A good team argues about severity with evidence, reproduces the issue, and agrees on the next step. A weak team escalates everything or quietly downgrades defects to protect the date. The conversation tells you more than any SLA document.

 

Check who owns the release decision. If QA can only recommend and never block, the role is ceremonial. If developers can merge without a quality gate, the process depends on heroics. The handoff to developers is where QA proves itself, because a reproducible bug with context gets fixed faster than a screenshot and a complaint.

 

Ask about security and data privacy. Test environments often contain copies of production data, and that is a risk. A serious partner can explain masking, access control, retention, and what happens when a tester needs realistic data. Quality without security is incomplete.

 

Review the commercial model. A per-test-case price encourages volume; a per-release or per-outcome model encourages judgment. Neither is automatically good, but the incentive should match the behavior you want. Ask what happens when scope changes, who absorbs the extra effort, and how the partner reports it.

 

Run a small pilot with a real release. Give the team one feature, one environment, and one deadline. Measure how quickly it understands the domain, how it communicates, and whether its work reduces risk. A pilot is cheaper than a bad annual contract, and it gives both sides an honest preview.

 

The final question is cultural. Does the team ask uncomfortable questions? Does it say "not ready" when the evidence supports it? Does it explain trade-offs without hiding behind process? A QA partner that only agrees with the client is not a partner. It is a mirror.

 

A strong QA operating model is calm, visible, and boring in the best sense. Work is prioritized, bugs are reproducible, automation is maintained, and releases are decisions rather than surprises. A QA team that only executes test cases is a ticket queue, and a ticket queue cannot give you confidence. Look for the system behind the demo, and you will know whether the partner can carry the responsibility.