TL;DR: Two testers rate the same bug differently because severity folds together several judgement calls: how many users are affected, how badly, and whether a workaround exists. Without a written rubric, each tester fills those gaps from their own experience. A rubric only closes the gap if something applies it the same way every time.
Key Takeaways
The core problem: Severity blends several judgement calls into one rating, and without a written rubric each tester fills the gaps differently.
Why it happens: Impact, how badly a flow breaks, and workaround availability are not the same axis, so two testers can weigh them differently and both be reasoning correctly.
Where a written rubric alone runs out: A document in a wiki still gets interpreted differently person to person, and drifts once new testers join without reading it first.
A different approach: a rubric only closes the gap between people if it is applied to every finding, not just referenced from memory.
TaloTrace's part: TaloTrace supports a per-project severity rubric, so ratings follow the triage guidance your team supplies.
What ships with each finding: a severity label shown with its code, like Critical (P0), backed by a screen recording and TaloTrace's reasoning.
Why Do Two Testers Rate the Same Bug at Different Severity Levels?
Two testers rate the same bug at different severity levels because severity is not one judgement, it is several folded together: how many users hit the issue, how badly it breaks their task, and whether a workaround exists. That context does not sit anywhere both testers can check against, so each person resolves the gap using their own sense of what matters. A written rubric only helps if it is actually consulted and applied the same way on every bug, not just available somewhere in a wiki.
What Does Severity Disagreement Look Like on a Real QA Team?
Severity disagreement usually shows up first in triage. Two testers look at the same broken checkout step: one tags it Critical because payment is involved, the other tags it Medium because the workaround, retrying the page, is obvious to anyone who has used the app before. Both are reasoning from something real, they are just weighing different things.
Over a few sprints, inconsistent severity ratings compound in three ways:
Labels stop driving escalation: a ticket gets escalated after a manager reads the description rather than the label.
Engineers stop trusting the field: severity becomes decoration because it does not reliably predict how bad a bug actually is.
New testers inherit one person's scale: whoever is newest calibrates against the senior tester who reviewed their first few tickets, not against a shared standard.
Why Does Severity Judgement Vary So Much Between People?
Severity is a compound judgement, not a single measurement. It usually mixes at least three things: how many users are affected, how badly the flow breaks for them, and whether there is a reasonable way around it. A tester who weighs 'how many users' heavily rates a rare edge case low even when it is unrecoverable for the people who hit it; a tester who weighs 'how badly it breaks' heavily rates that same bug high.
Judgement call | Question the tester answers | Why two testers differ |
|---|---|---|
Reach | How many users hit this bug? | One tester rates a rare but unrecoverable bug low because few people see it. |
Impact | How badly does it break the task? | Another rates the same bug high because it blocks everyone who does hit it. |
Workaround | Is there a reasonable way around it? | What counts as "obvious" depends on how well the tester knows the app. |
Tacit knowledge widens the gap between two testers' ratings. The tester who knows checkout is the highest-revenue flow in the app treats anything there as more severe by default. The tester who does not have that context has no reason to. Neither is wrong given what they know, they simply know different things, and the bug itself does not resolve that gap.
Inconsistent severity ratings are not a training failure. Two competent testers can look at identical evidence and land on different ratings because the rating depends on context that lives in their heads, not on the page in front of them.
What Do Teams Usually Try First to Fix Severity Disagreement?
A common first attempt is a written rubric: a table in a wiki or ticket template defining what Critical, High, Medium and Low mean, sometimes with example bugs attached. This helps, but it runs into two limits. Words like 'significant' or 'major' still mean different things to different readers, so two people can read the same definition and land on different ratings. A document also only helps the people who read it; testers who join after it was written often calibrate against whoever trains them instead of against the rubric itself.
Formal references exist as well. The IEEE Standard Classification for Software Anomalies (IEEE 1044-2009) sets out a uniform approach to classifying software anomalies. A general standard still leaves each team to decide what counts as severe for its own product, which is exactly the gap two testers fill in differently.
Another option is to review recent ratings together as a group and agree on where each one should have landed. That corrects drift for the specific bugs discussed, but it gets harder to keep up as ticket volume grows, and the shared understanding it builds still depends on someone applying it consistently between sessions.
Leaving it to individual judgement, the default starting point, is what produces the disagreement in the first place.
How Does a Shared, Enforced Rubric Fix Severity Disagreement?
A severity rubric is a written set of rules that maps what a bug does (how many users it affects, how badly it breaks their task, whether a workaround exists) to a severity level. It closes the gap between people only if it is attached to the work itself and applied the same way every time, not stored somewhere a tester has to remember to open. A document on its own cannot guarantee that, because it depends on whoever is triaging that day actually consulting it.
TaloTrace, an AI-powered QA testing platform, supports this with a per-project severity rubric: you supply your own triage guidance, and TaloTrace follows it when rating the findings it surfaces in that project. Because the rubric lives with the project rather than with whichever person is triaging, the same guidance applies to every finding TaloTrace rates there. For a closer look at how a per-project rubric works within the fixed five-level scale, see how TaloTrace applies a custom severity rubric.
What Makes TaloTrace Different When It Comes to Severity Ratings?
TaloTrace separates finding a bug from reporting it. It drives your app, records what looks wrong, and independently verifies each finding before it reaches you, so nothing you see is the raw, unchecked output of the step that produced it.
Three things come with every rated finding:
Evidence to check the rating against: a screen recording, the time window inside it where the defect shows, and TaloTrace's reasoning.
One issue per defect: repeated observations of the same defect within a run collapse into a single issue, so your team is not rating, and disagreeing about, the same bug under several tickets.
A fixed, readable scale: five levels, Critical, High, Medium, Low and Trivial, each shown with its code, for example Critical (P0). Your rubric changes how a finding gets rated, not which labels exist.
For more on how repeated observations collapse into one issue, see how TaloTrace deduplicates findings before they reach your tracker.
How Do You Get Started with TaloTrace?
TaloTrace is in open beta. Most tiers are published and buyable directly on the pricing page, with custom terms available at enterprise scale for teams that need them.
Beta is open now. Apply for early access and bring your severity rubric with you to see how TaloTrace rates real findings against it.
Frequently Asked Questions
What is the difference between bug severity and priority?
Severity describes how bad a bug is on its own: how many users it affects and how badly it breaks their task. Priority describes how soon it should be fixed, which also depends on things severity does not capture, like an upcoming release date or how much engineering time is available. Two bugs can share a severity and still get different priorities.
Does writing down a severity rubric fix inconsistent ratings by itself?
It helps, but not completely. A written rubric still needs a reader to interpret words like 'significant' or 'major' the same way someone else does, and it only helps the people who actually open and apply it. Ratings drift again once new testers join without reading it, or once people stop checking their judgement against it.
Can TaloTrace use our own severity rubric instead of a generic one?
Yes. TaloTrace supports a per-project severity rubric: you supply your own triage guidance, and TaloTrace follows it when rating the findings it surfaces in that project, so ratings reflect how your team defines severity rather than a generic default.
What severity levels does TaloTrace use?
TaloTrace rates findings on five levels, Critical, High, Medium, Low and Trivial, shown together with their underlying code, for example Critical (P0). The scale itself is fixed; what your rubric changes is how a finding gets rated against it, not the label options available.
How can you tell if your team's severity ratings are inconsistent?
Look for three signals:
Escalation ignores the label: tickets get escalated because of what the description says, not the severity.
Engineers skip the field: the severity rating plays no part in deciding what gets fixed first.
New testers mirror one person: newer testers' ratings follow a single senior colleague rather than a written standard.
Any one of these suggests the rating depends on who triaged the bug, not on the bug itself.
How does TaloTrace back up a severity rating so my team can check it?
Every finding TaloTrace surfaces is independently verified before it reaches you, and comes with a screen recording, the specific time window where the defect shows, and TaloTrace's reasoning. That gives your team something to check a rating against instead of just a label.
