What Should Stay Manual When You Automate QA With AI?

What Should Stay Manual When You Automate QA With AI?

Keep manual the work that depends on product intent or accountability: deciding what risk is acceptable, rating how severe a defect is for your business, and checking destructive or irreversible flows such as deleting an account or making a payment. Anything that needs physical hardware also stays manual with TaloTrace, which runs on a browser, an Android emulator and an iOS Simulator and does not offer physical devices.

TaloTrace is a QA testing platform for web, iOS and Android apps. In this post it serves as the example of where the line between automated checking and human judgement can sit.

Much of the remaining work is a candidate for automation:

• Exploring the app to find the journeys worth testing.
• Walking the same journeys each release.
• Confirming an outcome really happened.

The test for any task is simple. If a correct answer can be checked by a machine, automate it. If the answer is a decision someone has to own, keep it with a person.

What Does Over-Automation Look Like Week to Week?

Over-automation often shows up as a green dashboard nobody trusts or a flood of results nobody triages; the opposite failure, automating too little, is just as common. The first symptom is a green dashboard that nobody trusts. Checks pass, a release ships, and a customer finds a problem that the automated checks technically covered but that looks obviously wrong to a human eye. The team then spends the following sprint arguing about whether the check or the product was at fault.

The second symptom is a flood. When automation runs often, results can arrive faster than anyone has been assigned to triage them. Without someone deciding what matters, engineers get a stream of noise, and triage turns into a job nobody has been assigned.

The third symptom is the opposite failure. A team is nervous about handing anything over, so it automates a thin slice, keeps every other check manual, and discovers that the repeated regression pass still eats the week. The line was drawn at fear rather than at a reason.

Why Does the Line Get Drawn in the Wrong Place?

Checking means confirming whether a known outcome occurred, and judging means deciding whether that outcome is acceptable. The line gets drawn in the wrong place because the two look similar from the outside. Both end with someone saying a screen is fine or not fine. The first can be written down once and repeated. The second changes with the release, the customer and the business moment.

A scripted test encodes a checking decision made against one version of the interface. When the interface changes, a person often has to work out whether a failure is a real defect or an outdated assumption in the script.

The other cause is ownership. When nobody has said who decides severity, who approves a finding, or who signs off on a risky flow, automation inherits that gap. A faster tool can make an unclear process produce unclear results faster.

What Do Teams Usually Try, and Where Does It Run Out?

Teams usually try one of three approaches, and each fixes part of the problem while leaving a different part exposed.

• Keep everything manual: Judgement stays human, but repeated regression passes tend to consume the time that exploratory work needs, and coverage depends on who is available that week.
• Script the stable paths: Repeated journeys get covered, but the scripts tend to need upkeep as the interface changes, and a failure still needs a person to work out whether the product or the script broke.
• Automate everything and review nothing: Output is fast, but no one is accountable for what counts as a real defect, so the results tend to be hard to trust.

The pattern across all three is that the line sits in the wrong place for a reason unrelated to the work itself. The better question is what each task needs: repetition, or a decision.

What Makes TaloTrace Different When You Draw This Line?

TaloTrace takes on the repeatable checking, and a person approves findings and sets boundaries. You point TaloTrace at your app and, optionally, describe the flow you care about in plain language, such as completing a checkout. TaloTrace explores the running app, plans the journeys worth testing, drives each one to a verifiable outcome and turns completed journeys into replayable scenarios. There are no test scripts to write.

Several parts of that design map directly onto the human-owned list:

• Destructive actions are held back. TaloTrace commits real actions such as Create, Save and Submit and reads the resulting screen to confirm what changed, but holds back destructive or irreversible actions such as deleting an account or making a payment.
• You set boundaries. You can declare boundaries TaloTrace stays inside. A flow that would cross one is recorded as blocked, with a note on what would be needed to test it safely.
• Completion is proven. A goal is marked done only when a machine-checkable predicate proves the outcome, so a screen that already shows the right heading cannot falsely complete a goal.
• Findings are reviewed. TaloTrace independently verifies each finding before it reaches you. Results start hidden, and findings become visible only when a reviewer approves them or the project is configured to auto-approve.
• Severity follows your guidance. A per-project rubric lets you supply your own triage guidance, and TaloTrace follows it when rating findings across five levels, Critical (P0) to Trivial.

Each finding carries a screen recording and the time window inside it where the defect shows, so the person making the call has evidence in front of them. For mobile specifics, the post on autonomous mobile app testing goes deeper.

This does not remove human work. Human work centres on approving findings, setting boundaries and owning the risky flows, which is where judgement was needed anyway. Low-confidence findings are also held separately and can only be released by an explicit per-finding decision, so a bulk approval cannot push them through. The post on why QA test results stay hidden until someone approves them explains the approval step in more detail.

How Do You Draw the Line for Your Own Team?

Go through your flows one at a time and ask three questions of each:

• Can a machine confirm the outcome? If yes, the check is a candidate for automation.
• Does the check repeat each release? If yes, automating it saves the most time.
• Would a wrong result cost more than a person's time to review? If yes, keep a person on the sign-off.

A flow that is repeatable and checkable belongs with automation. A flow that is irreversible, depends on physical hardware, or needs a decision about intent belongs with a person. Where a flow is borderline, automate the exploring and keep the sign-off.

Some areas stay manual today for plain product reasons. Real-device checks are one, since TaloTrace does not offer physical devices. Drawing and signature input is another, since stroke input is in development and not something to plan around yet. Write these down as human-owned so they are a decision rather than a gap.

Finally, name an owner for severity and for approving findings before you switch anything on. Then the first week of results has someone responsible for reading it.

How Do You Get Started?

Start with one flow you run every release and one you want a person to keep. Give TaloTrace the first, set a boundary around the second, and see how the results read to the person who owns approval.

Get early access or book a demo