Results
Results show how each test case in a run ended, what the agent observed and why. Select a run, then click a test case in its Tests list to open the result details. Sandbox results are described in Sandbox.
Overview
- Latest Results above the run table is the share of finished test cases that passed: Passed divided by Passed, Failed, Error and Blocked. Recently Closed counts runs that finished in the last 7 days. The cards are calculated over the runs listed.
- A shield icon in a run’s Tests list marks a status that was overridden.
- For run statuses, see Run statuses.
Verdicts
When the agent finishes a test case, a separate review step reads the whole recorded session and decides the verdict.
| Verdict | Meaning |
|---|---|
Passed | The agent exercised the feature and it behaved as the test case expected. |
Failed | The agent exercised the feature and it behaved differently from what was expected. This points to a possible product defect. |
Blocked | The test could not be carried out, for example because a precondition was not met (wrong account, missing balance, game unavailable), the agent could not act on the screen (a dialog could not be dismissed), or the device or connection timed out. Blocked says nothing about whether the product works; fix the cause and launch again. |
Error | The session ended without the agent producing a report, for example because it crashed, stopped responding, ran out of time or could not start. There is no verdict on the product. |
Read Failure Reason for Failed and Blocked results before acting on them. A Blocked result does not make the run Failed; an Error result does.
Test case statuses
Besides the verdicts and the obvious Untested, Queued, Running and Cancelled:
| Status | Description |
|---|---|
Skipped | Not executed, for example because no device was set or the chosen device is not available to the person launching. |
Retest | Set by a user with Override Status to mark it for another attempt. |
N/A | Set by a user with Override Status when the test does not apply. |
Result details
- System / Agent Time splits the duration into time outside the test (queueing and setup) and the time the agent was actively testing.
- Launched by names the person or the schedule that launched it.
- Test Instructions (Prompt) is the full text the agent received: plan, run and test case instructions combined. Check it first when the agent did something unexpected.
- Expected Behavior, Actual Behavior, Failure Reason and Steps to Reproduce are written by the agent.
- Requested Output holds the data the instructions asked for; see Requested output.
- Override History lists every manual status change, with who made it, when and why.
View Session (or Watch Live while running) opens the Session viewer.
Execution history
Every launch of a test case adds an execution, so a test case that ran in several launches or rounds has several executions, each labelled with its round (R1, R2, for batch launches) and the test case version it used (v1, v2). Each execution keeps the test case as it was when it ran: its version number, the instructions the agent received, and the device and source. Later edits in the Library do not change past executions.
Overriding a status
When the verdict is wrong, or you want to record a decision (for example marking a known issue as N/A), click the
status badge in the result details. This is available once the test case has a final status other than
Cancelled. The new status can be Passed, Failed, Blocked, Retest, Skipped or N/A, and a reason is
required. The new status replaces the old one in the run’s counts, and the run’s status is recalculated.
Reporting an issue to Jira
When Jira is connected to the project and the test case ended Failed or Error, Report Issue creates a Jira issue pre-filled from the result and links it to both the result and the Library test case. See Reporting an issue from a result.
Everyone with access to the project can view results. Overriding a status and reporting issues requires the Tester or Manager role. See Roles.