Test an agent before you publish it

Before real customers talk to your agent, you can try it yourself.

The Test section lets you run simulated conversations.

Simulated Users play your customers.

Agent Evaluations score how your agent did.

And Test Suites bring them together, and run the conversations.

Imagine you have an agent called Emma, who helps customers who want to cancel.

To test her, we created a simulated user, called Jordan Retention Probe.

Its profile says what it tests: a mis-sold plan cancellation, escalation and handover.

Below is the System Prompt, where its behavior lives, in plain text.

It is written in sections. Role and Identity says who the simulated user is.

Your Goal says what it wants.

The Rules set how it behaves.

And the Finish Condition tells it when to end the conversation.

Next, the agent evaluation. This one scores how Emma handled the conversation.

The Evaluation Prompt tells it what to judge.

One tip: the conversation is recorded from the simulated user's side.

So in this prompt, call your agent the user.

Further down, the Output Data Elements are the results it collects.

Each has a type, such as pass or fail, or text, an output name, labels, and an extraction prompt.

Now put it together with a test suite. Open Run New Test Suite, and give it a name.

Then pick the agent you want to test. Open the list, and choose it from the agents of this project.

And the simulated user it will talk to. Open this list as well, and choose the one you prepared.

The revision defaults to the latest published version.

Under Advanced settings, set the timeout, the maximum turns, and how many conversations to run.

Save and run starts the conversations. That uses minutes, so run it only when you mean to.

When a run finishes, Test Suites shows its last run, its status, and how long it took.

Open the run, and start with the transcript: the full conversation between your agent and the simulated user.

The Reason line explains the result.

Then read the conversation, to see what each side said.

Metrics shows the Score column, and whether the evaluations passed.

When your suite includes agent evaluations, the Evaluations passed column shows the result for each run.

To try again after a change, use Edit and rerun.