Evaluation API

Conversations can be evaluated automatically. An agent is configured with evaluation definitions; each produces one or more insights — a value plus the reasoning behind it. Evaluations run either per turn or once at the end of a conversation.

Insights marked as user-facing are meant to be shown to the end user (for example a summary of how the conversation went); the rest are for analytics.

Result Availability

End-of-conversation evaluations start after the session ends, so results are not available immediately. Poll the status endpoint until it reports a terminal state.

status Meaning

NOT_READY

Evaluation has not finished yet. Keep polling.

FINISHED

Results are available in results / featured.

NO_EVALS

The agent has no user-facing evaluations configured; nothing will arrive.

FAILED

Evaluation could not be completed.

Whether a given agent produces user-facing results at all can be checked up front via the hasUserFacingEvaluations field of the Agent object.

GET /api/v2/evaluation/status

Return the evaluation status and results of one session. Open to unauthenticated callers, so a client can fetch the outcome of a session it just ran anonymously.

Query Parameters

Parameter Required Type Description

sessionId

Yes

UUID

Session to report on — the session ID used when the conversation was opened.

authorization

No

String

JWT bearer token, when not passed as a header. See Authentication.

Responds 404 when the session does not exist.

Response

EvaluationStatusResponse:

Field Type Description

status

String

NOT_READY, FINISHED, NO_EVALS or FAILED.

results

Array | absent

The ordered layout of sections. Omitted while no results exist.

featured

Object | absent

The headline outcome of the conversation. Omitted when the agent has no featured evaluation or it has not been decided.

Complete FINISHED Response

The example below is deliberately exhaustive: it contains every section type (FEATURED_INSIGHT, STANDALONE_INSIGHT, GROUP), every insight value type (boolean, number, text, list), both execution modes, an undecided insight (value: null) and an insight without evidence.

{
  "status": "FINISHED",
  "results": [
    {
      "type": "FEATURED_INSIGHT",
      "order": -2147483648,
      "evaluationDefinitionId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
      "sessionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "value": {
        "type": "number",
        "value": 4,
        "minValue": 1,
        "maxValue": 5
      },
      "subtitle": "Conversation quality",
      "text": "The agent understood the request and confirmed the outcome with you.",
      "created": "2026-09-09T14:31:07.412+02:00"
    },
    {
      "type": "STANDALONE_INSIGHT",
      "order": 1,
      "id": "b2f8c0de-1c2b-4b8f-8a4e-9d7f1a6c3e10",
      "evaluationInsightDefinitionId": "d1a4b6c8-3e5f-4a2b-9c7d-8e0f1a2b3c4d",
      "sessionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "ref": "taskCompleted",
      "name": "Task completed",
      "description": "Whether the user's request was fully resolved.",
      "value": {
        "type": "boolean",
        "value": true,
        "trueText": "Resolved",
        "falseText": "Not resolved"
      },
      "reasoning": "The user asked to reschedule the appointment and the new time was confirmed.",
      "evidence": "\"So that's Thursday at 9:30 — confirmed.\"",
      "executionMode": "ON_CONVERSATION_END",
      "created": "2026-09-09T14:31:07.412+02:00"
    },
    {
      "type": "STANDALONE_INSIGHT",
      "order": 2,
      "id": "c3a9d1ef-2d3c-4c9f-9b5f-0e8a2b7d4f21",
      "evaluationInsightDefinitionId": "e2b5c7d9-4f6a-4b3c-8d1e-9f0a1b2c3d4e",
      "sessionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "ref": "summary",
      "name": "Summary",
      "description": "Short summary of the conversation.",
      "value": {
        "type": "text",
        "value": "The user rescheduled an appointment and asked about parking."
      },
      "reasoning": "Derived from the full transcript.",
      "executionMode": "ON_CONVERSATION_END",
      "created": "2026-09-09T14:31:07.412+02:00"
    },
    {
      "type": "GROUP",
      "order": 3,
      "id": "a1b2c3d4-e5f6-4718-9a0b-1c2d3e4f5a6b",
      "sessionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
      "name": "Engagement",
      "description": "How the user engaged during the conversation.",
      "insights": [
        {
          "id": "f4c6a8b0-5a7b-4c8d-9e0f-1a2b3c4d5e6f",
          "evaluationInsightDefinitionId": "aa11bb22-cc33-4d44-8e55-6f778899aabb",
          "sessionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
          "ref": "engagementLevel",
          "name": "Engagement level",
          "description": "How engaged the user was, from 0 to 10.",
          "order": 1,
          "value": {
            "type": "number",
            "value": 7,
            "minValue": 0,
            "maxValue": 10
          },
          "reasoning": "The user answered every question and asked two follow-ups.",
          "evidence": "\"And what about parking, is that included?\"",
          "executionMode": "PER_TURN",
          "created": "2026-09-09T14:31:07.412+02:00"
        },
        {
          "id": "0b1c2d3e-4f50-4617-8829-9a0b1c2d3e4f",
          "evaluationInsightDefinitionId": "bb22cc33-dd44-4e55-8f66-7788990011cc",
          "sessionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
          "ref": "topicsDiscussed",
          "name": "Topics discussed",
          "description": "Topics the user raised.",
          "order": 2,
          "value": {
            "type": "list",
            "value": ["scheduling", "parking"]
          },
          "reasoning": "Both topics were raised by the user.",
          "executionMode": "ON_CONVERSATION_END",
          "created": "2026-09-09T14:31:07.412+02:00"
        },
        {
          "id": "1c2d3e4f-5061-4728-993a-ab0c1d2e3f40",
          "evaluationInsightDefinitionId": "cc33dd44-ee55-4f66-8077-889900112233",
          "sessionId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
          "ref": "escalationRequested",
          "name": "Escalation requested",
          "description": "Whether the user asked for a human.",
          "order": 3,
          "value": null,
          "reasoning": "Not enough signal in the conversation to decide.",
          "executionMode": "ON_CONVERSATION_END",
          "created": "2026-09-09T14:31:07.412+02:00"
        }
      ],
      "created": "2026-09-09T14:31:07.412+02:00"
    }
  ],
  "featured": {
    "evaluationDefinitionId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "value": {
      "type": "number",
      "value": 4,
      "minValue": 1,
      "maxValue": 5
    },
    "subtitle": "Conversation quality",
    "text": "The agent understood the request and confirmed the outcome with you."
  }
}

Note that the featured outcome appears twice — once as the first entry of results (so a client rendering the sections in order gets it in the right place) and once in featured (so a client that only wants the headline does not have to search the list).

Section Types

Every entry of results carries type and order; render them ascending by order.

type Fields

FEATURED_INSIGHT

evaluationDefinitionId, sessionId, value, subtitle, text, created. order is the minimum integer, so it always sorts first.

STANDALONE_INSIGHT

id, evaluationInsightDefinitionId, sessionId, ref, name, description, value, reasoning, evidence (optional), executionMode, created.

GROUP

id, sessionId, name, description, created and insights — an array of the same fields as a standalone insight, each with its own order within the group.

executionMode is PER_TURN (evaluated every turn, optionally over a lookback window) or ON_CONVERSATION_END (evaluated once with the full transcript).

Value Types

value is polymorphic on its own type property, and is null when an insight could not be decided. Clients must handle all four types and the null case.

type Shape

boolean

value plus the optional display labels defined for the insight:

{"type": "boolean", "value": true, "trueText": "Resolved", "falseText": "Not resolved"}

number

Integer value with the optional range it was defined with; the engine clamps out-of-range values into that range:

{"type": "number", "value": 7, "minValue": 0, "maxValue": 10}

text

Free text:

{"type": "text", "value": "The user rescheduled an appointment."}

list

List of strings. When the insight defines an allowed vocabulary, values outside it are dropped:

{"type": "list", "value": ["scheduling", "parking"]}

trueText, falseText, minValue and maxValue are omitted when the definition does not set them.

GET /api/evaluation/status

Version 1 of the same endpoint, with the same parameters and response shape. It reports results only once every insight has been decided, whereas the v2 endpoint also returns insights whose value is still null. New integrations should use the v2 endpoint.

GET /api/evaluation/insights

Return the latest user-facing insight sections for the caller across sessions, rather than for one specific session. Requires the consumer role.

Query Parameters

Parameter Required Type Description

agentRef

No

String

Restricts the result to one agent. Accepts a revision-qualified ref (acme-support.5); without it, insights of all agents of the caller are returned.

Response

An array of the same section objects as results above — FEATURED_INSIGHT, STANDALONE_INSIGHT and GROUP entries with the value types described in Value Types.

GET /api/evaluation/aggregations

Return aggregated end-of-conversation insight values over a time range — the data behind progress charts. Requires the consumer role.

Query Parameters

Parameter Required Type Description

agentRefs

No

Array of strings

One or more agent refs; repeat the parameter (?agentRefs=a&agentRefs=b). Defaults to all agents of the caller. Revision-qualified refs are accepted.

from

No

ISO 8601 date-time

Start of the aggregation window, e.g. 2026-08-01T00:00:00%2B02:00.

to

No

ISO 8601 date-time

End of the aggregation window.

Only insights whose definition runs ON_CONVERSATION_END and is marked user-facing are aggregated.