Scenarios

SDK examples on this page require the 0.5.0 source preview. npm publication is pending; use the HTTP endpoints directly in the meantime.

A Scenario groups related work, such as sales report checks or customer support answers. Create it once, then bind new Sessions to its scn_… ID. Each Turn and feedback record inherits that binding. This lets you compare results across Sessions and Agent revisions without depending on a display name.

Scenarios belong to your organization. Names may repeat, and renaming a Scenario keeps its ID. A Scenario does not configure an Agent, run a task, judge output or train a model. You can continue using unbound Sessions and Turn feedback.

Scenario binding is set when you directly create an Agent Session. Saved Agent defaults and Schedule targets do not accept a Scenario setting.

Create, run and assess

Use rebyte.scenarios, rebyte.sessions and rebyte.feedback from @rebyteai/agent-extensions. The official client also supports the same HTTP routes through client.get and client.post.

This example uses Node.js 22+, an organization key with tasks:read and tasks:write, and normal model credit. It creates no Sandbox. Keep the key on your server.

export REBYTE_BASE_URL=https://api.rebyte.ai/v1
export REBYTE_API_KEY=your-organization-key

import OpenAI from 'openai';
import { RebyteExtensions } from '@rebyteai/agent-extensions';
import { randomUUID } from 'node:crypto';
import { setTimeout as delay } from 'node:timers/promises';

const client = new OpenAI({
  apiKey: process.env.REBYTE_API_KEY,
  baseURL: process.env.REBYTE_BASE_URL ?? 'https://api.rebyte.ai/v1',
  maxRetries: 0,
});
const rebyte = new RebyteExtensions(client);
// Persist this key with the creation request; reuse both when retrying it.
const scenarioKey = randomUUID();
const scenario = await rebyte.scenarios.create({
  name: 'Arithmetic answer checks',
  description: 'Compare exact answers across Agent revisions.',
  metadata: { suite: 'arithmetic-v1' },
}, { headers: { 'Idempotency-Key': scenarioKey } });
const agent = await client.beta.agents.create({
  name: 'Arithmetic assistant', model: 'qwen3.6-35b-a3b',
  instructions: 'Answer arithmetic questions with just the number.',
});
const session = await rebyte.sessions.create({
  agent_id: agent.id, scenario_id: scenario.id,
  environment: { type: 'none' }, input: 'What is 19 + 23?',
});

// This new Session contains exactly one input and one Turn.
let turn;
const deadline = Date.now() + 120_000;
while (Date.now() < deadline) {
  turn = (await rebyte.sessions.turns.list(session.id)).data[0];
  if (turn && ['completed', 'failed', 'cancelled'].includes(turn.status)) break;
  await delay(1000);
}
if (!turn || turn.status !== 'completed') throw new Error('Task did not complete');
const text = [];
for await (const item of rebyte.sessions.items.list(session.id, { order: 'asc' })) {
  if (item.type === 'message' && item.role === 'assistant') {
    for (const part of item.content) if (part.type === 'output_text') text.push(part.text);
  }
}
const answer = text.join('\n').trim();
const feedbackKey = randomUUID();
const feedback = await rebyte.feedback.create({
  turn_id: turn.id,
  rating: answer === '42' ? 'positive' : 'negative',
  comment: answer === '42' ? 'Correct answer.' : 'Expected exactly 42.',
  source: { type: 'evaluator', id: 'arithmetic-exact-match-v1' },
}, { headers: { 'Idempotency-Key': feedbackKey } });
// scenario_id was inherited; no repeated binding field was needed.
console.log({ scenario_id: feedback.scenario_id, turn_id: feedback.turn_id, answer });
for await (const assessment of rebyte.feedback.list({ scenario_id: scenario.id })) {
  console.log(assessment.turn_id, assessment.rating, assessment.comment);
}

The example leaves resources available for inspection. To remove its Session and Agent, call rebyte.sessions.delete(session.id) and client.beta.agents.delete(agent.id). Session deletion makes its feedback inaccessible. Scenarios have no delete endpoint. The complete SDK recipe cleans up its Session and Agent automatically.

Scenario API

These Rebyte extension endpoints need no OpenAI-Beta header. Reads require tasks:read; creation and updates require tasks:write.

Create accepts a nonblank name of at most 256 characters, an optional nullable description of at most 4,000 characters, and optional object metadata. Omitted description becomes null; omitted metadata becomes {}. Updates leave omitted fields unchanged and replace supplied metadata. metadata: null clears metadata. There is no delete operation.

{
  "object": "scenario",
  "id": "scn_abc",
  "name": "Arithmetic answer checks",
  "description": null,
  "metadata": {},
  "created_at": 1791417600,
  "updated_at": 1791417600
}

Timestamps are Unix seconds. Creating with a stable Idempotency-Key returns HTTP 201 initially and 200 for an identical retry. Reusing that key with a different request body returns 409. Without a key, each accepted creation creates a new Scenario, even when its name already exists. Save the returned ID and use it when renaming or binding Sessions.

await rebyte.scenarios.update(scenario.id, { name: 'Arithmetic regression checks' });
const saved = await rebyte.scenarios.retrieve(scenario.id);
for await (const entry of rebyte.scenarios.list({ limit: 20 })) console.log(entry.id, entry.name);

Binding and filtering

Session creation accepts optional scenario_id: string | null. Omitted or null means unbound. A non-null ID must identify a Scenario in the same organization. The binding is immutable, including for initially unbound Sessions: create a new Session to choose another Scenario. Updating the saved Agent or renaming the Scenario does not change this binding.

Session and Turn resources, embedded Session/Turn lifecycle event resources, and feedback include scenario_id: string | null. Trace and history responses also carry the binding; bound traces include the rebyte.scenario_id OTLP resource attribute. The SDK's SessionResource, TurnResource, SessionEvent and Feedback types expose the corresponding fields.

POST /v1/agents/sessions/{session_id}/events and POST /v1/feedback accept an optional nullable scenario_id assertion. If supplied, it must equal the Session's existing binding. It does not rebind the Session or Turn. Omit it to inherit automatically. For example, feedback still needs only turn_id and rating.

GET /v1/agents/sessions?scenario_id=scn_abc
GET /v1/feedback?scenario_id=scn_abc
GET /v1/feedback?scenario_id=scn_abc&turn_id=turn_123

Feedback listing requires turn_id, scenario_id, or both; both filters are intersected. Session and Scenario lists support limit, order and after. Keep the same filters and order when following cursors. Organization keys do not authorize your application's end users: check their access before submitting events or feedback through your backend.

See Feedback API reference for assessment fields and Session events for streaming.

Review data and start training

Use rebyte.scenarios.turns.list and .retrieve for a Scenario's business Turns. Each review includes input, output, latest active feedback and training eligibility. Your application can delete one feedback record, clear a Turn's feedback, or soft-delete a terminal Turn. Admitted training snapshots retain their original examples when live data changes.

Use managed training runs to select all or untrained eligible data, optionally restrict Turn IDs, and explicitly start training. Run creation returns the frozen source, full settings and snapshot identity. No training starts merely because a Turn receives feedback.