Build a feedback loop

An Agent can finish a task without getting it right. Feedback tells you whether the result was useful: a user rejected the answer, a test passed, or a reviewer supplied a correction.

Rebyte records that judgment against the task's Turn ID. Start with one Agent, one task, and one assessment. No Scenario or separate receipt is needed.

The feedback loop

Rebyte runs the Agent’s execution loop for you. A Turn can span several model and tool calls, so its feedback judges the task as a whole.

1. Run. Your application sends input to a Session. Rebyte runs the Agent, including its model and tool calls, and exposes the result with a turn_id.
2. Assess. A user, your application, or an evaluator judges the result. Choose positive or negative; add a reason and, when useful, a correction.
3. Record. Submit the assessment to POST /v1/feedback. Rebyte stores it with the Turn, Session and Agent IDs so you can retrieve it later.
4. Improve and verify. Review failures, change instructions or tools, then run the same tasks in new Sessions. Compare results before adopting the change.

Available today: Turn execution, feedback submission, retrieval and listing. Judging results and improving the Agent belong to your application or team. Submitting feedback does not automatically change instructions, train model weights, or publish a new Agent version. A correction is stored evidence; it is not sent to the Agent as a new message.

One task, two IDs

A Turn can receive several assessments, such as a user rating and a test result. They remain separate records. Rebyte resolves session_id and agent_id from the Turn; you do not supply them in a feedback request.

Save the Turn ID alongside each result you display. In an active conversation, use the ID from the corresponding Session events or Turn history. Do not attach a delayed rating to whichever Turn happens to be newest.

Quickstart: record your first assessment

This example runs one small model task, checks its answer and records the result. It uses Node.js 22 or later and the official OpenAI client. It creates no Sandbox; normal model usage is billed. Use a Rebyte organization key with tasks:write and tasks:read, and keep it on your server.

pnpm add openai@7.15.0
export REBYTE_API_KEY="your-rebyte-api-key"

1. Run a task and keep its Turn ID

Create feedback.mjs. Copy the three JavaScript blocks in this quickstart into the same file, in order.

import OpenAI from 'openai';
import { randomUUID } from 'node:crypto';
import { setTimeout as delay } from 'node:timers/promises';

const client = new OpenAI({
  apiKey: process.env.REBYTE_API_KEY,
  baseURL: 'https://api.rebyte.ai/v1',
  maxRetries: 0,
});
const agent = await client.beta.agents.create({
  name: 'Feedback quickstart',
  model: 'gpt-6-luna',
  instructions: 'Answer arithmetic questions with just the number.',
});
const session = await client.beta.agents.sessions.create({
  agent_id: agent.id,
  input: 'What is 19 + 23?',
});
console.log({ agent_id: agent.id, session_id: session.id });

// This is a new Session with exactly one input, so it has one Turn.
let turn;
const deadline = Date.now() + 120_000;
while (Date.now() < deadline) {
  const page = await client.beta.agents.sessions.turns.list(session.id);
  turn = page.data[0];
  if (turn && ['completed', 'failed', 'cancelled'].includes(turn.status)) break;
  await delay(1000);
}
if (!turn || turn.status !== 'completed') {
  throw new Error('Task did not complete: ' + JSON.stringify(turn));
}
const text = [];
for await (const item of client.beta.agents.sessions.items.list(session.id, { order: 'asc' })) {
  if (item.type === 'message' && item.role === 'assistant') {
    for (const part of item.content) {
      if (part.type === 'output_text') text.push(part.text);
    }
  }
}
const answer = text.join('\n').trim();
console.log({ turn_id: turn.id, answer });

2. Judge the answer and submit feedback

Our evaluator checks the answer, not just the Turn's completion status. Replace this exact-match check with your own tests, rubric, or user rating.

const matched = answer === '42';
// Save this key with the assessment. Reuse it if you retry this submission.
const feedbackKey = randomUUID();
const feedback = await client.post('/feedback', {
  headers: { 'Idempotency-Key': feedbackKey },
  body: {
    turn_id: turn.id,
    rating: matched ? 'positive' : 'negative',
    comment: matched ? 'The answer is correct.' : 'Expected exactly 42.',
    correction: matched ? null : '42',
    source: { type: 'evaluator', id: 'arithmetic-exact-match-v1' },
  },
});
console.log({ feedback_id: feedback.id, rating: feedback.rating });

client.post is the official client's low-level HTTP method. /feedback is a Rebyte extension and does not require an OpenAI-Beta header or an additional SDK package. A new assessment returns HTTP 201; an identical retry with the same key returns the original assessment with HTTP 200.

3. Read the stored assessment

const saved = await client.get(`/feedback/${feedback.id}`);
const page = await client.get('/feedback', { query: { turn_id: turn.id } });
console.log({ saved, assessments: page.data });

Run the file:

node feedback.mjs

The output includes the task's turn_… ID, its answer, and a feedback_… record containing the rating, source, and linked Agent and Session IDs. The assessment remains available while the Session exists. The example leaves these resources in place for inspection. To clean up automatically, add these lines after the read step before running the file. To remove resources from an earlier run, substitute its saved Session and Agent IDs:

await client.beta.agents.sessions.delete(session.id);
await client.beta.agents.delete(agent.id);

Session deletion makes its feedback inaccessible through the API.

Connect feedback to your product

For example, a user reviewing a sales report could submit negative, explain that refunded orders are missing, and provide the correction “Include refunded orders and show refund amounts separately.” This gives the next review something specific to investigate.

Your backend must check that the user can access the Turn before forwarding the assessment with your organization key. source.id is attribution supplied by your application, not authentication. Keep one idempotency key per assessment: network retries reuse it; a new judgment gets a new key.

Close the loop in your application

Start with a small review process. Keep the task inputs, Turn IDs and your configuration revision in your application. Use the feedback list for each Turn to find repeat failures; the API currently requires a turn_id filter.

Turn representative failures into regression tasks. Revise your saved Agent's instructions, tool definitions or model choice, then create new Sessions to compare the old and new configurations on the same tasks. A saved Agent update does not change existing Sessions. Adopt the revision only after your checks pass, and continue collecting feedback on new Turns.

This is the improvement part of the loop you implement today. Rebyte provides the execution and feedback records; it does not yet provide an automatic feedback-to-training or feedback-to-release pipeline.

See the Feedback API reference for validation, pagination and retry rules, Agent configuration for settings, and execution traces to investigate a failure.