# Authentication
Source: https://docs.collinear.ai/api-reference/authentication

API key setup and authentication for the SimLab API

All API requests require an API key passed via the `API-Key` header.

```
API-Key: <your-api-key>
```

## Obtaining a Key

Via the CLI:

```bash theme={null}
simlab auth login
```

This opens [platform.collinear.ai](https://platform.collinear.ai) where you can find your key under **Developers → API Keys**. The key is stored in `~/.config/collinear/config.toml`.

Or set it as an environment variable:

```bash theme={null}
export COLLINEAR_API_KEY="<your-api-key>"
```

## Error Responses

| Status | Meaning                                    |
| ------ | ------------------------------------------ |
| `400`  | Invalid request - see response for details |
| `401`  | Invalid or missing API key                 |
| `403`  | Key valid but insufficient permissions     |
| `503`  | Service temporarily unavailable            |


# Rollouts
Source: https://docs.collinear.ai/api-reference/rollouts

List rollouts and retrieve rollout artifacts

**Base URL:** `https://rl-gym-api.collinear.ai`

## List Rollouts

List rollouts with optional filtering.

```
GET /rollouts
```

**Query parameters:**

| Parameter     | Type     | Description                                                                      |
| ------------- | -------- | -------------------------------------------------------------------------------- |
| `scenario_id` | `string` | Filter by template                                                               |
| `task_id`     | `string` | Filter by task                                                                   |
| `status`      | `string` | Filter by status: `pending`, `running`, `completed`, `failed`, `workspace_error` |
| `run_id`      | `string` | Filter by parent run                                                             |

**Response:** `200 OK`

```json theme={null}
{
  "total": 1,
  "rollouts": [
    {
      "rollout_id": "rol_001",
      "scenario_id": "hr",
      "task_id": "hr__100_weaver_schedule_phone_screen",
      "run_id": "run_abc123",
      "status": "completed",
      "phase": "completed",
      "current_step": null,
      "max_steps": 30,
      "created_at": "2026-03-20T10:00:00Z",
      "started_at": "2026-03-20T10:00:05Z",
      "completed_at": "2026-03-20T10:02:30Z",
      "result": { "success": true, "message": "All criteria met" },
      "termination_reason": null,
      "model_config": {
        "model": "gpt-4o",
        "provider": "openai",
        "temperature": 0.7,
        "max_steps": 30
      },
      "status_events": [
        { "status": "pending", "timestamp": "2026-03-20T10:00:00Z" },
        { "status": "running", "timestamp": "2026-03-20T10:00:05Z" },
        { "status": "completed", "timestamp": "2026-03-20T10:02:30Z" }
      ]
    }
  ]
}
```

### Rollout Status Values

| Status            | Meaning                                                    |
| ----------------- | ---------------------------------------------------------- |
| `pending`         | Queued, waiting to start                                   |
| `running`         | Currently executing                                        |
| `completed`       | Finished successfully (check `result` for verifier output) |
| `failed`          | Agent or verifier error                                    |
| `workspace_error` | Environment provisioning failed                            |

### Rollout Phase Values

Granular progress while status is `running`:

| Phase                  | Meaning                                 |
| ---------------------- | --------------------------------------- |
| `building_environment` | Provisioning tool server containers     |
| `seeding`              | Injecting seed data                     |
| `agent_running`        | Agent is executing (see `current_step`) |
| `verifying`            | Running verifier                        |
| `completed`            | Execution finished                      |

## Get Rollout

Get status and metadata for a single rollout.

```
GET /rollouts/{rollout_id}
```

**Response:** `200 OK` — returns a `RolloutResponse` (same schema as items in the list rollouts response).

**Errors:** `404` if rollout not found.

## Get Rollout Artifacts

Download the full `RunArtifacts` captured during a rollout.

```
GET /rollouts/{rollout_id}/artifacts
```

**Query parameters:**

| Parameter           | Type   | Default | Description                                                               |
| ------------------- | ------ | ------- | ------------------------------------------------------------------------- |
| `redact_tool_calls` | `bool` | `false` | When true, strips tool calls and tool responses from `artifacts.messages` |

**Response:** `200 OK`

```json theme={null}
{
  "rollout_id": "rol_001",
  "artifacts": {
    "version": "0.1",
    "task_id": "hr__100_weaver_schedule_phone_screen",
    "task": "Schedule a phone screen with the candidate...",
    "model": "gpt-4o",
    "provider": "openai",
    "messages": [...],
    "steps_taken": 5,
    "max_steps": 30,
    "final_observation": "Phone screen scheduled successfully.",
    "created_at": "2026-03-20T10:00:05Z"
  }
}
```

**Errors:** `404` if rollout not found or artifacts not yet available (rollout still in progress).


# Runs
Source: https://docs.collinear.ai/api-reference/runs

Launch, list, and manage agent evaluation runs

**Base URL:** `https://rl-gym-api.collinear.ai`

## Launch Runs

Start a batch of rollouts (agent evaluation runs) for one or more tasks.

```
POST /runs
```

**Request body:**

```json theme={null}
{
  "scenario_id": "hr",
  "task_ids": ["hr__100_weaver_schedule_phone_screen"],
  "rollout_count": 3,
  "max_parallel": 2,
  "model_config": {
    "model": "gpt-4o",
    "provider": "openai",
    "api_key": "sk-...",
    "temperature": 0.7,
    "max_steps": 30
  }
}
```

| Field               | Type                         | Required | Description                                     |
| ------------------- | ---------------------------- | -------- | ----------------------------------------------- |
| `scenario_id`       | `string`                     | Yes      | Template identifier (e.g. `hr`)                 |
| `task_ids`          | `list[string]`               | Yes      | Task IDs to execute (min 1)                     |
| `model_config`      | `ModelConfig`                | Yes      | LLM configuration for the agent (see below)     |
| `rollout_count`     | `int`                        | No       | Rollouts per task (default: 1)                  |
| `max_parallel`      | `int`                        | No       | Hint for desired parallel rollouts              |
| `run_name`          | `string`                     | No       | Custom run name/ID (must be unique)             |
| `user_model_config` | `ModelConfig`                | No       | Optional LLM config for simulated user persona  |
| `task_overrides`    | `dict[string, TaskOverride]` | No       | Per-task overrides (keys must be in `task_ids`) |

### ModelConfig

| Field                     | Type     | Default    | Description                                                                     |
| ------------------------- | -------- | ---------- | ------------------------------------------------------------------------------- |
| `model`                   | `string` | —          | LLM model identifier (e.g. `gpt-4o`, `claude-sonnet-4-6`)                       |
| `provider`                | `string` | `"custom"` | Provider: `custom`, `openai`, `anthropic`, `aws-bedrock`, `azure-openai`, `gcp` |
| `api_key`                 | `string` | —          | API credential for the provider (required)                                      |
| `base_url`                | `string` | `null`     | Custom API base URL (for proxies or self-hosted)                                |
| `temperature`             | `float`  | `0.7`      | Sampling temperature (0.0–2.0)                                                  |
| `max_steps`               | `int`    | `30`       | Maximum tool-calling turns per rollout (min: 1)                                 |
| `system_prompt`           | `string` | `null`     | Override system prompt injected before rollout                                  |
| `tool_server_url`         | `string` | `null`     | Override URL for the Tool Server                                                |
| `log_prob`                | `bool`   | `false`    | Capture token log-probabilities during rollout                                  |
| `summarize_every_n_turns` | `int`    | `0`        | Summarize tool-call history every N turns (0 = disabled)                        |
| `context_stride`          | `int`    | `3`        | Recent exchanges kept verbatim when summarizing (min: 1)                        |

### TaskOverride

| Field             | Type     | Description                                            |
| ----------------- | -------- | ------------------------------------------------------ |
| `task_json`       | `object` | Full task JSON override (replaces entire on-disk task) |
| `name`            | `string` | Display name override                                  |
| `description`     | `string` | Task prompt override                                   |
| `rubric_markdown` | `string` | Rubric markdown override for the universal verifier    |
| `client_context`  | `dict`   | Opaque caller metadata persisted in artifacts          |

**Response:** `200 OK`

```json theme={null}
{
  "run_id": "run_abc123",
  "run_status": {
    "run_id": "run_abc123",
    "scenario_id": "hr",
    "source": "rl-gym",
    "total": 3,
    "completed": 0,
    "failed": 0,
    "in_progress": 0,
    "pending": 3,
    "running": 0,
    "rollout_ids": ["rol_001", "rol_002", "rol_003"]
  },
  "rollouts": [
    {
      "rollout_id": "rol_001",
      "scenario_id": "hr",
      "task_id": "hr__100_weaver_schedule_phone_screen",
      "run_id": "run_abc123",
      "status": "pending"
    }
  ]
}
```

## List Runs

List runs with optional filtering and pagination.

```
GET /runs
```

**Query parameters:**

| Parameter     | Type           | Default    | Description                                                            |
| ------------- | -------------- | ---------- | ---------------------------------------------------------------------- |
| `scenario_id` | `string`       | —          | Filter by template                                                     |
| `source`      | `string`       | `"rl-gym"` | Filter by source: `rl-gym`, `task-editor`, `datadog-synthetics`, `all` |
| `limit`       | `int`          | `50`       | Results per page (1–200)                                               |
| `offset`      | `int`          | `0`        | Pagination offset                                                      |
| `expand`      | `list[string]` | —          | Pass `rollouts` to populate `rollout_ids`                              |

**Response:** `200 OK`

```json theme={null}
{
  "total": 1,
  "runs": [
    {
      "run_id": "run_abc123",
      "scenario_id": "hr",
      "source": "rl-gym",
      "total": 3,
      "completed": 2,
      "failed": 0,
      "in_progress": 1,
      "pending": 0,
      "running": 1,
      "rollout_ids": []
    }
  ]
}
```

## Get Run Status

Get aggregate status for a run, including per-rollout breakdowns.

```
GET /runs/{run_id}
```

**Response:** `200 OK` — returns a `RunStatusResponse` (same schema as items in the list runs response).

**Errors:** `404` if run not found.

## Cancel Run

Signal all active rollouts in a run to cancel gracefully.

```
POST /runs/{run_id}/cancel
```

**Response:** `200 OK`

```json theme={null}
{
  "run_id": "run_abc123",
  "cancelled_rollouts": 2
}
```

## Delete Run

Delete a run and all associated rollout records.

```
DELETE /runs/{run_id}
```

**Response:** `200 OK`

```json theme={null}
{
  "run_id": "run_abc123",
  "deleted_rollouts": 3
}
```

**Errors:** `404` if run not found.


# Scenarios
Source: https://docs.collinear.ai/api-reference/scenarios

List templates, tasks, rubrics, NPC profiles, seed data, and verifier bundles

**Base URL:** `https://rl-gym-api.collinear.ai`

## List Templates

List all available templates (scenario presets).

```
GET /scenarios
```

**Response:** `200 OK`

```json theme={null}
[
  {
    "scenario_id": "hr_recruiting",
    "name": "HR Recruiting Tasks",
    "description": "End-to-end recruiting workflow tasks",
    "num_tasks": 12,
    "num_npcs": 10,
    "tool_servers": [
      { "name": "hrms", "server_type": "http" },
      { "name": "email", "server_type": "http" }
    ]
  }
]
```

## List Tasks for a Template

Get all tasks within a template, including instructions, tool server config, rubrics, and seed data.

```
GET /scenarios/{scenario_id}/tasks
```

**Path parameters:**

| Parameter     | Type   | Description                                |
| ------------- | ------ | ------------------------------------------ |
| `scenario_id` | string | Template identifier (e.g. `hr_recruiting`) |

**Response:** `200 OK`

```json theme={null}
{
  "scenario_id": "hr_recruiting",
  "tasks": [
    {
      "task_id": "hr-102",
      "name": "Schedule Initial Candidate Screening",
      "version": "1.0",
      "category": "recruiting",
      "difficulty": "medium",
      "visibility": "public",
      "description": "Schedule an initial screening call with the candidate...",
      "tool_servers": [
        { "name": "email", "tool_server_url": "https://<environment-url>/email" },
        { "name": "calendar", "tool_server_url": "https://<environment-url>/calendar" }
      ],
      "rubric": {
        "task_id": "hr-102",
        "title": "Schedule Initial Candidate Screening",
        "overview": "Verify the agent correctly schedules...",
        "success_criteria": ["Calendar event created with correct time"],
        "scoring": [
          { "label": "Full Credit", "description": "All requirements met" },
          { "label": "Partial Credit", "description": "Calendar created but no notification" },
          { "label": "No Credit", "description": "Task not completed" }
        ]
      },
      "npc_profiles": [
        {
          "profile_id": "lisa-anderson",
          "first_name": "Lisa",
          "last_name": "Anderson",
          "email": "lisa.anderson@techcorp.com",
          "occupation": "VP of HR / CHRO"
        }
      ]
    }
  ]
}
```

## Get Task Rubric

Get the evaluation rubric for a specific task.

```
GET /scenarios/{scenario_id}/tasks/{task_id}/rubric
```

**Response:** `200 OK`

```json theme={null}
{
  "task_id": "hr-102",
  "title": "Schedule Initial Candidate Screening",
  "overview": "Verify the agent correctly schedules a screening call...",
  "success_criteria": ["Calendar event exists with correct attendees and time"],
  "testing_requirements": ["Verify calendar event was created", "Verify email was sent"],
  "quality_criteria": ["Professional tone in email"],
  "notes": ["Agent should not double-book the time slot"],
  "scoring": [
    { "label": "Full Credit", "description": "All requirements met" },
    { "label": "Partial Credit", "description": "Partial completion" },
    { "label": "No Credit", "description": "Task not completed" }
  ]
}
```

## List NPC Profiles

Get the NPC (Non-Playable Character) profiles for a template. NPCs are simulated users that interact with the agent in real time (e.g., a hiring manager who responds to messages).

```
GET /scenarios/{scenario_id}/npcs
```

**Response:** `200 OK`

```json theme={null}
{
  "scenario_id": "hr_recruiting",
  "npc_profiles": [
    {
      "profile_id": "lisa-anderson",
      "first_name": "Lisa",
      "last_name": "Anderson",
      "email": "lisa.anderson@techcorp.com",
      "occupation": "VP of HR / CHRO",
      "public_info": "15 years in HR leadership..."
    }
  ]
}
```

## List Seed Data Files

List available seed data files for a template.

```
GET /scenarios/{scenario_id}/seed_data
```

**Response:** `200 OK`

```json theme={null}
{
  "scenario_id": "hr",
  "files": ["leave_policy.md", "org_chart.md", "onboarding_policy.md"]
}
```

## Get Seed Data File

Get the contents of a specific seed data file.

```
GET /scenarios/{scenario_id}/seed_data/{filename}
```

**Path parameters:**

| Parameter     | Type   | Description                               |
| ------------- | ------ | ----------------------------------------- |
| `scenario_id` | string | Template identifier                       |
| `filename`    | string | File path within the seed\_data directory |

**Response:** `200 OK`

```json theme={null}
{
  "scenario_id": "hr",
  "filename": "leave_policy.md",
  "content": "# Leave Policy\n\nEmployees are entitled to..."
}
```

**Errors:** `400` if filename contains path traversal, `404` if file not found.

## Download Verifier Bundle

Download the verifier modules for a template as a ZIP archive. The ZIP contains an importable Python package structure.

```
GET /scenarios/{scenario_id}/verifiers/bundle
```

**Response:** `200 OK` — binary ZIP file (`Content-Type: application/zip`)

**Errors:** `404` if scenario not found or no verifiers available.


# BaseAgent
Source: https://docs.collinear.ai/api-reference/sdk/base-agent

The core contract for building custom agents

SimLab is agent-agnostic. The default reference agent uses a tool-calling loop with any LLM API (via LiteLLM), but you can bring your own agent by implementing the `BaseAgent` contract. Install via:

```bash theme={null}
pip install simulationlab
```

## Contract

Your agent extends `BaseAgent` and implements four methods:

```python theme={null}
from simlab.agents import BaseAgent, BaseEnvironment, RunArtifacts, ToolCall

class MyAgent(BaseAgent):
    @staticmethod
    def name() -> str:
        return "my-agent"

    def version(self) -> str | None:
        return "1.0.0"

    def setup(self, environment: BaseEnvironment) -> None:
        """Called once before run(). Use to discover tools, warm up models, etc."""
        self.tools = environment.list_tools()

    def run(
        self,
        instruction: str,
        environment: BaseEnvironment,
        context: RunArtifacts,
    ) -> None:
        """Execute the task using a think-act-observe loop."""
        # 1. Send the instruction to your LLM along with available tools
        messages = [{"role": "user", "content": instruction}]
        context.record_message("user", instruction)

        max_steps = context.max_steps or 20
        for step in range(max_steps):
            # 2. Think — ask your LLM to decide the next action
            llm_response = self.call_llm(messages, tools=self.tools)

            if llm_response.is_done:
                # LLM decided the task is complete
                context.set_final_observation(llm_response.text)
                break

            # 3. Act — execute the tool call the LLM chose
            tool_call = ToolCall(
                tool_server=llm_response.tool_server,
                tool_name=llm_response.tool_name,
                parameters=llm_response.parameters,
            )
            result = environment.call_tool(
                tool_call.tool_server,
                tool_call.tool_name,
                tool_call.parameters,
            )

            # 4. Observe — record the result and feed it back to the LLM
            context.record_tool_call(tool_call, result)
            messages.append({"role": "assistant", "content": f"Called {tool_call.tool_name}"})
            messages.append({"role": "tool", "content": str(result.observation)})
```

<Note>`self.call_llm()` is your own LLM integration — use any provider (OpenAI, Anthropic, etc.). SimLab is model-agnostic.</Note>

## Lifecycle

1. The runner calls `setup()` once, then `run()`.
2. If the agent exceeds the timeout, the run is terminated and artifacts captured up to that point are saved.

## Running Your Agent

Register your agent at runtime via the CLI:

```bash theme={null}
simlab tasks run --env my-env --task my-task --agent-import-path path.to.agent:MyAgent
```

If `--agent-import-path` is omitted, the CLI uses the built-in reference agent.


# BaseEnvironment
Source: https://docs.collinear.ai/api-reference/sdk/base-environment

The environment abstraction for interacting with tool servers

The environment is the interface your agent uses to discover and call tools at runtime.

## Methods

| Method                                                 | Description                                                                          |
| ------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| `async alist_tools(tool_server=None)`                  | Returns tool definitions (all servers, or a specific one)                            |
| `async acall_tool(tool_server, tool_name, parameters)` | Invokes a tool and returns a `ToolCallResult`                                        |
| `list_tool_namespaces()`                               | Returns `list[ToolNamespace]` describing available tool servers and their transports |
| `list_tools(tool_server=None)`                         | **Deprecated (removal in 0.4.0)** — sync wrapper around `alist_tools()`              |
| `call_tool(tool_server, tool_name, parameters)`        | **Deprecated (removal in 0.4.0)** — sync wrapper around `acall_tool()`               |

<Note>Since `BaseAgent.setup()` and `run()` are synchronous, the deprecated sync wrappers (`list_tools`, `call_tool`) are still usable from agent code. They run the async methods internally via a compatibility shim.</Note>

## UnifiedToolEnvironment

SimLab ships `UnifiedToolEnvironment`, which implements `BaseEnvironment` over HTTP and MCP transports against the [Tool Server Protocol](/api-reference/tool-server-protocol).

```python theme={null}
from simlab.agents import UnifiedToolEnvironment

env = UnifiedToolEnvironment(
    tool_servers={
        "email": "https://<environment-url>/email",
        "calendar": "https://<environment-url>/calendar",
    },
    timeout_seconds=30.0,
)

tools = env.list_tools()
result = env.call_tool("email", "send_email", {"to": "...", "subject": "...", "body": "..."})
```


# RunArtifacts
Source: https://docs.collinear.ai/api-reference/sdk/run-artifacts

The data object agents populate during execution

`RunArtifacts` is the structured record of an agent run. Your agent populates it during execution, and verifiers consume it to evaluate task completion.

## Fields

| Field               | Type                   | Required | Description                                            |
| ------------------- | ---------------------- | -------- | ------------------------------------------------------ |
| `version`           | `str`                  | Yes      | Schema version (currently `"0.1"`)                     |
| `task_id`           | `str`                  | Yes      | Task identifier                                        |
| `task`              | `str`                  | Yes      | Task instruction text                                  |
| `model`             | `str`                  | No       | Model used (e.g., `"gpt-4o"`)                          |
| `provider`          | `str`                  | No       | Model provider (e.g., `"openai"`)                      |
| `messages`          | `list[dict]`           | No       | Full message history (role + content)                  |
| `tool_calls`        | `list[ToolCall]`       | No       | All tool invocations made                              |
| `tool_results`      | `list[ToolCallResult]` | No       | Results from each tool call (parallel to `tool_calls`) |
| `metadata`          | `dict`                 | No       | Arbitrary metadata                                     |
| `final_observation` | `str`                  | No       | Agent's final output/summary                           |
| `error`             | `str`                  | No       | Error message if the run failed                        |
| `steps_taken`       | `int`                  | Yes      | Number of tool calls executed                          |
| `max_steps`         | `int`                  | No       | Step limit (if configured)                             |
| `created_at`        | `str`                  | Yes      | ISO 8601 timestamp                                     |

## Helper Methods

| Method                           | Description                                           |
| -------------------------------- | ----------------------------------------------------- |
| `record_message(role, content)`  | Append a message to the history                       |
| `record_tool_call(call, result)` | Record a tool call + result, increments `steps_taken` |
| `set_final_observation(text)`    | Set the agent's final output                          |
| `set_error(text)`                | Record an error                                       |
| `to_dict()`                      | Serialize to a JSON-compatible dict                   |
| `dump(path)`                     | Write artifacts to a JSON file                        |

## ToolCall

```python theme={null}
@dataclass
class ToolCall:
    tool_server: str          # e.g. "email"
    tool_name: str            # e.g. "send_email"
    parameters: dict[str, Any]  # tool-specific parameters
```

## ToolCallResult

```python theme={null}
@dataclass
class ToolCallResult:
    observation: Any          # response from the tool server
    is_error: bool = False    # True if the tool call failed
```


# Task Generation
Source: https://docs.collinear.ai/api-reference/task-generation

Generate tasks from tool definitions

**Base URL:** `https://rl-gym-api.collinear.ai`

Generate task bundles (instructions, rubrics, verifiers) from your tool definitions. Task generation is asynchronous — submit a job, poll for status, then download results.

> **Authentication required.** Task generation endpoints use the `/v1/` prefix and require the `API-Key` header.

## Submit a Task Generation Job

```
POST /v1/task-gen
```

**Request body:**

```json theme={null}
{
  "toolset": [
    {
      "name": "send_email",
      "description": "Send an email to the specified recipient.",
      "inputSchema": {
        "type": "object",
        "properties": {
          "to": { "type": "string" },
          "subject": { "type": "string" },
          "body": { "type": "string" }
        },
        "required": ["to", "subject", "body"]
      }
    }
  ],
  "num_tasks": 10,
  "model": "claude-sonnet-4-6",
  "preset": "recruiting",
  "complexity": { "easy": 0.3, "medium": 0.5, "hard": 0.2 }
}
```

| Field        | Type           | Required | Description                                                                                                                |
| ------------ | -------------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `toolset`    | `list[object]` | Yes      | MCP tool definitions (JSON from a `/tools` endpoint). Each object should include `name`, `description`, and `inputSchema`. |
| `num_tasks`  | `int`          | No       | Number of tasks to generate (default: 10, max: 200)                                                                        |
| `model`      | `string`       | No       | Any OpenAI-compatible model identifier (default: `claude-sonnet-4-6`)                                                      |
| `preset`     | `string`       | No       | Built-in preset (e.g. `"recruiting"`) — auto-fills agent, scenario, workflows                                              |
| `agent`      | `object`       | No       | Agent identity: `{ "role": "...", "description": "..." }`                                                                  |
| `scenario`   | `object`       | No       | Scenario context: `{ "name": "...", "role_label": "...", "conventions": "...", "policies": [...] }`                        |
| `categories` | `list[object]` | No       | Task categories: `[{ "id": "...", "label": "..." }]`                                                                       |
| `workflows`  | `list[object]` | No       | Example workflows: `[{ "name": "...", "steps": ["..."] }]`                                                                 |
| `npcs`       | `list[object]` | No       | NPC roles: `[{ "role": "...", "typical_asks": "..." }]`                                                                    |
| `workspace`  | `object`       | No       | Workspace branding: `{ "email_domain": "...", "agent_email": "..." }`                                                      |
| `complexity` | `object`       | No       | Difficulty distribution: `{ "easy": 0.3, "medium": 0.5, "hard": 0.2 }`                                                     |
| `diversity`  | `object`       | No       | Diversity config for scenario variations                                                                                   |

**Response:** `200 OK`

```json theme={null}
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "pending",
  "progress": null,
  "created_at": "2026-03-07T06:20:00Z"
}
```

## Check Job Status

```
GET /v1/task-gen/{job_id}
```

**Response:** `200 OK`

```json theme={null}
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "running",
  "progress": "Step 4/10: formatting tasks",
  "created_at": "2026-03-07T06:20:00Z"
}
```

| Status      | Meaning                               |
| ----------- | ------------------------------------- |
| `pending`   | Job queued, not yet started           |
| `running`   | Generation in progress                |
| `completed` | Results ready for download            |
| `failed`    | Generation failed (see `error` field) |

## Download Results

```
GET /v1/task-gen/{job_id}/results
```

**Response:** `200 OK` (only when status is `completed`)

```json theme={null}
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "tasks": [
    { "filename": "task_001.json", "content": "{ ... }" }
  ],
  "instructions": [
    { "filename": "task_001_instruction.md", "content": "..." }
  ],
  "rubrics": [
    { "filename": "task_001_rubric.md", "content": "..." }
  ],
  "verifiers": [
    { "filename": "task_001_verifier.py", "content": "..." }
  ],
  "skills_md": "# Skills\n\n..."
}
```

Returns `202` if the job is still running, `404` if not found, `500` if the job failed.

**CLI shortcut:**

```bash theme={null}
simlab tasks-gen run --config config.toml --num-tasks 10
```


# Test Model Configuration
Source: https://docs.collinear.ai/api-reference/test-model-config

Validate a model configuration with a test completion

**Base URL:** `https://rl-gym-api.collinear.ai`

Validate a model configuration by sending a test completion request.

```
POST /test-model-config
```

**Request body:** A `ModelConfig` object (same schema as in [Launch Runs](/api-reference/runs#modelconfig)).

**Response:** `200 OK`

```json theme={null}
{
  "success": true,
  "message": "Model responded successfully",
  "error_type": null
}
```

| Field        | Type     | Description                                                                                                                             |
| ------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `success`    | `bool`   | Whether the test completion succeeded                                                                                                   |
| `message`    | `string` | Human-readable result summary or error description                                                                                      |
| `error_type` | `string` | Error classification: `authentication_error`, `connection_error`, `timeout`, `invalid_model`, `rate_limit`, `bad_request`, or `unknown` |


# Tool Server Protocol
Source: https://docs.collinear.ai/api-reference/tool-server-protocol

HTTP interface that all tool servers expose

Every tool server (coding, email, calendar, etc.) exposes the same HTTP interface. This is what agents interact with at runtime.

**Base URL:** Provided per-task in the `tool_servers` field of the task response.

## List Tools

Get available tools and their parameter schemas.

```
GET /tools
```

**Response:** `200 OK`

```json theme={null}
{
  "tools": [
    {
      "name": "send_email",
      "description": "Send an email to the specified recipient.",
      "parameters": {
        "type": "object",
        "properties": {
          "to": { "type": "string", "description": "Recipient email address" },
          "subject": { "type": "string", "description": "Email subject line" },
          "body": { "type": "string", "description": "Email body text" }
        },
        "required": ["to", "subject", "body"]
      }
    }
  ]
}
```

## Execute a Tool

Invoke a tool action and get the result.

```
POST /step
```

**Request body:**

```json theme={null}
{
  "action": {
    "tool_name": "send_email",
    "parameters": {
      "to": "candidate@example.com",
      "subject": "Interview Scheduling",
      "body": "Hello, we'd like to schedule..."
    }
  }
}
```

**Response:** `200 OK`

```json theme={null}
{
  "observation": "Email sent successfully to candidate@example.com"
}
```

**Error response:**

```json theme={null}
{
  "observation": "Error: invalid email address format",
  "is_error": true
}
```


# Verifier Output
Source: https://docs.collinear.ai/api-reference/verifier-output

Output format from verifier evaluation

After an agent run, verifiers evaluate whether the task was completed successfully. Verifiers consume [RunArtifacts](/api-reference/sdk/run-artifacts) and return a `VerifierResult`.

## VerifierResult

```python theme={null}
class VerifierResult:
    success: bool     # Whether the task was completed successfully
    message: str      # Human-readable explanation
    output: str       # Additional detail (defaults to message if empty)
```

The `result` field on rollout responses contains the serialized verifier output:

```json theme={null}
{
  "success": true,
  "message": "All criteria met"
}
```

## Rubric Judge

For rubric-based evaluation, the `RubricJudgeResult` provides structured scoring:

| Field              | Type           | Description                                                                     |
| ------------------ | -------------- | ------------------------------------------------------------------------------- |
| `score`            | `float`        | Score from 0.0 to 1.0                                                           |
| `verdict`          | `string`       | `"PASS"` or `"FAIL"`                                                            |
| `confidence`       | `float`        | Confidence in the verdict (0.0–1.0)                                             |
| `evidence`         | `list[string]` | Bullet points with concrete evidence                                            |
| `failed_criteria`  | `list[string]` | Unmet rubric criteria                                                           |
| `dimension_scores` | `list[object]` | Per-dimension breakdowns: `{ "dimension": str, "score": float, "reason": str }` |
| `error`            | `string`       | Error message if evaluation failed                                              |


# NPCs
Source: https://docs.collinear.ai/core-concepts/npcs

Simulated users with configurable traits and personas

NPCs (non-player characters) are simulated users that populate the playground and interact with or are referenced by tasks. Each NPC has defined interests and personality traits that influence how they behave in the simulation.

NPCs serve two roles in the Simulation Lab:

1. **Task context** — NPCs appear as employees, customers, or other personas in seed data, giving the agent realistic people to interact with.
2. **User simulation** — NPCs can act as simulated end-users that the agent must serve, with varying traits that test the agent's robustness.

## TraitBasis

TraitBasis is the system for generating diverse, coherent NPC personas. It defines personas across multiple dimensions:

```python theme={null}
trait_basis = {
    "ages": ["25"],
    "genders": ["female"],
    "occupations": ["Employed"],
    "intents": ["check_flight_status"],
    "traits": {"patience": [1]},
    "locations": ["USA"],
    "languages": ["English"],
    "tasks": ["airline support"],
}
```

### Trait Dimensions

* **Demographics** — Age, gender, location, language, occupation.
* **Intents** — What the user is trying to accomplish.
* **Personality traits** — Configurable on a scale (e.g., patience, confusion, skepticism, incoherence).
* **Tasks** — The domain context for the interaction.

To learn more about TraitBasis, see the [research paper](https://arxiv.org/abs/2510.04491) and the [open-source repo](https://github.com/collinear-ai/simulations).

### Why TraitBasis Matters

By varying trait combinations, you can systematically test how your agent handles:

* **Different user personas** — Does the agent perform equally well for all demographics?
* **Challenging interactions** — How does it handle impatient or confused users?
* **Fairness** — Are there trait combinations where the agent underperforms?

TraitBasis-generated personas are used across task generation and NPC behavior, ensuring consistent and controllable diversity throughout the simulation.

To learn more about TraitBasis, read our [paper](https://arxiv.org/abs/2510.04491) and try out the [repo](https://github.com/collinear-ai/simulations)!


# Sandbox
Source: https://docs.collinear.ai/core-concepts/sandbox

Isolated execution environments for agent rollouts

Each rollout runs in its own sandbox — an isolated workspace with its own Docker network and containers. The sandbox is the unit of multi-tenancy in Simulation Lab.

## Workspace Isolation

A sandbox provides:

* **Isolated network** — Each rollout gets its own Docker network with dedicated containers. Parallel rollouts cannot interfere with each other.
* **Clean state** — Every rollout starts from a freshly provisioned environment with seed data applied from scratch.
* **Per-rollout configuration** — Environment variables and configuration are scoped to the individual rollout.

When a rollout finishes, the sandbox is torn down. When the next rollout starts, a fresh sandbox is provisioned from scratch.

## Local Sandboxes

By default, sandboxes run locally via Docker Compose. The `simlab env up` command starts the environment, and each `simlab tasks run` creates an isolated workspace within it.

```bash theme={null}
simlab env up my-env
```

## Remote Sandboxes (Daytona)

For cloud-based execution, Simulation Lab supports remote sandboxes via Daytona. This is useful for running evaluations at scale without local compute constraints.

```bash theme={null}
simlab env up my-env --daytona
```

The Daytona API key is resolved from `--daytona-api-key`, `[daytona].api_key` in config.toml, `SIMLAB_DAYTONA_API_KEY`, or `DAYTONA_API_KEY`.

## Why Sandboxes Matter

Sandbox isolation is what makes evaluation results trustworthy:

* **Parallel execution** — Run multiple rollouts concurrently to gather statistical confidence without interference.
* **Reproducibility** — Same environment, same seed data, same starting conditions every time.
* **Safety** — Agents interact with simulated services in isolation. Nothing leaks to production systems or other rollouts.


# Seed Data
Source: https://docs.collinear.ai/core-concepts/seed-data

Domain-specific data injected into environments to create realistic starting conditions

Seed data populates simulation environments with realistic records before the agent runs. There are two layers of seeding:

## Baseline Seed Data

When an environment starts (`simlab env up`), each tool server is seeded with baseline data — the standing records that exist independent of any task. For example, an HR environment might include:

* Employee records in HRIS
* Company policies and org charts
* Existing calendar events and email threads

Baseline seed data is defined by the scenario template and runs automatically via each tool's seed services.

## Task-Specific Seed Data

Each task injects additional data on top of the baseline to set up the specific scenario the agent must solve. This is what makes the task solvable — the agent discovers the seeded data through normal tool use.

Examples of task-specific seed data:

* An email from a hiring manager requesting an interview be scheduled
* A calendar event the agent needs to reschedule
* A chat message with a candidate question that needs a response

Task-specific seed data is defined in the task configuration and injected fresh at the start of each rollout, ensuring clean state across attempts.

## Why Seed Data Matters

Seed data is what makes simulation environments realistic rather than empty sandboxes. It provides:

* **Grounding** — Tasks reference real data present in the environment, so agents must read and reason over actual records.
* **Reproducibility** — The same seed data produces the same starting conditions across rollouts, enabling fair comparison between agents or models.
* **Isolation** — Each rollout starts from a known state, so results from one run don't leak into the next.


# Task Generator
Source: https://docs.collinear.ai/core-concepts/task-generator

How Simulation Lab generates tasks from templates, seed data, NPCs, and toolsets

The Task Generator produces complete task bundles by combining four inputs through an iterative generation process.

<img />

## Inputs

* **Template** (`task.json`) — The structural blueprint that defines the task format and constraints.
* **Seed Data** — Domain-specific records injected into the playground (e.g., employee records, resumes, job requisitions).
* **NPCs** — Simulated users with defined interests and personality traits that interact with or are referenced by the task.
* **Toolset** — The available tools described by their descriptions, schemas, and dynamics.

## Iterative Task Generation

The generator combines these inputs and iteratively improves the output for:

* **Grounding in seed data** — Tasks reference real data present in the playground.
* **Realism** — Instructions reflect plausible real-world scenarios.
* **Feasibility with toolset** — Tasks are solvable using the available tools.
* **Diversity** — Generated tasks cover a range of scenarios and difficulty levels.

## Skills.md

Each task is paired with a `Skills.md` file that provides the agent with:

* **Scenario context** — Background information about the task domain and situation.
* **Recurring workflows** — Common patterns and procedures the agent should follow.

## Task Bundle Output

A generated task bundle contains:

* **Task metadata** — Unique ID, display name, difficulty, and category.
* **Task instruction** — The natural language instruction given to the agent.
* **Tool servers** — Which tool servers are required.
* **Seed data** — Data injected into the playground before the agent starts.
* **Verifiers** — Automated checks that determine whether the agent succeeded.

See [Tasks & Verifiers](/simulation-lab/tasks-and-verifiers) for the full bundle format and how to run generated tasks.


# Verifiers
Source: https://docs.collinear.ai/core-concepts/verifiers

Programmatic and rubric-based evaluation of agent behavior. 

For each task, Collinear’s **Verifier Engine** generates two complementary sets of verifiers that together cover the full evaluation surface:

## Programmatic Verifiers

Programmatic verifiers inspect the playground state directly. They compare before/after snapshots of the playground to confirm the agent made the correct changes.

**Example:** "Did the agent send an email to the correct recipient?" is answered by querying the email tool server's state and reviewing the state diff.

Programmatic verifiers are deterministic — given the same playground state, they always produce the same result. Use them for objective, checkable criteria.

## Rubric-based Reward Models

Rubric-based verifiers use reward models to evaluate the agent's actions against a scoring rubric. The judge reviews the agent's full trace and assigns a reward score.

**Example:** "Did the agent communicate professionally?" is evaluated by an LLM reviewing the conversation against a rubric defining professional communication.

Rubric-based verifiers are useful for:

* Subjective quality criteria (tone, clarity, helpfulness)
* Multi-step reasoning evaluation
* Cases where the "correct" answer depends on judgment

## How Verifiers Produce Rewards

Both verifier types produce structured results:

* **Pass/fail** — Did the agent complete the task successfully?
* **Reward signal** — A numeric score (typically 0.0 to 1.0) indicating quality of completion.
* **Metadata** — Verifier-specific details (which checks passed, which failed, and why).


# What is Collinear?
Source: https://docs.collinear.ai/introduction

An interactive playground where agents learn new skills and improve existing capabilities.

Collinear is building the Simulation Lab: an interactive playground where agents learn new skills and improve existing capabilities. We simulate thousands of real-world users, tools, and workflows — so your agents can fail, learn, and improve before they ever touch production.

Collinear provides isolated, reproducible playgrounds populated with realistic simulated applications, generates tasks and automated verifiers, and produces grounded reward signals by running agents through multiple attempts per task. The result is a continuous loop: evaluate agent behavior, identify gaps, and iterate — all before deploying to real users.

<img />

## Why use a Simulation Lab?

**Without a Simulation Lab:**

* **Little to no control on agent environment:** You're testing against live or mock systems that don't reflect real-world complexity.
* **Hard to reproduce or predict outcomes:** Flaky results make it difficult to pinpoint what went wrong or why.
* **Static signal and rewards:** Evaluation criteria are fixed and can't adapt to evolving agent capabilities.

**With a Simulation Lab:**

* **Fully controlled behavior and actions:** Every tool, user, and workflow runs in an isolated, deterministic playground you configure.
* **Reproducible and predictable outcomes:** Run the same scenario thousands of times with clean state to get statistically meaningful results.
* **Dynamic signal and rewards:** Verifiers and reward functions evolve alongside your agent, giving you grounded feedback at every iteration.

## Features

<CardGroup>
  <Card title="Scalability" icon="gauge-high">
    IMPALA system design so GPUs stay saturated during async rollouts
  </Card>

  <Card title="Fidelity" icon="bullseye">
    Train and evaluate on realistic world interactions
  </Card>

  <Card title="Adaptability" icon="shuffle">
    Dynamically compose scenarios, tasks, and verifiers
  </Card>
</CardGroup>

## Use Cases

<CardGroup>
  <Card title="Surface failure modes early" icon="magnifying-glass">
    Discover real-world failures before they reach production
  </Card>

  <Card title="Close the loop" icon="arrows-rotate">
    Hill-climb on existing capabilities through iterative evaluation
  </Card>

  <Card title="Expand to new capabilities" icon="rocket">
    Extend your agent into new domains and skill sets
  </Card>
</CardGroup>


# Bring Your Own Agent
Source: https://docs.collinear.ai/simulation-lab/bring-your-own-agent

Integrate your own agent implementation with Simulation Lab

Simulation Lab is agent-agnostic. The default agent uses a tool-calling loop with any LLM API (via [LiteLLM](https://docs.litellm.ai/)), but you can bring your own agent implementation by implementing the `BaseAgent` contract.

## The Agent Contract

A custom agent extends `BaseAgent` and implements four methods:

```python theme={null}
class MyAgent(BaseAgent):
    @staticmethod
    def name() -> str:
        return "my-agent"

    def version(self) -> str | None:
        return "1.0.0"

    def setup(self, environment: BaseEnvironment) -> None:
        """Called once before the agent starts.

        Use this to discover available tools, configure state,
        or perform any one-time initialization.
        """
        self.tools = environment.list_tools()

    def run(
        self,
        instruction: str,
        environment: BaseEnvironment,
        context: RunArtifacts,
    ) -> None:
        """Execute the agent's task.

        Populate the context (RunArtifacts) as execution progresses
        so that partial results are captured even on timeout or error.
        """
        context.record_message("user", instruction)

        # Your agent logic here:
        # 1. Read the instruction
        # 2. Decide which tools to call
        # 3. Call tools via environment.call_tool(server, name, params)
        # 4. Record each tool call and result in context
        # 5. Iterate until done

        call = ToolCall(tool_server="email-env", tool_name="send_email", parameters={...})
        result = environment.call_tool(call.tool_server, call.tool_name, call.parameters)
        context.record_tool_call(call, result)
```

Register your agent at runtime via the CLI:

```bash theme={null}
simlab tasks run --env my-env --task my-task --agent-import-path path.to.agent:MyAgent
```

## The Environment Interface

The `environment` object passed to your agent provides:

* **`environment.list_tools(tool_server=None)`** — returns tool schemas (names, descriptions, input schemas) for one server or all servers in the workspace.
* **`environment.call_tool(tool_server, tool_name, parameters)`** — executes a tool call on a specific server and returns a `ToolCallResult`.
* **`environment.tool_servers`** — property returning a `dict[str, str]` mapping server names to base URLs.

Under the hood, these map to the tool server protocol: `list_tools()` calls `GET /tools`, and `call_tool()` calls `POST /step`.

## Run Artifacts

As the agent executes, it populates a `RunArtifacts` object — the structured record of the run (conversation history, tool calls, results, errors). Helper methods (`record_message`, `record_tool_call`, `set_error`) allow incremental recording so that partial results are captured even on timeout or error.

Verifiers consume `RunArtifacts` to determine whether the agent succeeded. This is the contract between agent execution and verification — your agent populates it, and verifiers read from it.


# Getting Started
Source: https://docs.collinear.ai/simulation-lab/getting-started

Install the CLI and run your first evaluation

Install the CLI and run your first evaluation against a simulated environment.

## Prerequisites

* Python 3.13
* A Collinear API key from [platform.collinear.ai](https://platform.collinear.ai) (Developers → API Keys)
* An API key for any [LiteLLM-supported model provider](https://docs.litellm.ai/docs/providers) (OpenAI, Anthropic, Google, etc.)
* **One of the following** for running environments:
  * A [Daytona](https://app.daytona.io) API key — for fast, ephemeral remote sandboxes (recommended)
  * Docker Desktop (or Docker Engine with Compose) — for local execution

## Installation

```bash theme={null}
uv tool install --python 3.13 "simulationlab[daytona]"
```

The PyPI package is named `simulationlab`. The installed CLI command is `simlab`.

## Authentication

Log in with your Collinear API key:

```bash theme={null}
simlab auth login
```

This saves your key to `~/.config/simlab/config.toml`.

Then export your model provider key:

```bash theme={null}
# Use whichever provider you prefer — SimLab uses LiteLLM under the hood.
export SIMLAB_AGENT_API_KEY="your-api-key"

# Optional: export Daytona key if using remote sandboxes
export DAYTONA_API_KEY="dtn_..."
```

### Supported providers

SimLab supports any [LiteLLM-compatible provider](https://docs.litellm.ai/docs/providers). Here are common examples:

| Provider  | Model format                         | `SIMLAB_AGENT_API_KEY` | Verifier `provider` value |
| --------- | ------------------------------------ | ---------------------- | ------------------------- |
| OpenAI    | `gpt-4o`                             | Your OpenAI API key    | `openai`                  |
| Anthropic | `anthropic/claude-sonnet-4-20250514` | Your Anthropic API key | `anthropic`               |
| Google    | `gemini/gemini-2.5-pro`              | Your Google AI API key | `gemini`                  |

The model format follows LiteLLM conventions: `<provider>/<model_name>`. OpenAI models don't require the provider prefix since it's the default.

**Full example using Anthropic:**

```bash theme={null}
export SIMLAB_AGENT_API_KEY="sk-ant-..."

simlab tasks run --env my-env \
  --task hr__0_weaver_flag_biased_compensation_adjustment_request \
  --agent-model anthropic/claude-sonnet-4-20250514 \
  --agent-api-key "$SIMLAB_AGENT_API_KEY"
```

## Starting an environment

Initialize an environment from a template and start it:

```bash theme={null}
# Initialize an HR-based scenario environment
simlab env init my-env --template hr
```

> To see all available templates: `simlab templates list`

## Choosing a task

Tasks are organized by the scenario template associated with your environment.

```bash theme={null}
# List tasks for your environment's template
simlab tasks list --env my-env
```

If you generated tasks locally (via `tasks-gen`), browse them directly:

```bash theme={null}
simlab tasks list --tasks-dir ./generated-tasks
```

## Running a rollout

The primary command is `simlab tasks run`. It automatically starts the environment, seeds data, runs the agent, verifies the result, and tears down when done.

**With Daytona (recommended — fast, ephemeral remote sandboxes):**

```bash theme={null}
simlab tasks run --env my-env \
  --task hr__0_weaver_flag_biased_compensation_adjustment_request \
  --daytona \
  --agent-model <model> \
  --agent-api-key "$SIMLAB_AGENT_API_KEY"
```

**Without Daytona (runs locally via Docker — first run may be slow while images pull):**

```bash theme={null}
simlab tasks run --env my-env \
  --task hr__0_weaver_flag_biased_compensation_adjustment_request \
  --agent-model <model> \
  --agent-api-key "$SIMLAB_AGENT_API_KEY"
```

Use any [LiteLLM-supported model](https://docs.litellm.ai/docs/providers) for `--agent-model` (e.g. `gpt-4o`, `anthropic/claude-sonnet-4-20250514`, `gemini/gemini-2.5-pro`).

You can also run tasks with your own agent implementation instead of the built-in one. See [Bring Your Own Agent](/simulation-lab/bring-your-own-agent) for the full interface and setup.

## Viewing results

Results are saved to `output/agent_run_<task_id>_<timestamp>/`:

* `artifacts.json` — full rollout trace (messages, tool calls, observations)
* `verifier/reward.txt` — `1` (pass) or `0` (fail)
* `verifier/reward.json` — e.g. `{"reward": 1.0}`

For more detail, see [Understanding Results](/simulation-lab/understanding-results).

## Configuring verifiers

Generated tasks use rubric-based verifiers that need a model to score results. Configure the verifier before running generated tasks:

```bash theme={null}
export SIMLAB_VERIFIER_MODEL="<provider>/<model>"    # e.g. gpt-4o, anthropic/claude-sonnet-4-20250514
export SIMLAB_VERIFIER_PROVIDER="<provider>"          # e.g. openai, anthropic, gemini
export SIMLAB_VERIFIER_API_KEY="your-api-key"
```

Or in `config.toml`:

```toml theme={null}
[verifier]
model = "<provider>/<model>"
provider = "<provider>"
api_key = "your-api-key"
```

> Built-in tasks use programmatic verifiers and don't require this setup. This is only needed for tasks you generate via `tasks-gen`.


# Overview
Source: https://docs.collinear.ai/simulation-lab/overview

A simulation lab to train, test and refine agents for the real world.

Collinear Simulation Lab is an training, testing, and evaluation platform for AI agents. It provides isolated, reproducible, interactive playgrounds populated with realistic simulated applications, generates tasks and automated verifiers for those environments, and produces grounded reward signals by running agents through multiple attempts per task.


# Scenarios
Source: https://docs.collinear.ai/simulation-lab/scenarios

Scenario templates and composition for simulation scenarios

## Scenario Templates

Templates are named toolset combinations for common scenarios. Browse them with:

```bash theme={null}
simlab templates list
```

| Template             | Included Tools                                |
| -------------------- | --------------------------------------------- |
| `hr`                 | Frappe HRMS, Rocket.Chat, Email, Calendar     |
| `customer-service`   | Frappe Helpdesk, Rocket.Chat, Email, Calendar |
| `coding`             | OpenHands, Bash                               |
| `finance`            | SEC Edgar, Twelve Data, Google Workspace      |
| `personal-assistant` | OpenTable, Outlook, Weather, WhatsApp         |

Templates are a starting point; users can build custom combinations by selecting individual tools during `env init`.

## Scenario Composition

The composition engine takes your selected tools and produces a complete Docker Compose stack:

1. Collects services from each tool's YAML definition (main services, preseed services, seed services)
2. Adds networking — all services join a shared bridge network
3. Maps ports — only agent-facing and web UI ports get host bindings; internal services (databases, MCP servers) stay network-only
4. Detects port conflicts and auto-resolves them
5. Generates a `.env` file with placeholders for any required API keys


# Tasks
Source: https://docs.collinear.ai/simulation-lab/tasks-and-verifiers

Task format, generation flow, and pre-built tasks

Simulation Lab includes a task generation pipeline that produces complete tasks from a configuration file. This is the programmatic path for creating evaluation content at scale.

## Task Generation Config

Task generation is driven by a TOML config file that defines the agent role, available toolsets, scenario conventions, and generation parameters. Initialize one from a preset or create it from scratch:

```bash theme={null}
# View task-gen templates
simlab tasks-gen templates

# Initialize a config from a template
simlab tasks-gen init --template hr --output-dir ./taskgen

# Run generation
simlab tasks-gen run --config ./taskgen/config.toml
```

Here is an example config for an HR scenario:

```toml theme={null}
template = "hr"
categories = [
    { id = "job_requisition", label = "Job requisition" },
    { id = "shortlist_resumes", label = "Shortlist the resumes" },
    { id = "schedule_interviews", label = "Schedule the interviews" },
    { id = "consolidate_feedback", label = "Consolidate feedback" },
    { id = "send_offers_or_rejections", label = "Send out offers or rejections" },
    { id = "negotiate_and_final_offer", label = "Negotiate and send final offer" },
    { id = "handle_candidate_rejections", label = "Handle candidate rejections" },
    { id = "candidate_questions", label = "Handle questions from candidates" },
]

[agent]
role = "HR recruiting coordinator"
description = "Handles end-to-end recruiting workflows: scheduling interviews, managing candidate pipelines, coordinating offer discussions, and communicating with hiring managers and candidates."

[[toolset]]
name = "HRIS"
description = "Query/update employee records, job requisitions, candidate profiles"
operations = ["search", "read", "create", "update"]

[[toolset]]
name = "Email"
description = "Send and read emails"
operations = ["send", "read"]

[[toolset]]
name = "Calendar"
description = "View and manage calendar events"
operations = ["list", "create", "update", "delete"]

[[toolset]]
name = "Chat"
description = "Send messages in Rocket.Chat channels and DMs"
operations = ["send", "read"]

[scenario]
name = "hr"
role_label = "HR recruiting professional"
conventions = """
- Always check all participants' calendars before scheduling
- Never share compensation details in group channels
- Document all candidate interactions in HRIS
- Get manager approval before extending offers
"""
policies = [
    "Interviews must include at least one diverse panel member",
    "Offers require VP approval for >$200k total comp",
    "Candidate data must not be shared outside recruiting team",
]

[generation]
num_tasks = 2
deduplicate = false
filter = true

[generation.complexity]
easy = 0.3
medium = 0.5
hard = 0.2

[generation.diversity]
variations = [
    "straightforward",
    "scheduling_conflict",
    "candidate_experience",
    "handoff_or_reporting",
    "offer_negotiation",
    "candidate_decline",
    "accommodation",
]
```

* **`template`** — See `simlab tasks-gen templates` for all available templates.
* **`categories`** — Task categories to generate across.
* **`[agent]`** — The role and description of the agent being evaluated.
* **`[[toolset]]`** — Tools available to the agent, with allowed operations.
* **`[scenario]`** — Domain conventions, policies, and constraints the agent should follow.
* **`[generation]`** — Number of tasks, complexity distribution, and diversity dimensions.

The config can also define `[[workflows]]` (multi-step procedures) and `[[npcs]]` (non-player characters like hiring managers or candidates) to increase task realism.

## Pre-Built Tasks

For common domains, pre-built tasks are available via the Scenario Manager API. Tasks are associated with templates — when you create an environment from a template, its tasks are automatically available:

```bash theme={null}
simlab tasks list --env my-env
simlab tasks run --env my-env \
  --task hr__0_weaver_flag_biased_compensation_adjustment_request \
  --agent-model <model> \
  --agent-api-key "$SIMLAB_AGENT_API_KEY"
```


# Toolsets
Source: https://docs.collinear.ai/simulation-lab/toolsets

Built-in tools, custom tool definitions, and MCP server integration

## Tool Server Protocol

Tool servers expose a standard HTTP API. Any service that implements this protocol can be used as a tool in the Simulation Lab. See the [Tool Server Protocol](/api-reference/tool-server-protocol) reference for the full spec.

## Multi-Tool Composition

A workspace can include multiple tool servers. The agent sees a unified tool list. Tools are automatically namespaced by their server name (e.g., `email-env__send_email`, `chronos-server__create_event`). Routing is handled transparently; the agent calls tools by name and the Simulation Lab forwards each call to the correct server.

## Tool Definition Format

Each tool server is defined as a YAML file that maps directly to Docker Compose services. Here is the email tool as an example:

```yaml theme={null}
name: email
display_name: Email
description: Email sending and inbox management via SMTP server.
category: communication

exposed_ports:
  - port: 8025
    description: Email web UI

services:
  smtp-server:
    image: <smtp-image>
    ports: ["8025", "1025"]
    healthcheck:
      interval: 10s
      retries: 6

  email-env:
    image: <tool-server-image>
    environment:
      SMTP_BASE_URL: http://smtp-server:8025
    depends_on:
      smtp-server:
        condition: service_healthy
```

The YAML specifies everything Docker needs: images, ports, health checks, environment variables, and dependency ordering. The CLI reads these definitions and composes them into a working `docker-compose.yml`.

For tools that are hosted externally and don't need Docker services, use the `tool_server_url` field instead:

```yaml theme={null}
name: harbor-main
display_name: Harbor Main
description: Custom tool scaffold for harbor-main.
category: custom
tool_server_url: http://localhost:9000
```

## Built-in Toolsets

Below is a selection of out-of-the-box tools that come with Simulation Lab. We are continually adding more domain-specific tools as part of our roadmap.

| Name               | Category      | Description                                        |
| ------------------ | ------------- | -------------------------------------------------- |
| `Calendar`         | Productivity  | CalDAV calendar (Baikal + Chronos MCP)             |
| `Coding`           | Development   | Sandboxed shell and filesystem (OpenHands)         |
| `Email`            | Communication | Email sending/inbox via MailHog SMTP               |
| `File Explorer`    | Development   | Read, write, and list files                        |
| `Frappe HRMS`      | Enterprise    | Full HR management system (Frappe HRMS)            |
| `Google Workspace` | Productivity  | Drive, Docs, Sheets, Gmail, Calendar               |
| `Nano Bananna Pro` | Creative      | Image generation and editing                       |
| `OpenTable`        | Lifestyle     | Restaurant search, reservations, and cancellations |
| `Outlook`          | Communication | Email inbox, drafts, and sent items                |
| `Playwright`       | Browser       | Browser automation via Playwright MCP              |
| `Rocket.Chat`      | Communication | Open source Slack-equivalent Team chat             |
| `SEC Edgar`        | Finance       | SEC EDGAR filings search                           |
| `Terminal`         | Development   | Run shell commands                                 |
| `Twelve Data`      | Finance       | Market data                                        |
| `Weather`          | Utilities     | Forecasts by location                              |
| `WhatsApp`         | Communication | Messaging with contacts, groups, and messages      |

## Bring Your Own Tools

Simulation Lab supports three paths for integrating custom tools: **MCP servers**, **env-local custom tool definitions**, and **custom CLI tools** for coding environments.

### Custom MCP Servers

You can add custom MCP servers to any environment so the agent can call them alongside built-in tools. SimLab supports two MCP transport patterns:

* **URL-based** — SimLab connects directly to an HTTP MCP endpoint. No extra container is added.
* **Command-based** — SimLab adds an `mcp-gateway` container that starts your stdio MCP servers and exposes them over HTTP inside the environment.

#### Configuration format

Create a `mcp-servers.json` file with a top-level `mcpServers` object. Each server must define exactly one of `url` or `command`:

```json theme={null}
{
  "mcpServers": {
    "docs": {
      "url": "https://example.com/mcp"
    },
    "weather": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/adhikasp/mcp-weather.git", "mcp-weather"],
      "env": {
        "ACCUWEATHER_API_KEY": "your_api_key_here"
      }
    }
  }
}
```

| Field     | Required                  | Description                                                     |
| --------- | ------------------------- | --------------------------------------------------------------- |
| `url`     | One of `url` or `command` | HTTP endpoint for URL-based servers                             |
| `command` | One of `url` or `command` | Executable for command-based (stdio) servers                    |
| `args`    | No                        | Arguments passed to the command                                 |
| `env`     | No                        | Environment variables (e.g. API keys) for command-based servers |

**Naming rules:**

* Server names may contain only letters, numbers, `_`, and `-`.
* Names must not collide with built-in tool server names (e.g. `email`, `calendar`).

#### Adding MCP servers to an environment

Pass your config at `env init`:

```bash theme={null}
# URL-based only
simlab env init my-env --mcp-servers ./mcp-servers.json --non-interactive

# Combined with a template
simlab env init my-env --template hr --mcp-servers ./mcp-servers.json

# Command-based (adds an mcp-gateway container automatically)
simlab env init my-env --mcp-servers ./mcp-servers-command.json --non-interactive
```

After init, SimLab persists the config as `environments/<env-name>/mcp-servers.json`. For command-based servers, it also generates an `mcp-gateway` service in `docker-compose.yml`.

#### Managing API keys and env vars

For command-based servers, set secrets in `environments/<env-name>/.env` before running `simlab env up`.

If an env var name is used by **only one** MCP server, set it directly:

```bash theme={null}
# environments/my-env/.env
ACCUWEATHER_API_KEY=your-real-key
```

If **multiple** servers share the same env var name, use the scoped form:

```bash theme={null}
# environments/my-env/.env
SIMLAB_MCP_WEATHER__API_KEY=weather-key
SIMLAB_MCP_DOCS__API_KEY=docs-key
```

Server names are normalized for scoped env vars by uppercasing and replacing non-alphanumeric characters with `_` (e.g. `my-docs` becomes `MY_DOCS`).

### Env-Local Custom Tool Definitions

For environment-specific tool servers that aren't part of the built-in catalog, you can scaffold custom tool definitions under the environment's `custom-tools/` directory. These use the same YAML schema as built-in catalog tools.

#### Scaffold a custom tool

```bash theme={null}
simlab env custom-tools add my-env harbor-main
```

This creates the following layout:

```text theme={null}
environments/my-env/
  env.yaml
  docker-compose.yml
  .env
  custom-tools/
    harbor-main.yaml
```

The command also adds `harbor-main` to the tools list in `env.yaml` and regenerates the environment's Docker Compose and related files.

#### Edit and regenerate

After editing `custom-tools/*.yaml`, `env.yaml`, or `mcp-servers.json`, regenerate the environment:

```bash theme={null}
simlab env init my-env --force
```

SimLab will also detect stale generated files when you run `simlab env up`, `simlab tasks run`, or `simlab tasks seed`, and prompt to regenerate in interactive sessions.

#### Inspect a custom tool

```bash theme={null}
simlab tools info harbor-main --env my-env
```

#### Naming rules

* Custom tool names must not shadow built-in tool names.
* Env-local custom tool names must be unique within the environment.
* MCP server names must not conflict with either built-in or env-local tool names.

### Custom CLI Tools (Coding Environments)

For coding environments that need extra CLI tooling installed in the sandbox, you can customize the environment with setup scripts, mounted fixtures, and reusable skills.

#### Layout

```text theme={null}
environments/my-env/
  env.yaml
  coding/
    setup/
      install-tools.sh      # Installs extra CLIs (e.g. pdftotext, xlsx2csv)
    fixtures/
      report.pdf             # Files mounted into /workspace/fixtures
      data.xlsx
    skills/
      pdf-xlsx-reporting/
        SKILL.md             # Reusable instructions for the coding agent
```

#### Setup script

The setup script runs inside the coding sandbox at startup. Use it to install any tools your tasks require:

```bash theme={null}
#!/usr/bin/env bash
set -euo pipefail
apt-get update -qq && apt-get install -y -qq poppler-utils
uv pip install --quiet --system xlsx2csv
```

#### Running with custom CLI tools

```bash theme={null}
simlab env init my-env --force
simlab tasks run --env my-env \
  --tasks-dir ./task-bundle \
  --task workspace_asset_report \
  --agent-model <model> \
  --agent-api-key "$SIMLAB_AGENT_API_KEY"
```

## Task Generation with Custom Tools

Custom tools (both MCP and env-local) are available for task generation. Provide your tool definitions and the pipeline produces tasks, seed data, and verifiers tailored to your tools:

```bash theme={null}
simlab tasks-gen run --config ./taskgen/config.toml
```

At runtime, the agent discovers custom tools through the same unified tool list as built-in tools and calls them transparently.


# Understanding Results
Source: https://docs.collinear.ai/simulation-lab/understanding-results

Rollout traces, scoring, statistical confidence, and state diffs

## Rollout Traces

Every rollout produces a complete trace: the full sequence of agent actions, tool calls, tool responses, and any state changes detected. Traces are saved as JSON and can be inspected programmatically.

## Scoring

Verifiers produce structured results for each rollout:

* **Pass/fail** — Did the agent complete the task successfully?
* **Reward signal** — A numeric score (typically 0.0 to 1.0) indicating quality of completion.
* **Metadata** — Verifier-specific details (which checks passed, which failed, and why).

## Statistical Confidence

A single rollout tells you whether the agent can solve a task. Multiple rollouts tell you whether it reliably solves it.

Running multiple rollouts will enable users to derive:

* **Pass rate** — What percentage of attempts succeed?
* **Variance** — How consistent is the agent? A model that passes 8/10 times is more reliable than one that passes 5/10.
* **Model comparison** — Run the same tasks with different models to compare reliability.

## State Diffs

Simulation Lab tracks what changed in the playground during each rollout. After each tool call that mutates state, the system captures a before/after snapshot and computes a human-readable diff.

For example, after the agent sends an email:

```
[STATE DIFF]
Email:
  Added:
    - To: employee@company.com
      Subject: "Welcome Package - First Day Information"
      Preview: "Hi Sarah, Welcome to the team! Here's what you need..."
```

After a calendar event is created:

```
[STATE DIFF]
Calendar:
  Added:
    - [marcus_johnson] New Hire Orientation
      2026-01-22 10:00 - 11:00 UTC
```

These diffs serve two purposes:

1. **Debugging** — Read the trace and immediately see what the agent changed at each step, without manually inspecting backing services.
2. **Verification** — Verifiers use diffs to confirm the agent made the right changes. "Did the agent send an email to the right person?" is answered by inspecting the email state diff.

Full-rollout diffs (comparing the playground state after seeding vs. after the agent finishes) provide a summary of everything that changed during the entire run.


# Verifiers & Reward Models
Source: https://docs.collinear.ai/simulation-lab/verifiers-and-reward-models

Programmatic verifiers and rubric-based reward models for agent evaluation

Verifiers consume `RunArtifacts` — the structured record of everything the agent did — and produce a pass/fail result.

There are two types:

1. **Programmatic verifiers** — Inspect the environment state directly. For example: "Did the agent send an email to the correct recipient?" is checked by querying the email tool server's state, as well as reviewing the state diffs (before/after environment snapshots).

2. **Rubric-based Reward Models** — Reward models that evaluates the agent's actions against a rubric. Useful for subjective criteria like "Did the agent communicate professionally?"

Both types receive the same `RunArtifacts` interface, so they work with any agent implementation.