Benchmarks

Benchmarks

@toothfairyai/sdk@latest…

Benchmark Management Module Handles benchmark CRUD operations for performance testing.

Accessed via client.benchmarks.

Methods

Method	HTTP	Endpoint
create	POST	POST /benchmark/create
update	POST	POST /benchmark/update
delete	DELETE	DELETE /benchmark/delete/{id}
get	GET	GET /benchmark/get/{id}
list	GET	GET /benchmark/list
run	POST	POST /benchmark/run
getRun	GET	GET /benchmark/run/{id}
listRuns	GET	GET /benchmark/runs
cancelRun	DELETE	DELETE /benchmark/run/{id}
waitForRun	—	derived

create

Create a new benchmark

async create(
    name: string,
    options: {
      description?: string;
      questions?: BenchmarkQuestion[];
      files?: string[];
    } = {}
)

Endpoint: POST /benchmark/create · API service

Request fields

Field	Type	Required	Description
id	string	no	Unique identifier
workspaceid	string	yes	Unique workspace identifier (UUID v4)
name	string	yes	Name of the resource
description	string	no	Detailed description of purpose and capabilities
questions	string	no	JSON-encoded list of benchmark questions
files	string	no	List of file keys associated with the benchmark
metadata	string	no	Arbitrary metadata key-value pairs (JSON)
createdBy	string	no	ID of the user who created this resource
updatedBy	string	no	ID of the user who last updated this resource
createdAt	string	no	Timestamp when this resource was created
creationTime	string	no	Unix timestamp when the resource was created
updatedAt	string	no	Timestamp when this resource was last updated

Response fields

Field	Type	Description
id	string	Unique identifier for the benchmark
name	string	Name of the resource
description	string	Benchmark description
questions	array<object>	List of benchmark questions
workspaceID	string	Unique workspace identifier (UUID v4)
createdAt	string	Timestamp when this resource was created
creationTime	integer	Unix timestamp when the resource was created
updatedAt	string	Timestamp when this resource was last updated
metadata	object	Arbitrary metadata key-value pairs (JSON)
files	array<string>	Associated file paths

Example

const result = await client.benchmarks.create('…');

update

Update an existing benchmark

async update(
    benchmarkId: string,
    options: {
      name?: string;
      description?: string;
      questions?: BenchmarkQuestion[];
      files?: string[];
    } = {}
)

Endpoint: POST /benchmark/update · API service

Request fields

Field	Type	Required	Description
id	string	yes	Unique identifier
workspaceid	string	no	Unique workspace identifier (UUID v4)
name	string	no	Name of the resource
description	string	no	Detailed description of purpose and capabilities
questions	string	no	JSON-encoded list of benchmark questions
files	string	no	List of file keys associated with the benchmark
metadata	string	no	Arbitrary metadata key-value pairs (JSON)
createdBy	string	no	ID of the user who created this resource
updatedBy	string	no	ID of the user who last updated this resource
createdAt	string	no	Timestamp when this resource was created
creationTime	string	no	Unix timestamp when the resource was created
updatedAt	string	no	Timestamp when this resource was last updated

Response

Returns the Benchmark object — fields documented in the create section above.

Example

const result = await client.benchmarks.update('agent-id');

delete

Delete a benchmark

async delete(benchmarkId: string): Promise<

Endpoint: DELETE /benchmark/delete/{id} · API service

Example

const result = await client.benchmarks.delete('agent-id');

get

Get a benchmark by ID

async get(benchmarkId: string)

Endpoint: GET /benchmark/get/{id} · API service

Response

Returns the Benchmark object — fields documented in the create section above.

Example

const result = await client.benchmarks.get('agent-id');

list

List all benchmarks in the workspace

async list(limit?: number, offset?: number)

Endpoint: GET /benchmark/list · API service

Example

const result = await client.benchmarks.list();

run

async run(
    benchmarkId: string,
    agentId: string,
    options: Omit<BenchmarkRunCreateData, 'benchmark_id' | 'agent_id'> = {}
)

Endpoint: POST /benchmark/run · API service

Request fields

Field	Type	Required	Description
benchmark_id	string	yes	ID of the benchmark to run
agent_id	string	yes	ID of the agent to test
type	string	no	Run type: 'bm' (standard evaluation) or 'abm' (consistency check)

Allowed: bm, abm | | mode | string | no | ABM mode - only for type='abm': 'static' (scripted questions) or 'dynamic' (AI-generated probing)

Allowed: static, dynamic | | reviewer_agent_id | string | no | ID of the reviewer agent (default: 'sorcerer') | | weights | object | no | Category weights for scoring, e.g. {"correctness": 0.5, "reasoning": 0.3, "context_usage": 0.2} | | passing_score | number | no | Minimum score to pass (0-1). Default: 0.9 | | num_questions | integer | no | Number of questions to use. Default: all questions in the benchmark | | name | string | no | Optional name for the run | | userid | string | no | Optional user ID for the run |

Response fields

Field	Type	Description
id	string	Unique identifier for the benchmark run
name	string	Name of the benchmark run
status	string	Current status of the run

Allowed: dispatched, inProgress, completed, inError, cancelled, cancellationRequested | | type | string | Run type: 'bm' (standard evaluation) or 'abm' (consistency check)

Allowed: bm, abm | | benchmark_id | string | ID of the benchmark that was run | | agent_id | string | ID of the agent that was tested | | reviewer_agent_id | string | ID of the reviewer agent used for scoring | | duration | number | Run duration in seconds | | score | number | Overall score (0-1 for BM: average_overall_score, 0-1 for ABM: average_consistency_score) | | batch_meta | object | Parsed batch metadata from a benchmark run | | created_at | string | ISO 8601 timestamp when the run was created | | updated_at | string | ISO 8601 timestamp when the run was last updated |

Example

const result = await client.benchmarks.run('agent-id', 'agent-id');

getRun

async getRun(runId: string)

Endpoint: GET /benchmark/run/{id} · API service

Response

Returns the BenchmarkRun object — fields documented in the run section above.

Example

const result = await client.benchmarks.getRun('agent-id');

listRuns

async listRuns(
    options: {
    agent_id?: string;
    benchmark_id?: string;
    type?: string;
    status?: string;
    limit?: number;
  } = {}
)

Endpoint: GET /benchmark/runs · API service

Example

const result = await client.benchmarks.listRuns();

cancelRun

async cancelRun(runId: string): Promise<

Endpoint: DELETE /benchmark/run/{id}

Field-level schema not available in the API spec for this endpoint.

Example

const result = await client.benchmarks.cancelRun('agent-id');

waitForRun

async waitForRun(
    runId: string,
    pollInterval: number = 10000,
    timeout: number = 1800000
)

Derived method — delegates to another SDK call and performs no direct HTTP request.

Example

const result = await client.benchmarks.waitForRun('agent-id');