Benchmarks

Benchmarks

toothfairyai@latest…

Manager for benchmarks operations.

This manager provides methods to create, update, and manage benchmarks, as well as run benchmarks and track run status.

Example:

>>> client = ToothFairyClient(api_key="...", workspace_id="...")
>>> benchmark = client.benchmarks.create(...)
>>> run = client.benchmarks.run(benchmark_id="...", agent_id="...")
>>> run = client.benchmarks.wait_for_run(run.id)
>>> print(run.score)

Accessed via client.benchmarks.

Methods

Method	HTTP	Endpoint
create	POST	POST /benchmark/create
update	POST	POST /benchmark/update
delete	DELETE	DELETE /benchmark/delete/{benchmark_id}
get	GET	GET /benchmark/get/{benchmark_id}
list	GET	GET /benchmark/list
search	—	derived
run	POST	POST /benchmark/run
get_run	GET	GET /benchmark/run/{run_id}
list_runs	GET	GET /benchmark/runs
cancel_run	DELETE	DELETE /benchmark/run/{run_id}
wait_for_run	—	derived

create

Create a new benchmark.

def create(
    name: str,
    description: Optional[str] = None,
    questions: Optional[List[Dict[str, str]]] = None,
    files: Optional[List[str]] = None
) -> Benchmark

Endpoint: POST /benchmark/create · API service

Request fields

Python kwarg	Wire field	Type	Required	Description
id	id	string	no	Unique identifier
workspace_id	workspaceid	string	yes	Unique workspace identifier (UUID v4)
name	name	string	yes	Name of the resource
description	description	string	no	Detailed description of purpose and capabilities
questions	questions	string	no	JSON-encoded list of benchmark questions
files	files	string	no	List of file keys associated with the benchmark
metadata	metadata	string	no	Arbitrary metadata key-value pairs (JSON)
created_by	createdBy	string	no	ID of the user who created this resource
updated_by	updatedBy	string	no	ID of the user who last updated this resource
created_at	createdAt	string	no	Timestamp when this resource was created
creation_time	creationTime	string	no	Unix timestamp when the resource was created
updated_at	updatedAt	string	no	Timestamp when this resource was last updated

Always pass snake_case keyword arguments — the Python kwarg column shows the exact name to use for each wire field (as a named parameter where it appears in the method signature, otherwise via **kwargs). The SDK converts it deterministically to the camelCase wire key the API expects. Passing camelCase directly is deprecated: it emits a warning and converts to the same wire key.

Response fields

Field	Type	Description
id	string	Unique identifier for the benchmark
name	string	Name of the resource
description	string	Benchmark description
questions	array<object>	List of benchmark questions
workspaceID	string	Unique workspace identifier (UUID v4)
createdAt	string	Timestamp when this resource was created
creationTime	integer	Unix timestamp when the resource was created
updatedAt	string	Timestamp when this resource was last updated
metadata	object	Arbitrary metadata key-value pairs (JSON)
files	array<string>	Associated file paths

Example

client.benchmarks.create(name="…")

update

Update a benchmark.

def update(benchmark_id: str) -> Benchmark

Endpoint: POST /benchmark/update · API service

Request fields

Python kwarg	Wire field	Type	Required	Description
id	id	string	yes	Unique identifier
workspace_id	workspaceid	string	no	Unique workspace identifier (UUID v4)
name	name	string	no	Name of the resource
description	description	string	no	Detailed description of purpose and capabilities
questions	questions	string	no	JSON-encoded list of benchmark questions
files	files	string	no	List of file keys associated with the benchmark
metadata	metadata	string	no	Arbitrary metadata key-value pairs (JSON)
created_by	createdBy	string	no	ID of the user who created this resource
updated_by	updatedBy	string	no	ID of the user who last updated this resource
created_at	createdAt	string	no	Timestamp when this resource was created
creation_time	creationTime	string	no	Unix timestamp when the resource was created
updated_at	updatedAt	string	no	Timestamp when this resource was last updated

Always pass snake_case keyword arguments — the Python kwarg column shows the exact name to use for each wire field (as a named parameter where it appears in the method signature, otherwise via **kwargs). The SDK converts it deterministically to the camelCase wire key the API expects. Passing camelCase directly is deprecated: it emits a warning and converts to the same wire key.

Response

Returns the Benchmark object — fields documented in the create section above.

Example

client.benchmarks.update(benchmark_id="agent-id")

delete

Delete a benchmark.

def delete(benchmark_id: str) -> Dict[str, bool]

Endpoint: DELETE /benchmark/delete/{benchmark_id} · API service

Example

client.benchmarks.delete(benchmark_id="agent-id")

get

Get a benchmark by ID.

def get(benchmark_id: str) -> Benchmark

Endpoint: GET /benchmark/get/{benchmark_id} · API service

Response

Returns the Benchmark object — fields documented in the create section above.

Example

client.benchmarks.get(benchmark_id="agent-id")

list

List all benchmarks.

def list(
    limit: Optional[int] = None,
    offset: Optional[int] = None
) -> ListResponse

Endpoint: GET /benchmark/list · API service

Example

client.benchmarks.list()

search

Search benchmarks by name.

def search(search_term: str) -> List[Benchmark]

Derived method — delegates to another SDK call and performs no direct HTTP request.

Example

client.benchmarks.search(search_term="…")

run

Start a benchmark run.

def run(
    benchmark_id: str,
    agent_id: str,
    type: str = 'bm',
    mode: Optional[str] = None,
    reviewer_agent_id: Optional[str] = None,
    weights: Optional[Dict[str, float]] = None,
    passing_score: Optional[float] = None,
    num_questions: Optional[int] = None,
    name: Optional[str] = None,
    userid: Optional[str] = None
) -> BenchmarkRun

Endpoint: POST /benchmark/run · API service

Request fields

Python kwarg	Wire field	Type	Required	Description
benchmark_id	benchmark_id	string	yes	ID of the benchmark to run
agent_id	agent_id	string	yes	ID of the agent to test
type	type	string	no	Run type: 'bm' (standard evaluation) or 'abm' (consistency check)

Allowed: bm, abm | | mode | mode | string | no | ABM mode - only for type='abm': 'static' (scripted questions) or 'dynamic' (AI-generated probing)

Allowed: static, dynamic | | reviewer_agent_id | reviewer_agent_id | string | no | ID of the reviewer agent (default: 'sorcerer') | | weights | weights | object | no | Category weights for scoring, e.g. {"correctness": 0.5, "reasoning": 0.3, "context_usage": 0.2} | | passing_score | passing_score | number | no | Minimum score to pass (0-1). Default: 0.9 | | num_questions | num_questions | integer | no | Number of questions to use. Default: all questions in the benchmark | | name | name | string | no | Optional name for the run | | userid | userid | string | no | Optional user ID for the run |

Always pass snake_case keyword arguments — the Python kwarg column shows the exact name to use for each wire field (as a named parameter where it appears in the method signature, otherwise via **kwargs). The SDK converts it deterministically to the camelCase wire key the API expects. Passing camelCase directly is deprecated: it emits a warning and converts to the same wire key.

Response fields

Field	Type	Description
id	string	Unique identifier for the benchmark run
name	string	Name of the benchmark run
status	string	Current status of the run

Allowed: dispatched, inProgress, completed, inError, cancelled, cancellationRequested | | type | string | Run type: 'bm' (standard evaluation) or 'abm' (consistency check)

Allowed: bm, abm | | benchmark_id | string | ID of the benchmark that was run | | agent_id | string | ID of the agent that was tested | | reviewer_agent_id | string | ID of the reviewer agent used for scoring | | duration | number | Run duration in seconds | | score | number | Overall score (0-1 for BM: average_overall_score, 0-1 for ABM: average_consistency_score) | | batch_meta | object | Parsed batch metadata from a benchmark run | | created_at | string | ISO 8601 timestamp when the run was created | | updated_at | string | ISO 8601 timestamp when the run was last updated |

Example

client.benchmarks.run(benchmark_id="agent-id", agent_id="agent-id")

get_run

Get a benchmark run by ID.

def get_run(run_id: str) -> BenchmarkRun

Endpoint: GET /benchmark/run/{run_id} · API service

Response

Returns the BenchmarkRun object — fields documented in the run section above.

Example

client.benchmarks.get_run(run_id="agent-id")

list_runs

List benchmark runs for the workspace.

def list_runs(
    agent_id: Optional[str] = None,
    benchmark_id: Optional[str] = None,
    type: Optional[str] = None,
    status: Optional[str] = None,
    limit: Optional[int] = None
) -> ListResponse

Endpoint: GET /benchmark/runs · API service

Example

client.benchmarks.list_runs()

cancel_run

Cancel a benchmark run.

def cancel_run(run_id: str) -> Dict[str, bool]

Endpoint: DELETE /benchmark/run/{run_id}

Field-level schema not available in the API spec for this endpoint.

Example

client.benchmarks.cancel_run(run_id="agent-id")

wait_for_run

Wait for a benchmark run to complete.

def wait_for_run(
    run_id: str,
    poll_interval: float = 10.0,
    timeout: float = 1800.0
) -> BenchmarkRun

Derived method — delegates to another SDK call and performs no direct HTTP request.

Example

client.benchmarks.wait_for_run(run_id="agent-id")