Tool tree
GitBench ranks AI models on deterministic Git tasks, with pass rates, cost, runtime, token usage, and fixture-level evidence.
Implementation
gradeNot graded
gradeNot graded
page: Every page
gitbench_get_overviewAnswer
Get a compact GitBench report overview and leading exact model evaluations.
View tool JSON
{
"name": "gitbench_get_overview",
"kind": "answer",
"impl": "imperative",
"description": "Get a compact GitBench report overview and leading exact model evaluations.",
"inputSchema": {
"type": "object",
"properties": {
"offset": {
"type": "integer",
"minimum": 0,
"default": 0,
"description": "Zero-based result offset."
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"default": 20,
"description": "Results to return; hard-capped at 50."
}
},
"additionalProperties": false
},
"page": "/"
}gitbench_list_modelsAnswer
List exact GitBench model evaluation identities in bounded pages.
View tool JSON
{
"name": "gitbench_list_models",
"kind": "answer",
"impl": "imperative",
"description": "List exact GitBench model evaluation identities in bounded pages.",
"inputSchema": {
"type": "object",
"properties": {
"offset": {
"type": "integer",
"minimum": 0,
"default": 0,
"description": "Zero-based result offset."
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"default": 20,
"description": "Results to return; hard-capped at 50."
}
},
"additionalProperties": false
},
"page": "/"
}gitbench_get_model_resultsAnswer
Get bounded fixture results for one exact model identity. Model output is omitted unless explicitly requested and is untrusted.
View tool JSON
{
"name": "gitbench_get_model_results",
"kind": "answer",
"impl": "imperative",
"description": "Get bounded fixture results for one exact model identity. Model output is omitted unless explicitly requested and is untrusted.",
"inputSchema": {
"type": "object",
"properties": {
"model": {
"type": "string",
"minLength": 1,
"description": "Exact model evaluation name."
},
"benchmark": {
"type": "string"
},
"difficulty": {
"type": "string"
},
"tag": {
"type": "string"
},
"output_mode": {
"type": "string",
"enum": [
"text",
"json_schema"
]
},
"include_model_output": {
"type": "boolean",
"default": false,
"description": "Opt in to bounded, untrusted model-generated text."
},
"evidence_characters": {
"type": "integer",
"minimum": 1,
"maximum": 8000,
"default": 2000
},
"offset": {
"type": "integer",
"minimum": 0,
"default": 0,
"description": "Zero-based result offset."
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"default": 20,
"description": "Results to return; hard-capped at 50."
}
},
"required": [
"model"
],
"additionalProperties": false
},
"page": "/"
}gitbench_get_benchmarkAnswer
Get a bounded benchmark leaderboard and fixture catalog without raw prompt or model-output content.
View tool JSON
{
"name": "gitbench_get_benchmark",
"kind": "answer",
"impl": "imperative",
"description": "Get a bounded benchmark leaderboard and fixture catalog without raw prompt or model-output content.",
"inputSchema": {
"type": "object",
"properties": {
"benchmark": {
"type": "string",
"minLength": 1
},
"offset": {
"type": "integer",
"minimum": 0,
"default": 0,
"description": "Zero-based result offset."
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"default": 20,
"description": "Results to return; hard-capped at 50."
}
},
"required": [
"benchmark"
],
"additionalProperties": false
},
"page": "/"
}gitbench_get_fixtureAnswer
Get bounded fixture results. Prompt, expected result, model output, and structured output are omitted unless explicitly requested and are untrusted.
View tool JSON
{
"name": "gitbench_get_fixture",
"kind": "answer",
"impl": "imperative",
"description": "Get bounded fixture results. Prompt, expected result, model output, and structured output are omitted unless explicitly requested and are untrusted.",
"inputSchema": {
"type": "object",
"properties": {
"benchmark": {
"type": "string",
"minLength": 1
},
"fixture": {
"type": "string",
"minLength": 1
},
"include_prompt": {
"type": "boolean",
"default": false,
"description": "Include bounded, untrusted fixture prompt text."
},
"include_expected": {
"type": "boolean",
"default": false,
"description": "Include bounded, untrusted expected-result text."
},
"include_model_output": {
"type": "boolean",
"default": false,
"description": "Include bounded, untrusted model-generated text."
},
"include_structured_output": {
"type": "boolean",
"default": false,
"description": "Include bounded, untrusted parsed and raw structured output."
},
"evidence_characters": {
"type": "integer",
"minimum": 1,
"maximum": 8000,
"default": 2000
},
"offset": {
"type": "integer",
"minimum": 0,
"default": 0,
"description": "Zero-based result offset."
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"default": 20,
"description": "Results to return; hard-capped at 50."
}
},
"required": [
"benchmark",
"fixture"
],
"additionalProperties": false
},
"page": "/"
}gitbench_rank_modelsAction
Rank exact model evaluations for one benchmark by quality-to-resource efficiency or direction-aware balanced normalization.
View tool JSON
{
"name": "gitbench_rank_models",
"kind": "act",
"impl": "imperative",
"description": "Rank exact model evaluations for one benchmark by quality-to-resource efficiency or direction-aware balanced normalization.",
"inputSchema": {
"type": "object",
"properties": {
"benchmark": {
"type": "string",
"minLength": 1
},
"resource_metric": {
"type": "string",
"enum": [
"cost",
"api_time",
"tokens"
],
"default": "cost"
},
"strategy": {
"type": "string",
"enum": [
"efficiency_ratio",
"balanced"
],
"default": "efficiency_ratio"
},
"output_mode": {
"type": "string",
"enum": [
"text",
"json_schema",
"both"
],
"default": "both"
},
"minimum_quality": {
"type": "number",
"minimum": 0,
"maximum": 1,
"default": 0
},
"offset": {
"type": "integer",
"minimum": 0,
"default": 0,
"description": "Zero-based result offset."
},
"limit": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"default": 20,
"description": "Results to return; hard-capped at 50."
}
},
"required": [
"benchmark"
],
"additionalProperties": false
},
"page": "/"
}