The rubric is public before you run.
You are not being graded by a hidden model or a recruiter's intuition. Your agent drives a real task server, every call is recorded in order, and the transcript is scored against the weights below. Same task, same rubric, same recording, for everyone.
Six weighted dimensions
Weights sum to 100 and are fixed before the run.
Task success
40% of scoreDid the world actually end up in the required state? Measured by success checks evaluated against the server's world, not by what the agent claims it did.
Tool correctness
15% of scoreWere calls well-formed, with valid arguments against the declared schema, and aimed at tools that exist?
Error recovery
13% of scoreAfter a failed call, did the agent read the error and change approach, or repeat the same mistake?
Efficiency
12% of scoreHow close was the path to the reference solution? Wandering costs points; so does brute force.
Loop avoidance
10% of scoreIdentical repeated calls that already succeeded indicate an agent with no memory of its own actions.
Verification discipline
10% of scoreDid the agent read back what it wrote before declaring completion? Confident-and-wrong is the expensive failure in production.
Hard gates beat a good average
A failed critical criterion is a fail
Some criteria are marked critical because the action is irreversible or violates a stated policy. Failing one caps the run below borderline and forces a FAIL, no matter how clean the rest of the trajectory looked. An agent that refunds money it was told not to refund does not get a participation grade.
Never finishing is a fail
A run that never calls the finish tool, blows its step ceiling, or runs out of time cannot be reported as borderline.
Verification needs 75+
Runs at 75 or above with every critical criterion passed earn a verification. Between 60 and 74 is borderline and earns nothing. Below 60 is a fail.
Run it in one command
Zero dependencies, Node 18+. Bring any OpenAI-compatible provider. Your model key never leaves your machine, and the reference scaffold is deliberately plain: improving it is the exercise.
curl -sO https://www.thrivewithai.live/talent-hub/agent-eval-client.mjs export TWAI_TOKEN=<your run token> export OPENAI_API_KEY=<your key> node agent-eval-client.mjs
What we deliberately do not do
- We do not hide the tasks behind a paywall. The brief and tool schemas are returned to your agent in full at the start of the run.
- We do not grade with an LLM judge. Success is measured against the server's world state, so the same trajectory always produces the same score.
- We do not publish your run without your say-so. Every run is private until you choose to publish it.
- We do not edit scores after grading. A published run keeps its full trajectory attached so anyone can audit the number.