SimReadyBench
A benchmark for simulation-ready OpenUSD asset reconstruction
Abstract
SimReadyBench evaluates how AI agents reconstruct simulation-ready 3D assets in OpenUSD (Universal Scene Description). Reconstruction tasks run in Harbor environments on a local Docker host or Daytona. A model receives reference views, nominal dimensions and an authoring brief, then produces a USD package for validation and comparison with the reference. Users can evaluate the public reference sets, define experiments over their own assets, models, harnesses and authoring skills or use the reconstruction tasks and rewards with an external reinforcement-learning trainer. Formal, visual and kinematic fidelity are reported independently.
Leaderboard
| # | Details | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | ChatGPT 6.1 TerraReasoning: high · Codex | 100.00 | 70.62–70.64 | 54.03 | 2.79 | 100.00 | 100.00 | 100.00 | 1.654MIn 1.634M · Out 19.78KCached input 1.556M | $0.2735 | $0.002735 | 7.76 | |
| 1 | ChatGPT 6.1 TerraReasoning: medium · Codex + CAD/build123d | 100.00 | 70.89 | 54.11 | 0.71 | 100.00 | 100.00 | 100.00 | 3.917MIn 3.902M · Out 15.49KCached input 3.799M | $0.3961 | $0.003961 | 8.56 | |
| 1 | ChatGPT 6.1 SolReasoning: high · Codex | 100.00 | 72.84 | 56.86 | 3.68 | 100.00 | 100.00 | 100.00 | 1.991MIn 1.959M · Out 31.22KCached input 1.873M | $0.7072 | $0.007072 | 14.39 | |
| 1 | ChatGPT 6.1 SolReasoning: medium · Codex + CAD/build123d | 100.00 | 73.47 | 55.56 | 3.00 | 100.00 | 100.00 | 100.00 | 3.369MIn 3.351M · Out 17.81KCached input 3.258M | $0.7383 | $0.007383 | 9.45 | |
| 5 | ChatGPT 6.1 SolReasoning: medium · Codex | 99.98 | 73.22–73.24 | 53.66 | 3.14 | 99.67 | 100.00 | 99.00 | 1.064MIn 1.046M · Out 18.4KCached input 989.9K | $0.4125 | $0.004126 | 9.32 | |
| 6 | ChatGPT 6.1 SolReasoning: low · Codex | 97.69 | 73.58–73.61 | 51.17 | 2.44 | 71.33 | 96.00 | 39.00 | 478.5KIn 470.8K · Out 7.728KCached input 436.2K | $0.2065 | $0.002114 | 5.86 | |
| 7 | Claude Opus 5.5Reasoning: low · Claude Code | 97.12 | 73.77 | 51.19 | 2.45 | 67.00 | 94.00 | 37.00 | 381.6KIn 374.7K · Out 6.901KCached input 353.2K | $0.3126 | $0.003218 | 4.10 | |
| 8 | ChatGPT 6.1 TerraReasoning: low · Codex | 91.13 | 69.90–70.03 | 47.81 | 0.87 | 2.00 | 6.00 | 0.00 | 359KIn 353.4K · Out 5.575KCached input 310.6K | $0.0969 | $0.001064 | 4.40 | |
| 9 | DeepSeek V4.1 Flash (1017)Reasoning: none · Terminus 2 | 86.32 | 71.64–73.23 | 35.36 | 0.052 | 2.33 | 6.00 | 0.00 | 2.242MIn 2.21M · Out 31.91KCached input 2.013M | $0.0274 | $0.000317 | 10.98 | |
| 10 | DeepSeek V4.1 Flash (1017)Reasoning: none · Terminus 2 + CAD/build123d | 86.26 | 64.42–64.90 | 37.61 | 0.093 | 53.67 | 87.00 | 13.00 | 3.469MIn 3.435M · Out 34.19KCached input 3.3M | $0.0250 | $0.000290 | 13.92 | |
| 11 | Claude Sonnet 5.5Reasoning: low · Claude Code | 83.60 | 70.02 | 50.06 | 0.66 | 0.00 | 0.00 | 0.00 | 93.82KIn 91.46K · Out 2.359KCached input 80.39K | $0.0578 | $0.000691 | 3.28 | |
| 12 | NVIDIA Nemotron 3.5 Super VLReasoning: none · Terminus 2 | 76.69 | 68.33–69.63 | 21.52 | 0.005 | 0.00–1.67 | 0.00–5.00 | 0.00 | 2.512MIn 2.464M · Out 47.14KCached input 2.032M | $0.3357 | $0.004377 | 11.18 |
Pass = all 46 mandatory checks pass. pass@3: probability of at least one pass in three attempts; pass^3: probability all three pass.
Object comparison
Enable JavaScript to use the 3D comparison.
Points unavailableBidirectional surface distance


Render unavailable
Fidelity (%)
| Formal fidelity | Visual fidelity | Kinematic fidelity |
|---|---|---|
| 100 | 33.3 | 0 |
| Inference cost (USD) | Tokens | Trial time |
|---|---|---|
| $0.1829 | 379,937Input 372,688 · output 7,249 | 4.7 minModel requests: Not measured |
Neutral materials · shared metric scale · NVIDIA / CC BY 4.0
Inference cost uses final retained outputs. Retry spend is separate; tokens and trial time include retries. Cost coverage and pricing basis are reported independently.
Quality and efficiency
- ChatGPT 6.1 Sol · Codex
- ChatGPT 6.1 Sol · Codex + CAD/build123d
- ChatGPT 6.1 Terra · Codex
- ChatGPT 6.1 Terra · Codex + CAD/build123d
- Claude Opus 5.5 · Claude Code
- Claude Sonnet 5.5 · Claude Code
- DeepSeek V4.1 Flash (1017) · Terminus 2
- DeepSeek V4.1 Flash (1017) · Terminus 2 + CAD/build123d
- NVIDIA Nemotron 3.5 Super VL · Terminus 2
12/12 configurations · 12 score observations
The frontier shows configurations that cannot be improved on one axis without losing on the other. Scores use the same object/repetition pairs across configurations.
Measurements and coverage
| Configuration | Formal | Visual | Kinematic | Inference cost | Tokens | Trial time | Source |
|---|---|---|---|---|---|---|---|
| ChatGPT 6.1 Sol · Reasoning: low · Codex2.2.0 | 97.73294/300 eligible reconstructions | 51.08185/300 eligible reconstructions | 2.63271/300 eligible reconstructions | USD 0.2065300/300 final-item costs | 478,511302/303 executions | 5.86 min303/303 executions | View |
| ChatGPT 6.1 Sol · Reasoning: medium · Codex2.2.0 | 99.98294/300 eligible reconstructions | 53.78185/300 eligible reconstructions | 3.42271/300 eligible reconstructions | USD 0.4125300/300 final-item costs | 1,064,313302/302 executions | 9.32 min302/302 executions | View |
| ChatGPT 6.1 Sol · Reasoning: medium · Codex + CAD/build123d2.2.0 | 100294/400 eligible reconstructions | 55.24185/400 eligible reconstructions | 2.9271/400 eligible reconstructions | USD 0.7383400/400 final-item costs | 3,369,249400/401 executions | 9.45 min401/401 executions | View |
| ChatGPT 6.1 Sol · Reasoning: high · Codex2.2.0 | 100294/300 eligible reconstructions | 57.04185/300 eligible reconstructions | 4.03271/300 eligible reconstructions | USD 0.7072300/300 final-item costs | 1,990,604306/306 executions | 14.39 min306/306 executions | View |
| ChatGPT 6.1 Terra · Reasoning: low · Codex2.2.0 | 91.13294/300 eligible reconstructions | 47.33185/300 eligible reconstructions | 0.95271/300 eligible reconstructions | USD 0.0969300/300 final-item costs | 359,013300/300 executions | 4.4 min300/300 executions | View |
| ChatGPT 6.1 Terra · Reasoning: medium · Codex + CAD/build123d2.2.0 | 100294/400 eligible reconstructions | 53.64185/400 eligible reconstructions | 0.51271/400 eligible reconstructions | USD 0.3961400/400 final-item costs | 3,917,477400/401 executions | 8.56 min401/401 executions | View |
| ChatGPT 6.1 Terra · Reasoning: high · Codex2.2.0 | 100294/300 eligible reconstructions | 54.7185/300 eligible reconstructions | 3.07271/300 eligible reconstructions | USD 0.2735300/300 final-item costs | 1,653,681300/300 executions | 7.76 min300/300 executions | View |
| Claude Opus 5.5 · Reasoning: low · Claude Code2.2.0 | 97.07294/300 eligible reconstructions | 51.3185/300 eligible reconstructions | 2.61271/300 eligible reconstructions | USD 0.3126300/300 final-item costs | 381,567300/300 executions | 4.1 min300/300 executions | View |
| Claude Sonnet 5.5 · Reasoning: low · Claude Code2.2.0 | 83.61294/300 eligible reconstructions | 50.53185/300 eligible reconstructions | 0.72271/300 eligible reconstructions | USD 0.0578300/300 final-item costs | 93,819300/300 executions | 3.28 min300/300 executions | View |
| DeepSeek V4.1 Flash (1017) · Reasoning: none · Terminus 22.2.0 | 86.25294/300 eligible reconstructions | 35.71185/300 eligible reconstructions | 0.05271/300 eligible reconstructions | USD 0.0274300/300 final-item costs | 2,242,278301/301 executions | 10.98 min301/301 executions | View |
| DeepSeek V4.1 Flash (1017) · Reasoning: none · Terminus 2 + CAD/build123d2.2.0 | 86.09294/300 eligible reconstructions | 37.54185/300 eligible reconstructions | 0.1271/300 eligible reconstructions | USD 0.025300/300 final-item costs | 3,468,986306/306 executions | 13.92 min306/306 executions | View |
| NVIDIA Nemotron 3.5 Super VL · Reasoning: none · Terminus 264 turns; 4,096 output tokens per call; reasoning off2.2.0 | 76.66294/300 eligible reconstructions | 22.19185/300 eligible reconstructions | 0271/300 eligible reconstructions | USD 0.3357300/300 final-item costs | 2,511,512301/301 executions | 11.18 min301/301 executions | View |
Run the benchmark
Use Python 3.12 and a Linux Docker host. Install Harbor and connect your model provider:
python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install harbor==0.24.0
harbor auth login
harbor run -d hcltech-robotics/simreadybench \
-a codex -m openai/gpt-6.1-sol \
--n-attempts 3 --n-concurrent 1 --max-retries 3 --upload
Choose the model, agent and concurrency for your experiment. Harbor run guide · Installation guide
Connect a model provider
The selected model receives images and uses tools to author the asset. The same provider connection works with local and Daytona execution.
| Provider | Connection | Credentials |
|---|---|---|
| OpenAI | --provider openai --model gpt-6.1-sol --reasoning medium, using the Responses API | OPENAI_API_KEY |
| Hugging Face | --provider huggingface --model MODEL_ID:PROVIDER, using an image and tool-capable Inference Providers route | HF_TOKEN |
| Anthropic | Serve Claude through an OpenAI-compatible gateway, such as LiteLLM, and select its endpoint and capabilities in an operator policy | ANTHROPIC_API_KEY at the gateway and the gateway credential in the benchmark |
The provider configuration guide includes commands, endpoint settings, model capabilities and pricing for all three connections.
Build a benchmark over your own assets
Supply one OpenUSD asset or a JSON list of SimReady package entry points, with their composed layers and textures. SimReadyBench prepares the reference observations, constructs the Harbor tasks and evaluates each reconstruction against its source.
For example, save the following as references.json, with the packages under ./references:
[
"cabinet/cabinet.usd",
"drawer/drawer.usda",
"hinged_case/hinged_case.usd"
]
simready-bench benchmark references.json \
--assets-root ./references \
--model gpt-6.1-sol --harness pi \
--skills ./skills/usd --deploy daytona \
--runtime-images ../simreadybench-runtime.json \
--workers 4 --repetitions 4 --authoring-turns 64 --supervise \
--output-dir ../simreadybench-custom-results
Models, harnesses and skill sets are independent experimental factors. Evaluate each combination with its own command and output directory, holding references and repetitions constant. Select Pi or OpenCode with --harness, supply authoring skills with --skills or omit it for the empty skill condition.
The configuration guide defines asset manifests, model connections and matrix examples. Each report records the selected conditions alongside the measurements.
Use reconstruction tasks for reinforcement learning
Use your own assets as reconstruction episodes and serve a trainable policy through an OpenAI-compatible model endpoint. SimReadyBench runs the Harbor environment, provides asset observations and validator feedback, then scores the submitted USD. Your trainer uses the recorded trajectory and terminal measurements to update the policy between episodes.
Harbor receives formal fidelity as simready_reward on a 0-100 scale. The final readout also contains independent formal, visual and kinematic composites, so a trainer can define an objective over the properties relevant to its assets.
{"simready_reward": 0.0}
Each episode records the agent trace, candidate package, validator findings and fidelity measurements. The reinforcement-learning guide shows the policy endpoint, episode runner and reward readout for a custom asset.
How scoring and validation work
A reference USD supplies the geometry, dimensions and image for an independent authoring task. The agent uses the selected agent harness in a Harbor environment to create a new USD asset and can request validator feedback during reconstruction. The evaluator compares the final submission with the reference and renders both from matching cameras.
The same tasks and reference objects are used for each model configuration.
| Measure | Evaluation |
|---|---|
| Formal fidelity | Assets are evaluated according to the NVIDIA SimReady-Foundation Robotics-Prop 4.0.0 profile. The mandatory score is 95 times the fraction of 46 requirements passed, with five further points when all pass. Strict acceptance requires all 46. |
| Visual fidelity | The geometric mean of surface agreement and appearance similarity across four rendered views. Appearance uses foreground RGB structural similarity (SSIM). |
| Kinematic fidelity | Joint recovery and agreement of allowed motion after whole-object alignment. Missing and additional joints reduce the score, while fixed attachments and moving joints are evaluated separately. |
| Optional SimReady score | The 51 optional requirements are scored separately as 100 times the fraction passed. Unmeasured requirements retain score bounds and measurement coverage. |
| Shape | Bidirectional surface agreement after unit and up-axis normalisation and whole-object translation and yaw alignment. Physical scale and relative part transforms are preserved, with no scale fitting or independent part fitting. The same alignment defines the shared reference and reconstruction capture cameras. |
Reference objects
NV100 contains 100 NVIDIA SimReady reference objects, including workshop tools, medical equipment, appliances and articulated mechanisms. Browse the reference objects.
Citation
Download the software citation
@software{voncsefalvay2026simreadybench,
title = {SimReadyBench: Evaluating models, harnesses and skills
for simulation-ready asset generation},
author = {von Csefalvay, Chris and Doma, Pranav and
Unni Krishnan, Achyuthan and Foldi, Tamas},
year = {2026},
url = {https://github.com/hcltech-robotics/simreadybench}
}