SimReadyBench

A benchmark for simulation-ready OpenUSD asset reconstruction

Chris von Csefalvay1, Pranav Doma1, Achyuthan Unni Krishnan1 and Tamas Foldi1
1 HCLTech

Abstract

SimReadyBench evaluates how AI agents reconstruct simulation-ready 3D assets in OpenUSD (Universal Scene Description). Reconstruction tasks run in Harbor environments on a local Docker host or Daytona. A model receives reference views, nominal dimensions and an authoring brief, then produces a USD package for validation and comparison with the reference. Users can evaluate the public reference sets, define experiments over their own assets, models, harnesses and authoring skills or use the reconstruction tasks and rewards with an external reinforcement-learning trainer. Formal, visual and kinematic fidelity are reported independently.

Leaderboard

Download CSV
NV100 · 100 objects · formal fidelity (0–100)
#Details
1ChatGPT 6.1 TerraReasoning: high · Codex100.0070.62–70.6454.032.79100.00100.00100.001.654MIn 1.634M · Out 19.78KCached input 1.556M$0.2735$0.0027357.76
1ChatGPT 6.1 TerraReasoning: medium · Codex + CAD/build123d100.0070.8954.110.71100.00100.00100.003.917MIn 3.902M · Out 15.49KCached input 3.799M$0.3961$0.0039618.56
1ChatGPT 6.1 SolReasoning: high · Codex100.0072.8456.863.68100.00100.00100.001.991MIn 1.959M · Out 31.22KCached input 1.873M$0.7072$0.00707214.39
1ChatGPT 6.1 SolReasoning: medium · Codex + CAD/build123d100.0073.4755.563.00100.00100.00100.003.369MIn 3.351M · Out 17.81KCached input 3.258M$0.7383$0.0073839.45
5ChatGPT 6.1 SolReasoning: medium · Codex99.9873.22–73.2453.663.1499.67100.0099.001.064MIn 1.046M · Out 18.4KCached input 989.9K$0.4125$0.0041269.32
6ChatGPT 6.1 SolReasoning: low · Codex97.6973.58–73.6151.172.4471.3396.0039.00478.5KIn 470.8K · Out 7.728KCached input 436.2K$0.2065$0.0021145.86
7Claude Opus 5.5Reasoning: low · Claude Code97.1273.7751.192.4567.0094.0037.00381.6KIn 374.7K · Out 6.901KCached input 353.2K$0.3126$0.0032184.10
8ChatGPT 6.1 TerraReasoning: low · Codex91.1369.90–70.0347.810.872.006.000.00359KIn 353.4K · Out 5.575KCached input 310.6K$0.0969$0.0010644.40
9DeepSeek V4.1 Flash (1017)Reasoning: none · Terminus 286.3271.64–73.2335.360.0522.336.000.002.242MIn 2.21M · Out 31.91KCached input 2.013M$0.0274$0.00031710.98
10DeepSeek V4.1 Flash (1017)Reasoning: none · Terminus 2 + CAD/build123d86.2664.42–64.9037.610.09353.6787.0013.003.469MIn 3.435M · Out 34.19KCached input 3.3M$0.0250$0.00029013.92
11Claude Sonnet 5.5Reasoning: low · Claude Code83.6070.0250.060.660.000.000.0093.82KIn 91.46K · Out 2.359KCached input 80.39K$0.0578$0.0006913.28
12NVIDIA Nemotron 3.5 Super VLReasoning: none · Terminus 276.6968.33–69.6321.520.0050.00–1.670.00–5.000.002.512MIn 2.464M · Out 47.14KCached input 2.032M$0.3357$0.00437711.18

Pass = all 46 mandatory checks pass. pass@3: probability of at least one pass in three attempts; pass^3: probability all three pass.

Object comparison

Enable JavaScript to use the 3D comparison.

Enable JavaScript to use the 3D comparison.

Reference input image
Reference input
Rendered reconstruction
Reconstruction render

Neutral materials · shared metric scale · NVIDIA / CC BY 4.0

Inference cost uses final retained outputs. Retry spend is separate; tokens and trial time include retries. Cost coverage and pricing basis are reported independently.

Quality and efficiency

Score and inference efficiencyHigher scores and lower inference use are better. Focus or select a marker for configuration, measurement coverage and source. Colours identify model families. Circles carry model initials; an outer ring identifies a custom agent. Filled markers lie on the frontier; outlined markers are dominated.7580859095100$0.00$0.10$0.20$0.30$0.40$0.50$0.60$0.70$0.80$0.90Fidelity / 100Sgpt-6.1-sol/lowSgpt-6.1-sol/mediumSgpt-6.1-sol/mediumSgpt-6.1-sol/highTgpt-6.1-terra/lowTgpt-6.1-terra/mediumTgpt-6.1-terra/highCclaude-opus-5-5/lowCclaude-sonnet-5-5/lowDdeepseek-v4.1-flash-1017/noneDdeepseek-v4.1-flash-1017/noneNnemotron-3.5-super-vl/none
  • ChatGPT 6.1 Sol · Codex
  • ChatGPT 6.1 Sol · Codex + CAD/build123d
  • ChatGPT 6.1 Terra · Codex
  • ChatGPT 6.1 Terra · Codex + CAD/build123d
  • Claude Opus 5.5 · Claude Code
  • Claude Sonnet 5.5 · Claude Code
  • DeepSeek V4.1 Flash (1017) · Terminus 2
  • DeepSeek V4.1 Flash (1017) · Terminus 2 + CAD/build123d
  • NVIDIA Nemotron 3.5 Super VL · Terminus 2

12/12 configurations · 12 score observations

The frontier shows configurations that cannot be improved on one axis without losing on the other. Scores use the same object/repetition pairs across configurations.

Measurements and coverage
Matched comparison means and usage per reconstruction.
ConfigurationFormalVisualKinematicInference costTokensTrial timeSource
ChatGPT 6.1 Sol · Reasoning: low · Codex2.2.097.73294/300 eligible reconstructions51.08185/300 eligible reconstructions2.63271/300 eligible reconstructionsUSD 0.2065300/300 final-item costs478,511302/303 executions5.86 min303/303 executionsView
ChatGPT 6.1 Sol · Reasoning: medium · Codex2.2.099.98294/300 eligible reconstructions53.78185/300 eligible reconstructions3.42271/300 eligible reconstructionsUSD 0.4125300/300 final-item costs1,064,313302/302 executions9.32 min302/302 executionsView
ChatGPT 6.1 Sol · Reasoning: medium · Codex + CAD/build123d2.2.0100294/400 eligible reconstructions55.24185/400 eligible reconstructions2.9271/400 eligible reconstructionsUSD 0.7383400/400 final-item costs3,369,249400/401 executions9.45 min401/401 executionsView
ChatGPT 6.1 Sol · Reasoning: high · Codex2.2.0100294/300 eligible reconstructions57.04185/300 eligible reconstructions4.03271/300 eligible reconstructionsUSD 0.7072300/300 final-item costs1,990,604306/306 executions14.39 min306/306 executionsView
ChatGPT 6.1 Terra · Reasoning: low · Codex2.2.091.13294/300 eligible reconstructions47.33185/300 eligible reconstructions0.95271/300 eligible reconstructionsUSD 0.0969300/300 final-item costs359,013300/300 executions4.4 min300/300 executionsView
ChatGPT 6.1 Terra · Reasoning: medium · Codex + CAD/build123d2.2.0100294/400 eligible reconstructions53.64185/400 eligible reconstructions0.51271/400 eligible reconstructionsUSD 0.3961400/400 final-item costs3,917,477400/401 executions8.56 min401/401 executionsView
ChatGPT 6.1 Terra · Reasoning: high · Codex2.2.0100294/300 eligible reconstructions54.7185/300 eligible reconstructions3.07271/300 eligible reconstructionsUSD 0.2735300/300 final-item costs1,653,681300/300 executions7.76 min300/300 executionsView
Claude Opus 5.5 · Reasoning: low · Claude Code2.2.097.07294/300 eligible reconstructions51.3185/300 eligible reconstructions2.61271/300 eligible reconstructionsUSD 0.3126300/300 final-item costs381,567300/300 executions4.1 min300/300 executionsView
Claude Sonnet 5.5 · Reasoning: low · Claude Code2.2.083.61294/300 eligible reconstructions50.53185/300 eligible reconstructions0.72271/300 eligible reconstructionsUSD 0.0578300/300 final-item costs93,819300/300 executions3.28 min300/300 executionsView
DeepSeek V4.1 Flash (1017) · Reasoning: none · Terminus 22.2.086.25294/300 eligible reconstructions35.71185/300 eligible reconstructions0.05271/300 eligible reconstructionsUSD 0.0274300/300 final-item costs2,242,278301/301 executions10.98 min301/301 executionsView
DeepSeek V4.1 Flash (1017) · Reasoning: none · Terminus 2 + CAD/build123d2.2.086.09294/300 eligible reconstructions37.54185/300 eligible reconstructions0.1271/300 eligible reconstructionsUSD 0.025300/300 final-item costs3,468,986306/306 executions13.92 min306/306 executionsView
NVIDIA Nemotron 3.5 Super VL · Reasoning: none · Terminus 264 turns; 4,096 output tokens per call; reasoning off2.2.076.66294/300 eligible reconstructions22.19185/300 eligible reconstructions0271/300 eligible reconstructionsUSD 0.3357300/300 final-item costs2,511,512301/301 executions11.18 min301/301 executionsView

Download chart data (JSON)

Run the benchmark

Use Python 3.12 and a Linux Docker host. Install Harbor and connect your model provider:

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install harbor==0.24.0
harbor auth login
harbor run -d hcltech-robotics/simreadybench \
  -a codex -m openai/gpt-6.1-sol \
  --n-attempts 3 --n-concurrent 1 --max-retries 3 --upload

Choose the model, agent and concurrency for your experiment. Harbor run guide · Installation guide

Connect a model provider

The selected model receives images and uses tools to author the asset. The same provider connection works with local and Daytona execution.

ProviderConnectionCredentials
OpenAI--provider openai --model gpt-6.1-sol --reasoning medium, using the Responses APIOPENAI_API_KEY
Hugging Face--provider huggingface --model MODEL_ID:PROVIDER, using an image and tool-capable Inference Providers routeHF_TOKEN
AnthropicServe Claude through an OpenAI-compatible gateway, such as LiteLLM, and select its endpoint and capabilities in an operator policyANTHROPIC_API_KEY at the gateway and the gateway credential in the benchmark

The provider configuration guide includes commands, endpoint settings, model capabilities and pricing for all three connections.

Build a benchmark over your own assets

Supply one OpenUSD asset or a JSON list of SimReady package entry points, with their composed layers and textures. SimReadyBench prepares the reference observations, constructs the Harbor tasks and evaluates each reconstruction against its source.

For example, save the following as references.json, with the packages under ./references:

[
  "cabinet/cabinet.usd",
  "drawer/drawer.usda",
  "hinged_case/hinged_case.usd"
]
simready-bench benchmark references.json \
  --assets-root ./references \
  --model gpt-6.1-sol --harness pi \
  --skills ./skills/usd --deploy daytona \
  --runtime-images ../simreadybench-runtime.json \
  --workers 4 --repetitions 4 --authoring-turns 64 --supervise \
  --output-dir ../simreadybench-custom-results

Models, harnesses and skill sets are independent experimental factors. Evaluate each combination with its own command and output directory, holding references and repetitions constant. Select Pi or OpenCode with --harness, supply authoring skills with --skills or omit it for the empty skill condition.

The configuration guide defines asset manifests, model connections and matrix examples. Each report records the selected conditions alongside the measurements.

Use reconstruction tasks for reinforcement learning

Use your own assets as reconstruction episodes and serve a trainable policy through an OpenAI-compatible model endpoint. SimReadyBench runs the Harbor environment, provides asset observations and validator feedback, then scores the submitted USD. Your trainer uses the recorded trajectory and terminal measurements to update the policy between episodes.

Harbor receives formal fidelity as simready_reward on a 0-100 scale. The final readout also contains independent formal, visual and kinematic composites, so a trainer can define an objective over the properties relevant to its assets.

{"simready_reward": 0.0}

Each episode records the agent trace, candidate package, validator findings and fidelity measurements. The reinforcement-learning guide shows the policy endpoint, episode runner and reward readout for a custom asset.

How scoring and validation work

A reference USD supplies the geometry, dimensions and image for an independent authoring task. The agent uses the selected agent harness in a Harbor environment to create a new USD asset and can request validator feedback during reconstruction. The evaluator compares the final submission with the reference and renders both from matching cameras.

The same tasks and reference objects are used for each model configuration.

MeasureEvaluation
Formal fidelityAssets are evaluated according to the NVIDIA SimReady-Foundation Robotics-Prop 4.0.0 profile. The mandatory score is 95 times the fraction of 46 requirements passed, with five further points when all pass. Strict acceptance requires all 46.
Visual fidelityThe geometric mean of surface agreement and appearance similarity across four rendered views. Appearance uses foreground RGB structural similarity (SSIM).
Kinematic fidelityJoint recovery and agreement of allowed motion after whole-object alignment. Missing and additional joints reduce the score, while fixed attachments and moving joints are evaluated separately.
Optional SimReady scoreThe 51 optional requirements are scored separately as 100 times the fraction passed. Unmeasured requirements retain score bounds and measurement coverage.
ShapeBidirectional surface agreement after unit and up-axis normalisation and whole-object translation and yaw alignment. Physical scale and relative part transforms are preserved, with no scale fitting or independent part fitting. The same alignment defines the shared reference and reconstruction capture cameras.

Scoring documentation

Reference objects

NV100 contains 100 NVIDIA SimReady reference objects, including workshop tools, medical equipment, appliances and articulated mechanisms. Browse the reference objects.

Citation

Download the software citation

@software{voncsefalvay2026simreadybench,
  title = {SimReadyBench: Evaluating models, harnesses and skills
           for simulation-ready asset generation},
  author = {von Csefalvay, Chris and Doma, Pranav and
            Unni Krishnan, Achyuthan and Foldi, Tamas},
  year = {2026},
  url = {https://github.com/hcltech-robotics/simreadybench}
}