Evaluation Protocol
Reference hardware for current leaderboard runs and the scoring methodology used to rank models on holdout cases.
Reference Environment
Plate and airfoil reference runs typically finish in tens of minutes on grossular-1. OpenRadioss Car is longer: ~2 hours for GeoTransolver and ~12 hours for MeshGraphNet. DrivAerML native-surface defaults training is multi-hour on the same class of node (architecture-dependent). Training and inference times on the leaderboard are informational only and not used for ranking.
How we score results
Rankings reward models that are accurate on engineering scalars and simulation fields when both exist. A high combined score requires strong performance on holdout cases the model did not train on (unseen angles of attack, hole diameters, crash thicknesses, or DrivAer morphs). When only nodal or only global scores are available, the public score is that score alone.
Holdout inference
Each submission is evaluated on hidden test inputs. Ground truth is never exposed during inference. Predictions are compared to simulation reference values after the run completes.
Global scalar accuracy
For decision-driving quantities (Cl, Cd, peak von Mises stress, stress concentration factor, etc.), each holdout case gets an accuracy score: max(0, 1 − |predicted − truth| / |truth|). We average these across all global metrics and holdout cases for the submission. DrivAerML globals are force_mom_Cd and force_mom_Cl.
Nodal field R²
Mesh field outputs (pressure, velocity, stress, displacement time series, mean surface Cp, wall shear, etc.) are scored with R² on nodal values. Negative R² from poor fits is clamped to zero before aggregation. Submission scores average these per-case field R² values across holdout cases (mean). For transient displacement fields, each case first uses median R² across timesteps. On DrivAerML, wall-shear components are aggregated to a single nodal score per case.
Combined score
When both globals and fields exist, the headline score is the geometric mean √(global_avg × nodal_r²_avg). When only one side is available (nodal-only or global-only), the headline score is that score alone.
Domain and leaderboard rollups
Domain columns (Structural, Fluid 2D / airfoil, Fluid 3D / DrivAerML, Crash) average combined scores for that column only. Fluid 2D and Fluid 3D stay separate so small-mesh aero is not mixed with native ~8M-node vehicle aero. Overall leaderboard rank uses the cross-benchmark average combined score over published rows.
What is not ranked
Training time, inference time, and hardware core type are recorded for context but do not affect rank. See submission scores on the leaderboard for per-run breakdowns.
Submission scoresFor each model on a benchmark with both globals and nodal fields:
combined = √(global_avg × nodal_r²_avg)
Both inputs are on a 0–1 scale. Example: 0.99 global accuracy and 0.97 mean field R² → combined ≈ 0.98. When only nodal or only global scores exist, combined equals that score alone.