Skip to content

Validation ​

This page shows how the tests check each method. The aim is that you can see what a number rests on before you trust it.

Most checks compare jaxgsa with a closed-form answer. A few compare it with another library. The rest check that the code agrees with itself. The page lists all three kinds, and it says plainly where the outside evidence is thin.

Tiers ​

Each test names its tier in its docstring. A lower number means stronger evidence.

  • T0, closed form. The expected value comes from a formula worked out on paper, such as the analytic Ishigami indices.
  • T1, published numbers. The expected value is a table of reference numbers from a paper, typed into the test.
  • T2, permissive-licence library. The expected value comes from a library such as SALib, SciPy, POT or OpenTURNS. The test either runs the library, or compares with values that a script in scripts/oracles/ recorded from it.
  • T3, copyleft library. The test runs an R package as a black box, in a separate process, and compares the numbers. It skips when R is missing.
  • T4, internal consistency. The test checks invariants, shapes, batching, jit and vmap agreement, and error handling. It also covers checks against an independent re-implementation in NumPy, and checks of one jaxgsa estimator against another. No outside number is involved.

Every method also has many T4 tests. The table below lists the checks that use an outside value (T0 to T3), plus the few T4 checks that compare with independent code. Every method in the table has at least one T0 to T3 check.

What the table shows ​

  • Tolerance is the value in the test code. atol is an absolute error. rtol is a relative error. A bare number such as 0.05 is an absolute bound on the largest error.
  • Slow tests are marked slow. The default pytest run skips them. Run them with uv run pytest -m slow.
  • Sampling methods carry Monte Carlo error. Their tolerances are wide on purpose. They check that the error is sampling error, not that the code is exact.

Results by method ​

MethodReferenceTierToleranceTest file
sobolIshigami closed-form S1, ST, S2 (small run)T0atol 0.02 (S2: 0.03)test_sobol_estimators.py
sobolSobol G-function closed-form S1, STT0atol 0.03test_sobol_estimators.py
sobolIshigami, Sobol G and Oakley-O'Hagan closed forms, large runs; tolerance shrinks with NT0 (slow)0.15 / 0.05 / 0.02 at N = 1024 / 4096 / 16384; 5% to 10% relative for the otherstest_analytical.py
sobolLinear additive model, exact S1 = STT0default assert_allclose (rtol 1e-7)test_analytical.py
sobolJansen, Martinez and Mauntz-Kucherenko estimators on Ishigami, against values recorded from OpenTURNS 1.27T2atol 1e-9 (Martinez; Jansen ST up to the shared denominator); 0.02 for the rest, a known convention gaptest_sobol_estimators.py
efastIshigami closed-form S1, STT0atol 0.08test_efast.py
efastLinear model: S1 equals the closed form, and ST equals S1T0atol 0.08 (S1), 0.05 (ST vs S1)test_efast.py
kucherenkoLinear Gaussian with correlated inputs, closed-form S1, ST and varianceT0atol 5e-3; variance rtol 2e-2test_kucherenko.py
kucherenkoClosed and total indices of all 15 subsets of 4 correlated inputs; Shapley effects by exact permutationT0atol 0.015test_kucherenko_subsets.py
kucherenkoR sensitivity::shapleyLinearGaussian; R pme_knn (loose)T3 (slow)atol 0.015; atol 0.08test_kucherenko_subsets.py
morrisLinear model: mu* equals the coefficients, mu equals mu*, sigma is zeroT0atol 1e-5test_morris.py
morrisIshigami: mu* ranking follows the analytic ST ranking; x3 reads as nonlinearT0 (ranking only)exact rank ordertest_morris.py
morrisScaling law under Y -> a*Y + bT0rtol 1e-4 to 1e-3test_morris.py
dgsmLinear model: exact nu, mu, and the lower bound against analytic STT0atol 1e-5 (nu, mu); rtol 0.08 (bound)test_dgsm.py
dgsmTruncated-normal variance used for the Poincare constant, against scipy.stats.truncnormT2rtol 1e-11test_dgsm_followups.py
borgonovoLinear Gaussian closed-form delta and S1T0atol 0.04 (delta), 0.05 (S1)test_borgonovo.py
borgonovoClass counts of the Plischke heuristic at seven sample sizes, against values recorded from SALib 1.5.2T2exact (integers)test_borgonovo.py
pawnY = x1: closed-form KS distance per bin, for the median and max statisticsT0abs 0.03 on the active input; below 0.15 on the otherstest_pawn.py
hsicNumPy V-statistic oracle built from scratch in float64 (R2, T and raw HSIC)T4rtol 1e-8test_hsic.py
hsicPopulation HSIC of a Gaussian pair (closed form) against the V-statisticT0relative 0.15 (default), 0.05 (slow)test_hsic.py
hsicSobol G ranking of R2 and TT0 (ranking only)rank ordertest_hsic.py
optimal_transportGaussian additive and Gaussian linear models, closed-form index, advective and diffusive partsT0atol 0.05 (N = 4096); 0.015 to 0.02 (N = 32000, slow)test_optimal_transport.py
optimal_transportCorrelated Gaussian inputs, conditional-normal formulaT0 (slow)atol 0.02test_optimal_transport.py
optimal_transportMultivariate index rebuilt from POT's log-domain Sinkhorn solverT2 (slow)rtol 1e-4 (OT cost), 1e-6 (advective part)test_optimal_transport.py
optimal_transportS1 against the Borgonovo S1 estimateT4atol 0.03test_optimal_transport.py
rankIshigami S1; linear Gaussian S1 with a copula and with a custom samplerT0atol 0.03test_rank.py
rankA written-out loop of the paper's formula (eq. 20)T4atol 2e-5test_rank.py
rankR sensitivity::sobolrank on Ishigami data, after the exact n/(n-1) rescaleT3 (slow)atol 2e-4test_rank.py
pceLinear additive model: S1, ST, S2 and the fitT0atol 0.02 with rtol 0.05test_pce.py
pceIshigami closed-form S1, ST and S2 (order 8)T0atol 0.01test_pce.py
pceHermite and Legendre bases are orthonormal under Gauss quadratureT0atol 1e-6test_pce.py
hdmrIshigami closed-form S1 and ST (order 2)T030% relative; an index that should be zero stays below 0.1test_hdmr.py
hdmrIshigami with n_intervals=4 and N = 4096, S1, ST and S2[x1, x3]T0atol 0.05 (S1), 0.06 (ST, S2)test_hdmr_order2.py
hdmrIndices do not change under Y -> 3Y + 17T0rtol and atol 2e-4test_hdmr.py
shapleyLinear model closed-form Shapley effects, PCE and HDMR backendsT0atol 0.05; effects sum to 1 within 1e-5test_shapley.py
shapleyIshigami closed-form Shapley effects, PCE backend (order 8)T0atol 0.04test_shapley.py
shapleyCorrelated Gaussian pair, closed form (2/7, 5/7), HDMR backend with the correlative termT0atol 0.06test_shapley.py
vkogaLinear Gaussian with correlated inputs, closed-form S_TC, S_TU and varianceT0atol 1e-2 (S_TC), 2e-2 (S_TU); variance rtol 5e-2test_vkoga.py
vkogaAdditive model has no interaction partT0atol 3e-2test_vkoga.py
vkogaCustom Gaussian sampler recovers the closed form; Ishigami under a custom copula agrees with the built-in copulaT0, T4atol 0.04; 0.06 (default tier)test_vkoga_custom_sampling.py

Where the evidence is thinner ​

  • Most evidence is T0. A closed form needs a model with a known answer. It checks the maths on a few test functions. It does not prove the code is right on your model.
  • dgsm, pawn, borgonovo and morris have a small number of outside checks in the default suite. Each rests on one or two closed-form cases, plus one SciPy check for dgsm and one recorded SALib check for borgonovo. For morris, the Ishigami check tests the rank order, not the values. For pawn, the closed form covers one active input.
  • hdmr has wide tolerances (30% relative in the basic check, 0.05 to 0.06 with n_intervals=4). The fit is a polynomial approximation. A script compares it with a direct-form estimator by hand (see below). No test runs that comparison.
  • hsic has no comparison with an outside library. Its checks are a NumPy re-derivation (T4) and a population closed form.
  • Outside libraries in the test suite. Three checks run in the default suite: dgsm against scipy.stats.truncnorm, and two recorded-value checks (sobol against OpenTURNS, borgonovo against SALib). The live checks against R (rank, kucherenko) and POT (optimal_transport) are slow, and they skip when the package is missing. Live comparisons with SALib, POT and SciPy for sobol, morris, borgonovo, pawn and optimal_transport are in a separate script (see below).
  • Nothing here is a proof. The tests run on a few functions, at fixed seeds, in float32 and float64. They cannot cover every model.

External oracles ​

An oracle is a source of truth outside jaxgsa. The oracles never ship in the package. CI does not run them.

OracleUsed forHow to run
POT (ot), a development extraoptimal_transport multivariate index (T2, slow)uv run --extra dev pytest -m slow tests/test_optimal_transport.py
R sensitivity (sobolrank, shapleyLinearGaussian, pme_knn)rank, kucherenko Shapley and PME (T3, slow)Install R and the packages. Then run uv run pytest -m slow tests/test_rank.py tests/test_kucherenko_subsets.py.
SciPy (truncnorm)dgsm Poincare constant (T2)Runs in the default suite.
Recorded OpenTURNS and SALib valuessobol estimators and borgonovo class counts (T2)Run in the default suite. The libraries are not needed.

R code runs in a separate process through tests/_r_oracle.py. jaxgsa never reads or copies the R code. If Rscript or a package is missing, the test skips.

Four scripts under scripts/oracles/ run by hand. Three of them print a table and exit non-zero on disagreement.

ScriptOracleChecks
openturns_sobol_estimators.pyOpenTURNS 1.27The Jansen, Martinez and Mauntz-Kucherenko Sobol estimators on Ishigami, against recorded literals
salib_delta_class_counts.pySALib 1.5.2The class counts of the Plischke heuristic that borgonovo uses
hdmr_direct_form.pyRe-derived NumPy direct-form HDMR, scipy.stats.f.ppfhdmr.analyze against an independent estimator. This is T4: it has no external reference.
external_checks.pySALib, POT, SciPy (live)A pytest file. See below.

The first two scripts record values from a T2 library. The default test suite compares with those values, so it needs neither OpenTURNS nor SALib:

  • TestOpenTurnsOracle in tests/test_sobol_estimators.py compares the estimators with the OPENTURNS_ISHIGAMI literals.
  • TestPlischkeHeuristic in tests/test_borgonovo.py compares the class counts with the SALib values.

To run one script:

bash
uv run --extra dev scripts/oracles/hdmr_direct_form.py

The OpenTURNS script is not in the dev extra. Run it with uv run --with openturns scripts/oracles/openturns_sobol_estimators.py.

Live checks ​

external_checks.py imports the outside libraries at run time. It is not in the default test run, and CI does not run it. It covers:

  • sobol: the default estimator against SALib.analyze.sobol.
  • morris: mu, mu_star and sigma against SALib.analyze.morris on the same design.
  • borgonovo: delta and S1 against SALib.analyze.delta.
  • pawn: the median index against SALib.analyze.pawn, and the tie-aware KS statistic against scipy.stats.ks_2samp.
  • optimal_transport: the W2 convention of ot.wasserstein_1d, per-bin W2, the Sinkhorn solver and the multivariate index against POT.

Run it with:

bash
uv run --extra dev pytest scripts/oracles/external_checks.py

It takes about 20 seconds.

Run the tests yourself ​

bash
uv run pytest -n auto            # default tier
uv run pytest -m slow -n auto    # tight tolerances and the R and POT checks

The default tier takes about three minutes on a cold cache. The slow tier takes longer.

Released under the BSD-3-Clause License.