Skip to content

Benchmarks ​

This page gives measured analysis times for every jaxgsa method. The numbers show how time changes with the sample count, the number of time points, the number of outputs and the number of inputs. They apply to the machine and software below. They are not package limits.

What is timed ​

Each case times one call to analyze. Sampling and model evaluation are not in the timed region. The end-to-end table is the only exception: it times sampling, model evaluation and analysis separately.

The warm time is the median of five calls after the first call. The first call includes tracing and compilation, so it is reported separately. Each call waits until all its result arrays are ready before the clock stops.

ComponentValue
Date25 October 2026
MachineMacBook Pro, Apple M1 Pro, 8 CPU cores, 16 GiB memory, on mains power
Operating systemmacOS 15.6.1, arm64
Python3.12.13
jaxgsa1.3.1 development tree, commit 0750177
JAX0.11.2, CPU backend
NumPy2.5.3
Precisionfloat32 and float64, each in a separate run
Memory budget2048 MiB, the default of jaxgsa.config.set_memory_budget
IsolationOne fresh subprocess per case, killed above 3 GiB resident memory or after 600 s

No GPU or TPU result follows from these CPU measurements.

Results at a glance ​

All figures are warm float32 times unless the text says otherwise.

  • Sobol analysis at D=8 and base N=4096 (40,960 runs, scalar output) takes 0.7 ms.
  • Sobol analysis of 1000 time points at the same size takes 78 ms in float32 and 163 ms in float64.
  • A full Sobol study of 81,920 runs with 1000 outputs per run takes 0.26 s in float32 and 0.44 s in float64. This time includes sampling, model evaluation and analysis. The first run, with compilation, takes 0.44 s and 0.61 s.
  • With 5000 output slices per run (N=4096, T=100, K=50), the warm analysis times are:
MethodTime for 5000 slices
morris21 ms
rank97 ms
pce, shapley135 ms
dgsm0.65 s
efast0.74 s
sobol1.03 s
kucherenko1.10 s
optimal_transport1.30 s
pawn2.13 s
borgonovo7.21 s
vkoga42.5 s
hdmrkilled above the 3 GiB cap
hsictimed out after 600 s
  • The first call of a new shape takes between 0.06 s (kucherenko) and 3.8 s (vkoga) at N=4096. Later calls with the same shapes reuse the compiled code.

The model ​

All cases use the Sobol G-function from jaxgsa.benchmarks.sobol_g with D inputs, each uniform on [0,1]. The coefficients are a=(0,1,4.5,9,99,99,…), so four inputs are important and the other inputs are almost inert.

A wrapper adds T time points and K outputs per run:

y(x,t,k)=g(x)e−t(0.2+x2)(1+0.25k)+x3sin⁡(t+0.5k)

Here t takes T equally spaced values on [0,10] (t=1 when T=1), and k=0,1,…,K−1. Each slice has its own indices, so no slice is a copy of another. The output has shape (N,) for one slice, (N,T) for K=1, and (N,T,K) otherwise.

Method settings ​

N has a different meaning for each family of methods:

  • sobol, kucherenko: N is the base sample size. The design has N(D+2) runs. Sobol does not compute second-order indices.
  • efast: N is the number of points per curve.
  • morris: the design has N/16 trajectories.
  • Given-data methods: N is the number of Monte Carlo rows.

Other settings are the defaults, except these:

  • hsic: 50 permutations.
  • pce: order 3.
  • hdmr: order 2.
  • shapley: PCE backend, order 3.
  • dgsm: the call receives the model function. The time includes the derivatives that JAX computes from it.

Results ​

All values are warm median analysis times. Values are in milliseconds unless they show a unit. The last column gives the float64 time at the largest size.

These words mark cells that did not give a time:

  • skipped: the case did not start. HSIC and VKOGA build N×N matrices, so the sweep skips them above N=8192. Other skipped cases had input and output arrays too large for the 3 GiB cap.
  • timed out: the case ran for more than 600 s. Each HSIC slice is a full kernel analysis, so HSIC with many slices does not finish in this limit.
  • over cap: the case used more than 3 GiB of resident memory and was stopped.

Sample count ​

D=8, scalar output.

MethodN=1024N=4096N=16384N=65536float64, N=65536
sobol0.60.71.43.44.6
kucherenko0.60.82.06.26.1
efast0.50.61.44.25.3
morris0.60.50.60.60.7
dgsm1.41.41.93.44.8
borgonovo2.05.29.038.758.5
pawn1.83.07.027.127.1
hsic166.11.32 sskippedskippedskipped
optimal_transport1.55.38.128.930.1
rank0.93.75.016.116.7
pce3.79.127.8108.1229.7
hdmr22.143.6198.0801.7937.2
shapley3.89.327.8109.0239.0
vkoga819.41.54 sskippedskippedskipped

At N=4096 in float64, hsic takes 2.25 s and vkoga takes 4.30 s.

Time points ​

D=8, N=4096, K=1.

MethodT=1T=10T=100T=1000float64, T=1000
sobol0.81.36.778.3163.0
kucherenko0.82.517.9171.2121.1
efast0.61.57.494.7162.5
morris0.40.51.04.47.3
dgsm1.52.37.054.7133.6
borgonovo5.217.9117.21.29 s2.56 s
pawn3.09.455.8674.6648.4
hsic1.32 s7.68 s71.3 stimed outtimed out
optimal_transport5.39.327.0234.6258.8
rank3.74.05.520.935.7
pce9.310.219.035.881.3
hdmr41.3119.7403.62.50 s496.5
shapley9.19.819.536.683.0
vkoga1.52 s1.62 s2.05 s7.34 s12.38 s

PCE, Shapley and VKOGA fit one surrogate for all slices, so their time grows slowly with T. In float64, HSIC also timed out at T=100.

Outputs per time point ​

D=8, N=4096, T=100. K=50 gives 5000 slices per run.

MethodK=1K=10K=50float64, K=50
sobol6.4103.41.03 sskipped
kucherenko17.0190.31.10 sskipped
efast7.5101.3738.2skipped
morris0.95.121.131.8
dgsm7.289.5647.2934.1
borgonovo114.01.29 s7.21 s13.30 s
pawn53.6663.52.13 s2.61 s
hsic70.9 stimed outtimed outtimed out
optimal_transport27.1243.61.30 s1.36 s
rank5.422.097.0180.4
pce19.838.8134.6293.1
hdmr399.72.52 sover capover cap
shapley19.638.6135.4282.8
vkoga2.05 s7.25 s42.5 s53.5 s

In float64 at K=50, the estimated design, output and temporary arrays need about 4.6 GiB for Sobol and Kucherenko and 3.7 GiB for eFAST. These cases did not start. In float64 at K=10, sobol takes 189.1 ms and hdmr takes 517.0 ms.

Number of inputs ​

N=4096, scalar output. At D=128, a Sobol design has 532,480 runs.

MethodD=4D=8D=16D=32D=64D=128float64, D=128
sobol0.60.81.01.21.72.93.6
kucherenko0.60.81.32.13.96.0over cap
efast0.50.61.01.22.09.610.9
morris0.50.50.50.50.50.70.8
dgsm1.41.41.71.92.73.64.7
borgonovo3.45.36.19.916.227.641.1
pawn2.23.14.85.39.015.920.0
hsic897.51.31 s2.45 s7.68 s12.00 s21.43 s45.49 s
optimal_transport3.15.25.56.18.315.215.3
rank2.03.73.94.36.712.612.4
pce3.19.0119.748.010.420.927.0
hdmr14.742.1214.11.15 s16.74 s93.26 sover cap
shapley3.59.2120.148.110.721.526.9
vkoga945.51.55 s2.61 s6.11 s19.66 s62.57 s69.07 s

Read the PCE, Shapley and HDMR rows with these points:

  • PCE and Shapley lower the polynomial order when the term count is too large for N. The order falls from 3 to 2 at D=32 and to 1 at D=64. This is why D=16 is the slowest case. At D≥64 both methods warn that the fit explains less than half of the output variance.
  • HDMR of order 2 has more coefficients than samples at D≥32, and it warns about this. The times at large D measure a fit that N=4096 cannot support.
  • VKOGA warns at D≥32 that its surrogate misses most of the output variation.

First call ​

D=8, N=4096, scalar output. The first call includes tracing, compilation and one analysis.

Methodfloat32float64
sobol167 ms143 ms
kucherenko58 ms56 ms
efast127 ms134 ms
morris191 ms168 ms
dgsm445 ms471 ms
borgonovo324 ms316 ms
pawn373 ms376 ms
hsic2.08 s3.12 s
optimal_transport275 ms263 ms
rank214 ms117 ms
pce946 ms1.01 s
hdmr2.13 s2.02 s
shapley1.02 s1.09 s
vkoga3.78 s6.52 s

For most methods, the compilation part of the first call changes little across the sizes in this sweep. When the analysis itself is slow, the first call is approximately the warm time plus the compilation time.

End to end ​

Sobol, D=8, base N=8192 (81,920 runs), T=1000. The output has shape (81920,1000). The model is a jitted JAX function.

PrecisionSampleEvaluateAnalyzeTotalFirst run total
float3213.5 ms39.8 ms205.0 ms258.4 ms436.3 ms
float6413.8 ms91.2 ms330.4 ms436.0 ms614.4 ms

Memory ​

The memory budget controls how jaxgsa splits work into chunks. It is not a hard limit on process memory. Some HDMR, VKOGA and HSIC cases used more resident memory than the 2 GiB budget plus the input data, up to 2.9 GiB. The cases marked "over cap" used more than 3 GiB.

What the benchmarks do not show ​

  • Model evaluation and sampling are excluded. For an expensive simulator, estimator speed may have little effect on total study time.
  • First calls include tracing and compilation. The warm figures apply after compiled programs are reusable for the same shapes.
  • The sweep uses one model and fixed method settings. It does not rank the scientific value or accuracy of different sensitivity measures.
  • Surrogate timings do not establish surrogate quality. Check the fit before interpreting PCE, HDMR, Shapley, or VKOGA indices.
  • Results on other hardware and software versions will differ.

Reproduce the results ​

The sweep script is not part of the package. To repeat the measurement, use the model and method settings above. Sweep one size at a time from the base case D=8, N=4096, T=1, K=1. Run each case in a new Python process, so that the first call includes compilation and memory belongs to one case only. Time one call to analyze, then the median of five more calls, and block on every result array before you stop the clock. Use an idle machine on mains power. Record the commit, hardware, dependency versions, backend, precision and memory budget with the results.

Analytical benchmark functions ​

jaxgsa.benchmarks ships five models with analytical or published reference indices. Use them to test estimator accuracy separately from runtime.

ModuleParametersMarginalsUseful check
ishigami3Uniform [−π,π]Separates a zero first-order effect from a non-zero total effect
sobol_g8Uniform [0,1]Provides clear influential and nearly inert parameter tiers
linear3Uniform [0,1]Has equal first/total indices and zero interactions
gaussian_linear3GaussianIncludes analytical Borgonovo and optimal-transport indices
oakley_ohagan15GaussianTests a flatter, interaction-rich sensitivity profile

Each module provides a PROBLEM, batched evaluate(X), and analytical or published arrays. Read the module API for non-default parameters and the method comparison example for a worked accuracy comparison.

Released under the BSD-3-Clause License.