Compilation: caching and diagnostics

Every JIT-compiled function in tengri (every backend, every fitter) goes through XLA. The first compile of a given shape is expensive: geoVI compiles in ~75 seconds, MGVI’s lax.while_loop in ~10 seconds, MCMC warmups in tens of seconds. The compiled artifact loads fast but builds slowly.

JAX’s persistent on-disk compilation cache turns building into loading for every later invocation. The HLO is serialized to disk on first compile and loads in ~100 milliseconds by every later subprocess, notebook restart, slurm worker, or sweep rerun.

tengri auto-enables this cache on import tengri.

Default behavior

import tengri  # cache auto-enabled at ~/.cache/tengri_jax_cache
  • Cache directory: ~/.cache/tengri_jax_cache.

  • Compiles faster than min_compile_time_secs are not persisted. The default is tuned so that every kernel costing real wall time (from per-filter photometry kernels up to geoVI) survives a restart. See tengri.enable_persistent_cache for the knob.

Configuration

Override the cache directory via environment variable:

export TENGRI_JAX_CACHE_DIR=/scratch/$USER/jax_cache

Useful when ~/.cache is on a small/quota’d filesystem (clusters, shared workstations).

Disable entirely:

export TENGRI_DISABLE_JAX_CACHE=1

Programmatic control (for notebook cells):

import tengri
tengri.enable_persistent_cache("/scratch/jax_cache")

enable_persistent_cache is idempotent. Calling it again with the same directory is a no-op.

Cache key

JAX’s persistent cache hashes:

  • The jaxpr (function structure)

  • Abstract input shapes / dtypes

  • static_argnames values (e.g. n_samples in native_vi_*)

  • JAX / jaxlib version

  • CPU class (or GPU model)

Implications:

  • Different RNG seeds hit the same cache. PRNGKey concrete values are not part of the key. Re-running with key=jax.random.PRNGKey(42) versus PRNGKey(43) is free after the first compile.

  • Different n_samples triggers one fresh compile. Subsequent runs at the same n_samples hit the cache.

  • A new JAX/jaxlib install invalidates everything. Wipe the cache after pip install -U jax:

    import tengri
    tengri.clear_cache()
    

When to wipe

Trigger

Action

pip install -U jax / jaxlib

tengri.clear_cache()

Moved machine / different CPU class

tengri.clear_cache()

Disk usage too high (du -sh ~/.cache/tengri_jax_cache)

tengri.clear_cache() then re-run

Inspect current size:

import tengri
print(tengri.cache_size_bytes() / 1024**2, "MB")

Is the cap actually binding, and on what?

A size alone cannot tell you whether to raise TENGRI_JAX_CACHE_MAX_GB or to raise min_compile_time_secs, because a full cache looks the same either way. cache_stats() separates them:

from tengri.utils.jax_cache import cache_stats

s = cache_stats()
print(s["fraction_of_cap"])                       # is the cap reached at all
print(s["top100_bytes"] / s["total_bytes"])       # a few big modules, or many small ones
print(s["n_entries_under_64kb"], s["bytes_under_64kb"])

Measured on a four-worktree development box at 99.0 % of an 8 GiB cap (bench/reports/2026-08-31_fast_nuts.md), 1 585 of 2 284 entries were under 64 KB and together held 0.11 % of the bytes, while the largest 100 entries held 79.2 % and the single largest was 124.8 MB. The cap is reached by a few dozen compiled samplers, not by the per-filter micro-kernels that min_compile_time_secs = 0.05 deliberately persists. Raising that threshold would have freed nothing, so the cap is the knob that matters. Read your own numbers before choosing; the ratio is a property of your models, not of tengri.

Use cases

  • Notebook iteration. Restart kernel → cache hit → first cell of a fit runs in seconds instead of minutes.

  • Slurm arrays. Each task is a fresh process; without persistent cache each pays full compile, possibly hundreds of times. With it: pay once globally.

  • Benchmark sweeps. bench/scripts/benchmark_vi_xlarge.py spawns one subprocess per (N, K) cell for clean peak-RSS measurement; the cache amortizes compile cost across the entire grid.

  • Resume after crash. Cache entries are persisted after compile completes. A worker that crashes mid-run leaves a usable artifact for the next attempt.

What it doesn’t fix

  • First-compile cost at any new shape. That work still has to be done once globally.

  • The XLA compile cliff at very large shapes (e.g. MGVI K=1 N≥16384, geoVI K=1 N≥8192) where XLA’s optimizer goes from O(10 s) to unbounded. The cache helps subsequent runs once any worker has built the artifact, but the first build can still hang. Tracked separately.

  • Different n_samples. n_samples is a static_argname, so changing it is a different cache key. The benchmark grid holds it fixed at 6.

  • The sub-band IGM fold on a free-redshift model. This is a build-time constant, not a compile, so neither the JAX cache nor the photometry z-table cache covers it. It is re-paid on every SEDModel.build, including repeat builds in one process.

Build cost: the free-redshift cliff

Compilation is not the only cost before your first number. Building a free-redshift model with approx=WavePrecomp() folds the IGM transmission into the sub-band quadrature weights, and that fold dominates:

build

time

free redshift, WavePrecomp()

~9 s (~37 s on the first build of a new SSP × filter × z-grid combination, which also pays for the z-table)

free redshift, WavePrecomp(), igm={'type': 'none'}

~0.3 s

fixed redshift, WavePrecomp()

~0.4 s

free redshift, approx=None

negligible

Two consequences worth planning around:

  • Fixing the redshift is the cheapest way out when your science allows it. A photometric-redshift fit does not, and pays the fold.

  • The cost is per build, not per process. Building the same configuration three times in one session pays it three times. If you loop over models that differ only in parameters, build once and vary the parameters. They are runtime inputs, not build inputs.

Absolute times vary with the machine, band set, and z-grid density; the ratios are the durable part.

Measuring this yourself: omitting redshift= does not give you a free redshift (see the default below), so a benchmark that leaves it out measures the fixed-z path and will not reproduce any of this.

Runtime cost: free redshift is ~7x a gradient, not ~1x

Build cost is paid once. The gradient cost is paid every leapfrog step, and for a sampler that is the number that sets your wall clock.

Measured on the compiled gradient graph (cost_analysis, so these are exact and do not depend on the machine), five SDSS bands, WavePrecomp(n_z=250) over z ∈ [0.5, 12]:

model

FLOPs per gradient

fixed redshift

2.7 M

free redshift, igm={'type': 'none'}

17.9 M

free redshift, with IGM

18.8 M

Free redshift interpolates the SSP × filter table, the sub-band quadrature tensors and the sub-band IGM transmission to the sampled z on every call. That is ~7x a fixed-z gradient, and it is the dominant term in any photometric- redshift fit.

Budgeting a sampler. Multiply it out before you launch:

gradients = (n_warmup + n_samples) * n_leapfrog_steps

fit_interim(n_warmup=300, n_samples=1000, n_leapfrog_steps=100) is 130,000 gradients per galaxy. At a free-redshift gradient of a few milliseconds, that is minutes to tens of minutes per galaxy, so an eight-galaxy population fit is hours whether or not anything is wrong. If a fit takes longer than you expect, compute this product first: it usually explains the whole thing, and no amount of profiling will make 130,000 gradients cheap.

n_z is not the speed knob it looks like. It sets the accuracy of the z interpolation (the default 250 is calibrated for less than 1% error). It no longer buys speed because the interpolation contracts only the five grid nodes the triweight kernel supports, so the per-gradient cost is the same at n_z=250 as at n_z=10 byte for byte in the compiled graph. Lower it for build time and memory if you need to, not for gradient throughput.

A note on reading cost_analysis. Its bytes accessed field counts logical operand bytes per operation, not DRAM traffic after fusion. The windowing change above cut FLOPs 6.8x and bytes accessed 5.1x and moved the clock about 1.9x. Use these counters to find where cost lives; use a stopwatch to find out what it costs.

State the redshift: the build refuses without one

SEDModel.build(...) has no redshift default. A build that names no redshift= is refused with the accepted spellings, because a silent default puts every galaxy at one distance and every cost measurement on one path.

model = SEDModel.build(ssp_data=ssp, observation=obs,
                       sfh={'type': 'dpl', 'all_params': FREE},
                       redshift=Fixed(0.1))              # known redshift

model = SEDModel.build(ssp_data=ssp, observation=obs,
                       sfh={'type': 'dpl', 'all_params': FREE},
                       redshift=Uniform(0.0, 3.0))       # free, and fitted

State it deliberately whenever the redshift matters for what you are measuring: redshift=Fixed(z) at your galaxy’s known redshift, or a prior to fit it.

  • A free redshift exercises the redshift-table path, where the IGM fold above costs ~9 s at build; a fixed redshift takes the fixed-z path at ~0.4 s. Compare build costs only between models that make the same choice.

Three-layer cache architecture

The on-disk persistent cache described above is one of three independent caches that work together. Each handles a different reuse boundary:

Layer

Lives in

Reuse boundary

Cleared by

L1: structural prediction kernels

RAM, _STRUCTURAL_KERNEL_CACHE (LRU=4)

Across SEDModel instances with the same compile_signature()

tengri.gc()

L2: loss / grad / log-density / log-likelihood

RAM, _SHARED_*_CACHE keyed on (compile_sig, mode)

Across Fitter instances with the same fingerprint

tengri.gc() (no cache policy touches it)

L3: inference scan body (HMC leapfrog, NUTS tree, VI loop)

RAM, _SHARED_ENGINE_CACHE (LRU=2)

Across Fitter.run calls with the same (compile_sig, method)

iterate (default) drops only non-matching entries; sweep drops all; tengri.gc() drops all

Disk cache

~/.cache/tengri_jax_cache

Across processes, notebook restarts, slurm tasks

tengri.clear_cache()

Each fit picks one of three cache policies before it runs. The default is iterate, which clears L3 with keep_sig=(compile_signature(), method), dropping only entries whose key does not match the current run. Thus:

  • Multi-phase notebook (MAP → HMC → posterior-predictive): the prior phase’s L3 entry is dropped (different mode), the current phase’s entry is kept. Peak RSS stays at one inference body, never two.

  • Catalog loop (100 galaxies, same model + method): every run has the same (compile_sig, method), so the entry is preserved and the leapfrog compile is paid once for the whole catalog. No persistent() context needed for the common case.

The other two policies are opt-in, and both are reached through a context manager rather than a keyword:

Policy

How to select it

L3 effect

iterate (default)

nothing to do

keep the matching entry, drop stale ones

sweep

tengri.lean() or TENGRI_LEAN=1

drop every L3 entry

persistent

tengri.persistent() or TENGRI_PERSISTENT=1

keep everything, matching or not

The older fit(..., lean=True) keyword is retired. It is still honored for back-compat (lean=True → sweep, lean=False → iterate) but emits a DeprecationWarning; prefer the context managers.

tengri.gc() calls clear_shared_caches(scope="all", drop_xla=True), which nukes L1 plus L2 plus L3 plus JAX’s internal caches. Use it between iteration loops that build many slightly-different SEDModel or Fitter configurations.

tengri.persistent() is only useful for the rare case where you want to keep a stale L3 entry alive across phases (e.g. running MAP and HMC repeatedly in alternation and reusing both compiles). The default keep-matching path covers the catalog and multi-phase cases without it.

Tuning the in-memory caches

The L1–L3 caches above are also tunable from the environment, which is useful when a process is memory-bound rather than compile-bound:

export TENGRI_ENGINE_CACHE_MAXSIZE=1   # default 2; L3 entries held before eviction
export TENGRI_DISABLE_SHARED_CACHES=1  # never populate the shared caches at all
export TENGRI_LEAN=1                   # force the sweep policy process-wide
export TENGRI_PERSISTENT=1             # keep every L3 entry, stale included

Each L3 entry holds a compiled XLA executable, so lowering TENGRI_ENGINE_CACHE_MAXSIZE to 1 trades one extra compile for a smaller resident set. TENGRI_DISABLE_SHARED_CACHES=1 is the strongest guarantee. Every Fitter compiles fresh, and nothing is retained between fits, at the cost of paying the compile on every run.

The in-process equivalents are tengri.lean() / tengri.persistent() (context managers) and tengri.gc() / tengri.clear_shared_caches(). All four are importable from the top-level package, though they are not listed in tengri.__all__.

The disk cache underneath all three means even a full tengri.gc() on a warm process does not recompile from scratch on the next call: XLA reads the already-optimized binary from ~/.cache/tengri_jax_cache (~3 s versus the ~80 s cold compile).

Diagnosing recompilations

The cache above makes a repeated compile cheap. When wall time is still dominated by compilation, the question is which kernels are recompiling and why. That is what the compile-event tracer answers.

When working with JAX inference on large models, cold compilation can dominate wall-clock time. The compile-event tracer shows what is recompiling and why.

Quick Start

1. Enable logging

Set the environment variable before running your notebook:

export TENGRI_LOG_COMPILES=1

2. Run your notebook

Execute your notebook as normal. Events will be logged to ~/.cache/tengri_jax_cache/compile.log (or override via TENGRI_COMPILE_LOG_PATH).

3. Analyze the log

python scripts/analyze_compile_log.py

or specify a custom log path:

python scripts/analyze_compile_log.py --log /path/to/compile.log

What the Analysis Shows

The report includes:

  • Total compile events: How many JIT compilations occurred

  • Total wall time: Aggregate compilation time (seconds)

  • Cache-hit ratio: Proportion of fast (cached) vs. slow (cold) compiles

  • Per-method breakdown: Count, total, mean, and max duration for each inference method

  • Spurious recompiles: Consecutive events with different signatures (indicates unnecessary recompilation)

Example output:

================================================================================
TENGRI COMPILE LOG ANALYSIS
================================================================================

SUMMARY
-------
Total compile events:        14
Total compile wall time:     42.53 s
Cache hits (inferred):       8
Cache misses (inferred):     6
Hit ratio:                   57.1%

PER-METHOD BREAKDOWN
-------
Method                 Count      Total (s)      Mean (s)      Max (s)
-------
geovi                      2         15.23          7.61         8.10
vi                         6          8.54          1.42         2.31
unknown                    6         18.76          3.13         6.54

SPURIOUS RECOMPILES (consecutive events with different signatures)
-------
[2→3] signal_response (None) → run_evi (vi)
  sig[2]: ((...shape_sig..., ...model_sig...),)
  sig[3]: ((...different_model_sig...,),)

Configuration

Environment Variables

  • TENGRI_LOG_COMPILES=1 – Enable compile logging (default: off)

  • TENGRI_COMPILE_LOG_PATH=/custom/path.log – Override log file location

Disabling Logging

By default, the logger is completely disabled and adds zero overhead. To confirm:

from tengri.utils.compile_log import is_enabled
print(is_enabled())  # False if TENGRI_LOG_COMPILES not set

Cache-Hit Heuristic

The log marks events as “cache hits” if they complete in < 1.0 s. This is a rough approximation:

  • True hits (file on disk): typically 0.1–0.5 s

  • Cold compiles: typically 5–30+ seconds (depends on graph size)

  • Hybrid cases (e.g., warm iteration with some tracing): 1–5 s

On fast hardware, the heuristic may flag slow loads as hits. On network-mounted storage, it may misclassify cache loads as cold compiles.

Integration

The tracer hooks into:

  • get_or_build_signal_response() – Physics kernel compilation

  • build_jit_engine() – Inference engine (VI, geoVI, etc.)

Each compilation site is wrapped with a context manager that records timing and metadata. No code changes required.

Notes

  • Logging is thread-safe: multiple threads can write events simultaneously

  • The log file grows indefinitely; manually clean it up as needed

  • Timestamps are UTC ISO 8601 format

  • Signatures are stringified tuples for easy diffing to detect spurious recompiles