Fitting a catalog of spectra in one batch¶
Use the batched spectroscopic stack when many galaxies share one instrument setup and one model family, and differ only in their data: pixel grid, redshift, resolution, masks and photometry. The stack compiles the forward model once per bucket of galaxies and evaluates every galaxy in the bucket through that compiled function.
Use the per-galaxy Fitter when the model differs between galaxies: a different parameter
set, a different line catalog, a Gaussian-process star formation history, or spectral indices.
The per-galaxy path also remains the reference the batched path is tested against.
When to use it¶
Situation |
Path |
|---|---|
Many spectra, one instrument, one model family, free redshift |
Batched stack |
Different model per galaxy, or a model feature the batch does not evaluate |
Per-galaxy |
A handful of spectra with a fixed redshift each |
Per-galaxy |
The workflow¶
The stack has four steps. Each galaxy is an observation record, the template is one
SEDModel that fixes everything the galaxies share, and the fits run on buckets of records.
import numpy as np
import jax.numpy as jnp
from tengri import (
DEFAULT,
FREE,
Fixed,
Observation,
Photometry,
SEDModel,
Spectroscopy,
Uniform,
make_agebins_from_zred,
)
from tengri.observation.banded import gaussian_resolution_bands
from tengri.observation.line_list import LineList
wave = np.linspace(4000.0, 7500.0, 600)
resolution = 2000.0 + 3000.0 * (wave - 3600.0) / (9800.0 - 3600.0)
spectroscopy = Spectroscopy(
wave_obs=jnp.asarray(wave),
resolution=resolution,
resolution_matrix=gaussian_resolution_bands(jnp.asarray(wave), resolution, 11),
resample="conserving",
eline_mode="marginalized",
eline_catalog=LineList.default_optical(),
eline_prior_sigma=1e-27,
eline_prior_type="flat",
)
template = SEDModel.build(
ssp_data=ssp,
observation=Observation(
spectroscopy=spectroscopy,
photometry=Photometry.from_names(["sdss_g", "sdss_r", "sdss_i"]),
),
sfh={
"type": "continuity",
"all_params": FREE,
"bin_edges_gyr": [float(e) for e in make_agebins_from_zred(zred=0.2)],
"log_total_mass": Uniform(6.0, 10.5),
},
dust_attenuation={
"type": "two_component",
"law": "calzetti",
"tau_bc": Uniform(0.0, 2.0),
"tau_diff": Uniform(0.0, 1.5),
"other_params": Fixed(DEFAULT),
},
met={"logzsol": Uniform(-1.5, 0.2), "all_params": Fixed(DEFAULT)},
neb={"type": "none"},
redshift=Uniform(0.0, 0.2),
)
from tengri.observation.batched import GalaxySpectrum, build_spectro_batches
from tengri.inference.batched_spectro import fit_spectra_map_vmap, fit_spectra_catalog_mcmc
records = [
GalaxySpectrum(
wave=w_i, flux=f_i, ivar=iv_i, resolution=bands_i, z=z_i,
mask=m_i, phot_flux=pf_i, phot_err=pe_i,
)
for (w_i, f_i, iv_i, bands_i, z_i, m_i, pf_i, pe_i) in catalog
]
batches = build_spectro_batches(records, quantum=256, model=template)
map_fit = fit_spectra_map_vmap(template, batches, n_steps=300)
posterior = fit_spectra_catalog_mcmc(template, batches, n_warmup=200, n_samples=200)
Records.
GalaxySpectrumholds one galaxy: the observed gridwave[Angstrom],fluxandivarin F_nu [erg/s/cm^2/Hz], the resolution as a banded matrix (theresolutionfield also accepts a scalar or per-pixel R), the redshiftz, an optional pixelmask, and optional photometryphot_fluxandphot_err. Pixels with a non-finite or non-positiveivardrop out. The template must use the same kind of resolution as the records.Template. One
SEDModelwith a free redshift fixes the model family, the parameter set, the line catalog, the filter union and the SFH ladder.Fixed(z)is refused, because the batch carries each galaxy’s redshift.Buckets.
build_spectro_batches(records, quantum=..., model=template)pads each pixel count up to a multiple ofquantum, groups galaxies with the same static specification, and stacks each group into one batch.Fits.
fit_spectra_map_vmapruns a MAP optimization for every galaxy in a bucket with one Adam scan.fit_spectra_catalog_mcmcsamples the posterior of every galaxy with the catalog engine. Treat MAP as a warm start for the sampler. For a non-parametric star formation history, MAP is not an estimate of the posterior median (see Measured numbers).
What each galaxy carries, and what the template fixes¶
Each galaxy supplies its own data, traced through the compiled function as arrays. A batch holds these 20 leaves, in this order, each with a leading galaxy axis:
Leaf |
Content |
|---|---|
|
Observed grid [Angstrom], padded to the bucket’s pixel count |
|
1.0 for a good real pixel, 0.0 for padded, masked or zero-weight pixels |
|
Observed F_nu [erg/s/cm^2/Hz]; 0 on excluded pixels |
|
1-sigma flux error; 1 on excluded pixels |
|
Banded resolution rows; empty when the resolution is Gaussian |
|
Per-pixel spectral resolution R; empty when the resolution is banded |
|
Redshift |
|
Smallest and largest real wavelength [Angstrom], the calibration range |
|
Photometry and its errors [erg/s/cm^2/Hz]; empty without photometry |
|
1.0 where a filter has a finite flux and a positive error |
|
Per-galaxy SFH bin edges [Gyr]; empty when the template ladder is used |
|
1.0 for each catalog line that lands on the galaxy’s real grid at its redshift |
|
Spectral resolution R that sets the emission-line profile width |
|
Milky Way E(B-V) of the foreground, per galaxy |
|
Bands of the lower Cholesky factor of the galaxy’s pixel covariance; empty without one |
|
Calibration segment id of each pixel; zeros without segments |
|
|
The template fixes everything that changes a shape or a branch of the compiled function: the
SED family and its parameter set, the emission-line catalog, the filter union, the SFH type and
the number of its edges, the resolution form, the number of calibration segments, the covariance
offsets, and quantum. A change to any of these is a new bucket and a new compile.
Choosing quantum¶
quantum sets how many distinct padded pixel counts the stack creates. Each distinct padded
count is one bucket, and each bucket compiles once. Larger quantum means fewer buckets and more
padding, so more pixels evaluated per galaxy.
For 24 galaxies with fitted pixel counts from 4,159 to 4,751. Padding is the padded pixels minus the fitted pixels, divided by the fitted pixels summed over the 24 galaxies:
|
Buckets |
Padded counts ( |
Padding |
|---|---|---|---|
256 |
3 |
4,352, 4,608, 4,864 |
3.0% |
4864 |
1 |
4,864 |
9.2% |
One bucket means quantum is at least the span of the pixel counts. Choose the value by how many
compiles you can afford against how much padding you can accept. The padded pixels carry zero
weight and do not change the likelihood of any real pixel.
Emission lines and calibration¶
Emission lines. eline_mode sets how the catalog lines enter the likelihood. "fitted"
samples each amplitude. "marginalized" with eline_prior_type="flat" integrates the amplitudes
out analytically, so no line parameters are sampled. The marginal is exact for a Gaussian
amplitude prior and a linear design. Use "marginalized" for catalog-wide fits where the line
amplitudes are nuisance parameters.
Model-centered line prior. Spectroscopy(eline_mode="marginalized", eline_prior_type="model")
replaces the flat prior with a Gaussian on each line amplitude, centered on the model’s own
prediction for that line. The prediction is carried through the same projection as the continuum,
including the dust and the Milky Way foreground. The width is eline_prior_width_dex, in dex
(default 0.3). The parameter eline_aperture_frac scales the prior mean by the fraction of the
line flux that falls inside the fiber. It is Fixed(1.0) by default, and
SEDModel.build(eline_aperture_frac=...) frees it or sets it to a prior. The prior works with the
lines marginalized alone and inside the joint marginal with the calibration polynomial. A line
outside a galaxy’s grid or under a pixel mask contributes nothing. Use it instead of the flat prior
when the lines should follow the model’s dust and star formation history, so that the Balmer
decrement constrains the attenuation.
Lines outside a galaxy’s grid. A catalog line that falls outside a galaxy’s observed grid at
its redshift contributes nothing to that galaxy. Its column is removed from the design, so it
neither adds to the model nor changes the marginal. The line_mask leaf of each batch records
which lines are in the grid. Its row sums give the number of lines each galaxy carries. In the
measured set below, a 3.3 to 3.9 thousand-angstrom window with sky, camera-join and Ca K masks held
12 to 24 of the 39 catalog lines.
Balmer series. LineList.default_optical() carries H8 to H14, He I 3889 and [Ne III] 3968
beside the lower-order lines. With neb={"type": "cue", "balmer_series_n_max": 50}, the default,
Cue’s emission appends H11 to H50 at the Case-B ratios of Storey and Hummer (1995) to H-beta.
balmer_series_n_max=None leaves the series at Cue’s own last line.
Calibration. calibration_marginalize=True with cal_n_poly=3 integrates a Chebyshev
calibration polynomial out of the spectral term, with the prior width cal_prior_sigma. The
joint marginal of calibration and lines is exact, so the two marginals are not applied one after
the other. The calibration marginal requires calibration_order=0 in the template, and the
combination with fitted lines (eline_mode="fitted") is refused. Use "marginalized" lines
with it.
Per-segment calibration scales. For a spectrum made of several camera segments, give the
template Spectroscopy(calibration_segments=...), either the pixel count of each segment or one
integer id per pixel, and each segment gets its own free scale in the calibration marginal. A
galaxy supplies its own ids with GalaxySpectrum(segment_ids=...). Galaxies with different segment
counts form separate buckets. cal_prior="scale_free" replaces the Gaussian prior on the
coefficients with a prior flat in data-flux units; the default is "gaussian".
Fiber-fraction prior. Spectroscopy(cal_scale_prior=(log10_center, sigma_dex)) puts a
log-normal prior on the mean calibration multiplier q over a window, and q enters the fit as one
sampled latent, cal_log_q. cal_scale_window=(lo, hi) [Angstrom] sets the window; the default
is the segment that holds the r band’s effective wavelength. GalaxySpectrum(cal_scale_prior=...)
sets the prior galaxy by galaxy.
Star formation history and stellar grids¶
Young-fraction coordinates. sfh={"type": "continuity", "coordinates": "young_fraction"}
samples the mass fractions of the continuity SFH from the youngest bin outward, each
Uniform(0, 1). Use it when the photometry pins the young mass: the posterior then has its own
axis for that mass. The default, "ratios", samples the log SFR ratios.
[alpha/Fe] grids. A 4-D SSP grid with an [alpha/Fe] axis is contracted over metallicity at each
alpha node before the alpha interpolation, so a sampled met_alpha_fe adds no per-draw copy of
the cube. A free AGB dust-shell weight (agb_dust_weight) collapses alpha first and runs the
rest of the stellar sum on a 3-D cube.
Choosing a sampler¶
fit_spectra_catalog_mcmc takes sampler="nuts" (the default), "hmc", "chees" or "ghmc".
Use "chees" for catalog-wide posteriors. "ghmc" runs MEADS (Hoffman & Sountsov 2022) with
a fixed-length warmup in two phases and returns the per-galaxy nested_rhat. The first phase
samples in coordinates whitened at each galaxy’s MAP with the analytic metric of precondition=
(full strength unless precondition names another), and MEADS adapts a diagonal scale per fold,
the step size, the damping and the slice translation on top. At rewhiten_at (default 0.5) of
the warmup the coordinates are re-estimated from the ensemble (the mean and the Stan-regularized
covariance of every chain’s positions over the second half of the first phase), the chains are
mapped into them, the momenta are redrawn, and the adaptation continues with its state carried
over. Sampling is plain GHMC with the warmup’s fold-averaged parameters held fixed.
Each fold needs at least 2 D chains, where D is the number of sampled coordinates, because a
fold’s standard deviation becomes the next fold’s momentum scale and one outlying chain among too
few sets it. Fewer raises ConfigError naming the minimum n_chains (2 D times n_folds). The
default n_chains is the smallest multiple of n_superchains (8) and n_folds (4) that is at
least 32 and gives every fold 2 D chains. A galaxy whose metric or ensemble covariance cannot be
factorized returns NaN draws with every transition flagged divergent. At DESI-like signal to noise
on the two-galaxy mock catalog the posterior medians agree with NUTS to within 0.07 of the NUTS
full 16-84 percent width, and the nested R-hat is at most 1.01.
Measured numbers¶
The numbers were measured at commit a608d682f. This page describes the code at fb0aa11c0.
The numbers come from 12 mock spectra fit two ways on one machine: the per-galaxy Fitter on a
Fixed(z) build of the template, and the batched stack on the free-redshift template. The mocks
have 560, 600 or 640 pixels over 4000 to 7500 angstrom, banded resolution R = 300, a 5 percent
calibration tilt and three injected emission lines (H beta, [O III] 5007 and H alpha). The model is
a DPL star formation history with the mass free, eline_mode="marginalized", and
calibration_marginalize=True with cal_n_poly=3. The batched path used quantum=64, which gives
two buckets: 4 galaxies at 576 padded pixels and 8 at 640.
The machine is an Apple M4 Pro with 14 cores and 48 GiB of memory, on CPU at float64.
Per-galaxy path. Times are summed over the galaxies of each pixel count, four each.
Stage |
560 pixels |
600 pixels |
640 pixels |
All 12 |
|---|---|---|---|---|
|
0.27 s |
0.36 s |
0.51 s |
1.13 s |
|
2.49 s |
2.23 s |
2.36 s |
7.08 s |
First |
1.64 s |
2.47 s |
2.43 s |
6.53 s |
300 Adam MAP steps |
3.10 s |
4.36 s |
3.49 s |
10.95 s |
Batched path. All 12 galaxies in one call per step.
Stage |
Wall |
JIT cache misses |
|---|---|---|
|
0.05 s |
not counted |
|
22.72 s |
573 |
|
0.82 s |
2 |
The jit-miss counter counts build-time compiles too. The second call logged 2 cache misses at the same step count; their cause was not traced.
Posterior check. Three galaxies, with n_warmup=100 and n_samples=200. The batched run is
fit_spectra_catalog_mcmc on the free-redshift template. The Fitter run is mcmc_nuts on the
galaxy’s Fixed(z) build. The table gives the median of log10 M and the 16 to 84 percent width.
Galaxy |
Pixels |
z |
Truth |
Batched median (width) |
|
Difference / |
Wall, batched / |
|---|---|---|---|---|---|---|---|
0 |
560 |
0.050 |
9.00 |
9.0024 (0.0020) |
9.0023 (0.0021) |
0.07 |
11.1 s / 11.2 s |
4 |
600 |
0.086 |
10.00 |
9.9996 (0.0020) |
9.9996 (0.0017) |
0.02 |
12.1 s / 11.2 s |
11 |
640 |
0.150 |
9.25 |
9.2517 (0.0014) |
9.2518 (0.0015) |
0.02 |
12.4 s / 9.1 s |
With 200 draws the median of each run has a Monte Carlo error of about 0.04 widths, so the difference of two medians has about 0.06 widths. The largest difference, 0.07 widths on galaxy 0, is within that error.
Peak memory. Each build ran in a fresh process. The table gives the peak RSS of one build on the 100-pixel synthetic grid. Most of that figure is the process baseline: about 494 MB after importing the package and the grid, so the build itself added about 11 MB.
Build (synthetic grid) |
Peak RSS after build |
Build wall |
|---|---|---|
Free-redshift template, process-baseline figure |
504.8 MB |
0.19 s |
|
508.3 MB |
0.14 s |
The build size on a real grid is the figure to plan memory with. On the real grid of the 24-galaxy run,
the free-redshift template built at a peak of 632 MB and one Fixed(z) build at 754 MB. That model has
no dust emission and no IGM term. The larger build peak of models with those terms is tracked in
#3153.
Larger set. A measurement on 24 galaxies at 4,159 to 4,751 fitted pixels, on the same machine, gave a first batched MAP call of 45 s including compile and 127 s
for 300 steps (5.3 s per galaxy). A per-galaxy Fitter NUTS run on one galaxy with a dense mass matrix
ran for more than 14.5 minutes and 2.6 GB before a memory kill. That timing used the Fitter defaults at
ten free parameters and is not comparable to the table above.
What these numbers show:
Batched MAP over 12 galaxies costs about the same as the whole per-galaxy path on its first call, 22.7 s against 25.7 s including compile. Once compiled, the 300-step batched call costs 0.82 s.
The batched and per-galaxy posteriors agree on the three sampled galaxies within 0.07 widths.
After 300 Adam steps the per-galaxy MAP estimates for the three mocks with truth 9.00 sit 0.048 to 0.049 dex high, while the batched MAP estimates are within 0.002 dex of truth on all 12 mocks.
For a non-parametric star formation history, use
fit_spectra_catalog_mcmcfor the posterior.
After 300 Adam steps the two paths’ MAP values can differ by 0.05 dex on the same mock, while their posterior medians agree within Monte Carlo error. The MAP is therefore a warm start for the sampler, not an estimate.
Float32¶
Float32 is a supported mode. It needs the data and σ in the model’s flux units (F_nu, erg/s/cm²/Hz), and σ must be representable in float32 (above about 1e-38). The stack’s tests check that the float32 likelihoods and their gradients are finite and nonzero for a galaxy of 10^10 Msun on a physically normalized SSP, with data in those units.
What the stack does not do¶
Each limit below names what to do instead.
A template covariance. Give each
GalaxySpectrumits owncovariance=BandedMatrixand build the template without one.Student-t noise and a calibration floor. The batched likelihood is Gaussian with no floor, and a template with
NoiseModel(student_t_dof=...)or a nonzerocalibration_flooris refused. Use the per-galaxyFitter, which takes both with marginalized calibration and marginalized lines: a floor gives an exact marginal, and Student-t noise gives a Laplace approximation at the iteratively reweighted least-squares mode (a pixel covariance makes it one multivariate Student-t).The Cloudy line prior. Use
eline_prior_type="flat"or"model", oreline_mode="fitted"or"off". Lines in kinematic groups (eline_kinematics) are fitted on the batch as on the per-galaxyFitter, each line at its group’s width and offset.Spectral indices. Use the per-galaxy
Fitter.Gaussian-process star formation histories. Use a parametric history with the batch, or the per-galaxy
Fitterfor a Gaussian-process history.Line-flux and line-ratio channels. Remove them from the template, or use the per-galaxy
Fitter.