feat: big_tf_unet_2 — intern-driven single-scale teacher pipeline

[nathanchenseanwalter]

Semi-automated, human-in-the-loop training pipeline. One run trains one single-scale (nfft, hop) teacher model; many runs produce per-scale teachers for later multiscale distillation (distillation out of scope).

Rebuilt on big_tf_unet_ablation internals (single-h5-per-step I/O, auto_params compute-once, forkserver+threadpoolctl baseline pool, SLURM-aware parmap, legacy-API TF32) with a workflow layer for non-expert operators:

• Run facade + 6 Jupyter notebook templates: run/submit a step, view a gallery, tweak a knob in run.yaml, clear + rerun, accept. All indexing, memory, and data-placement is automated; the operator only turns knobs.
• Single validated run.yaml per run (pydantic + OmegaConf-merge over defaults; unknown keys and out-of-range values fail loudly before compute).
• Task matrix with human sign-off + params-hash staleness (a knob change stales downstream steps). File-fenced clears replace the rmtree footgun.
• sbatch integration routes GPU steps to GPU nodes and heavy CPU steps to CPU nodes; submitted jobs are detached from the notebook kernel.

Methodology changes over the original recipe:

• One normalization convention N_a(x) = aasinh((x - median)/(a1.4826*MAD)) at every model-facing site (robust median/MAD; asinh soft compression, no hard clamps). a=1 at the denoiser input, a=3 at the dataset (smooth generalization of the old clip(z, +/-3)). Per-modality robust stats are recorded and exported in the deploy manifest.
• Knee-point thresholds via the kneed package (Kneedle on the positive robust-z ECDF) replacing the hand-rolled triangle method; knobs are knee_sensitivity and delta in robust-sigma units.
• Energy-level edge-bin detector (catches broad low-frequency plateaus the gradient detector missed).
• Refine stage replaced with leakage-free out-of-fold soft pseudo-labels: each fold predicts only its held-out samples; the final model trains on q = (1 - model_trust)knee_label + model_trustp_oof. Fixes the old refiner's cross-fold leakage and its backwards entropy criterion, and drops MC-dropout (cheaper).
• Scale-covariant auto parameters (baseline lam, UNet depth, batch size) so one config works at every resolution.

Adds a quick TJ-II benchmark step (per-image IoU / F1 at the F1-optimal threshold) as a relative tracker against the deployed anchor.

Adds kneed + threadpoolctl to the train dependency group. Tests are collection-safe without train deps.