Commit f03dc20
feat(vl-jepa): scale up to 200M-500M params with SIGReg and multi-step rollout
Phase B scale-up of VL-JEPA addressing the capacity bottleneck identified in
VALIDATION_FINDINGS.md (all four real-data runs collapsed at 18M params).
Changes:
- VLJEPAConfig: add dynamics_depth, sigreg_weight, sigreg_num_projections fields
- SIGReg: Spectral Instance-Global Regularization (LeJEPA/LeWorldModel, 2025-26)
Enforces isotropic Gaussian structure via 4-moment matching on random 1D
projections. Added as weighted term in self_supervised_loss (weight=0 disables).
- LatentDynamicsPredictor: transformer that maps latent plan -> next-step latent plan.
Zero-initialized residual delta projection for stable training start.
- VLJEPA.rollout(context_embeddings, num_steps): autoregressive rollout in embedding
space with per-step uncertainty and variance tracking for degradation curves.
- VLJEPA_PRESETS + get_vljepa_preset(): named configs for three scales:
small - ~20M params (existing 18M config, validated on synthetic data)
medium - ~235M params (minimum viable per survey, K=64, D=768, 224px)
large - ~580M params (Phase B target, K=128, D=1024)
- Backward compatible: existing forward/generate/self_supervised_loss APIs unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>1 parent 2731f95 commit f03dc20
1 file changed
Lines changed: 432 additions & 2 deletions
0 commit comments