Skip to content

Commit f03dc20

Browse files
EightRiceclaude
andcommitted
feat(vl-jepa): scale up to 200M-500M params with SIGReg and multi-step rollout
Phase B scale-up of VL-JEPA addressing the capacity bottleneck identified in VALIDATION_FINDINGS.md (all four real-data runs collapsed at 18M params). Changes: - VLJEPAConfig: add dynamics_depth, sigreg_weight, sigreg_num_projections fields - SIGReg: Spectral Instance-Global Regularization (LeJEPA/LeWorldModel, 2025-26) Enforces isotropic Gaussian structure via 4-moment matching on random 1D projections. Added as weighted term in self_supervised_loss (weight=0 disables). - LatentDynamicsPredictor: transformer that maps latent plan -> next-step latent plan. Zero-initialized residual delta projection for stable training start. - VLJEPA.rollout(context_embeddings, num_steps): autoregressive rollout in embedding space with per-step uncertainty and variance tracking for degradation curves. - VLJEPA_PRESETS + get_vljepa_preset(): named configs for three scales: small - ~20M params (existing 18M config, validated on synthetic data) medium - ~235M params (minimum viable per survey, K=64, D=768, 224px) large - ~580M params (Phase B target, K=128, D=1024) - Backward compatible: existing forward/generate/self_supervised_loss APIs unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 2731f95 commit f03dc20

1 file changed

Lines changed: 432 additions & 2 deletions

File tree

0 commit comments

Comments
 (0)