Factorial Code Forge Lab

Verified project card

# Factorial Code Forge

Factorial Code Forge mixes two statistically independent non-Gaussian sources and
asks a tiny autoencoder to recover a useful two-dimensional code. In the experimental
variant, two adversarial predictors try to infer each latent coordinate from the
other while the encoder tries to make those predictions fail. Reconstruction and
variance constraints prevent a constant-code shortcut.

The historical anchor is Schmidhuber's 1992
[Learning Factorial Codes by Predictability
Minimization](https://doi.org/10.1162/neco.1992.4.6.863). The modern experiment
compares predictability minimization with an architecture-matched reconstructive
encoder, PCA whitening, and FastICA. It is a new controlled benchmark, not a
reproduction of the original paper's implementation or datasets.

## Verified results

Models trained on 20,000 mixed samples and were measured once on 5,000 new samples:

| Code | Absolute correlation | Symmetric MI estimate |
| --- | ---: | ---: |
| Autoencoder control | 0.02200 | 0.22345 |
| Online neural-adversarial ablation | 0.01886 | 0.22678 |
| Validation-selected predictability minimum | 0.01158 | 0.00280 |
| PCA whitening | 0.02039 | 0.22275 |
| FastICA | 0.00590 | 0.00326 |

The selected 12-parameter linear codec reconstructed the mixtures at
`1.08e-14` MSE. Its rotation was selected using polynomial cross-predictors on a
held-out portion of the training set, never the test set. The online neural
minimax game did not improve nonlinear dependence and is retained as a negative
stability result rather than omitted.

## Reproduce

```powershell
uv run python projects/factorial-code-forge/train.py
```

Evaluation snapshot

{
  "benchmark": "Two-source factorial code recovery",
  "codec_parameters": 12,
  "adversary_parameters": 1346,
  "test_reconstruction_mse": 1.0849382441166684e-14,
  "selected_rotation_radians": 0.6335255903647925,
  "validation_polynomial_predictor_error": 2.0412044050069174,
  "results": {
    "autoencoder_control": {
      "absolute_correlation": 0.02199843078760486,
      "symmetric_mutual_information_estimate": 0.22345146065238453,
      "linear_source_recovery_r2": 1.0
    },
    "neural_adversarial_ablation": {
      "absolute_correlation": 0.018864639802400177,
      "symmetric_mutual_information_estimate": 0.22677681116066317,
      "linear_source_recovery_r2": 1.0
    },
    "predictability_minimization": {
      "absolute_correlation": 0.011581345795229356,
      "symmetric_mutual_information_estimate": 0.002795710068105528,
      "linear_source_recovery_r2": 1.0
    },
    "pca_whitened": {
      "absolute_correlation": 0.02038519159561663,
      "symmetric_mutual_information_estimate": 0.22274881647833844,
      "linear_source_recovery_r2": 1.0
    },
    "fastica": {
      "absolute_correlation": 0.005903947226495519,
      "symmetric_mutual_information_estimate": 0.003260830744820531,
      "linear_source_recovery_r2": 1.0
    }
  },
  "training_history": [
    {
      "step": 100,
      "reconstruction_loss": 0.5751938223838806,
      "whitening_loss": 0.4999920129776001,
      "predictor_loss": 0.03917834907770157
    },
    {
      "step": 200,
      "reconstruction_loss": 0.10436813533306122,
      "whitening_loss": 0.4999969005584717,
      "predictor_loss": 0.012760449200868607
    },
    {
      "step": 300,
      "reconstruction_loss": 0.09469099342823029,
      "whitening_loss": 0.49983352422714233,
      "predictor_loss": 0.0026066915597766638
    },
    {
      "step": 400,
      "reconstruction_loss": 0.0830593854188919,
      "whitening_loss": 0.49922531843185425,
      "predictor_loss": 0.004627525806427002
    },
    {
      "step": 500,
      "reconstruction_loss": 0.08060496300458908,
      "whitening_loss": 0.4975007474422455,
      "predictor_loss": 0.01004670187830925
    },
    {
      "step": 600,
      "reconstruction_loss": 0.07426909357309341,
      "whitening_loss": 0.4930780827999115,
      "predictor_loss": 0.02585626393556595
    },
    {
      "step": 700,
      "reconstruction_loss": 0.0617077499628067,
      "whitening_loss": 0.48229295015335083,
      "predictor_loss": 0.06387390196323395
    },
    {
      "step": 800,
      "reconstruction_loss": 0.040828146040439606,
      "whitening_loss": 0.4678165912628174,
      "predictor_loss": 0.1077573299407959
    },
    {
      "step": 900,
      "reconstruction_loss": 0.022691549733281136,
      "whitening_loss": 0.3986736834049225,
      "predictor_loss": 0.36198264360427856
    },
    {
      "step": 1000,
      "reconstruction_loss": 0.021030452102422714,
      "whitening_loss": 0.23594792187213898,
      "predictor_loss": 0.9913151860237122
    },
    {
      "step": 1100,
      "reconstruction_loss": 0.04022720083594322,
      "whitening_loss": 0.009166543371975422,
      "predictor_loss": 1.8695604801177979
    },
    {
      "step": 1200,
      "reconstruction_loss": 0.003848953638225794,
      "whitening_loss": 0.00017961427511181682,
      "predictor_loss": 1.7930779457092285
    },
    {
      "step": 1300,
      "reconstruction_loss": 0.0014683017507195473,
      "whitening_loss": 0.004436715971678495,
      "predictor_loss": 1.7725040912628174
    },
    {
      "step": 1400,
      "reconstruction_loss": 0.0005974461091682315,
      "whitening_loss": 0.0012278903741389513,
      "predictor_loss": 1.8158133029937744
    },
    {
      "step": 1500,
      "reconstruction_loss": 0.0003378715191502124,
      "whitening_loss": 0.003598805284127593,
      "predictor_loss": 1.7867335081100464
    },
    {
      "step": 1600,
      "reconstruction_loss": 6.85915001668036e-05,
      "whitening_loss": 0.0005436374340206385,
      "predictor_loss": 1.8233861923217773
    },
    {
      "step": 1700,
      "reconstruction_loss": 0.0001249544438906014,
      "whitening_loss": 0.00044188593165017664,
      "predictor_loss": 1.8262327909469604
    },
    {
      "step": 1800,
      "reconstruction_loss": 1.3117482012603432e-05,
      "whitening_loss": 0.00011215780978091061,
      "predictor_loss": 1.8308453559875488
    },
    {
      "step": 1900,
      "reconstruction_loss": 1.1593769158935174e-05,
      "whitening_loss": 0.0016796954441815615,
      "predictor_loss": 1.7771589756011963
    },
    {
      "step": 2000,
      "reconstruction_loss": 0.0004471217398531735,
      "whitening_loss": 0.004560444969683886,
      "predictor_loss": 1.7743375301361084
    },
    {
      "step": 2100,
      "reconstruction_loss": 2.6598576368996873e-05,
      "whitening_loss": 0.0024056474212557077,
      "predictor_loss": 1.7827980518341064
    },
    {
      "step": 2200,
      "reconstruction_loss": 4.374821946839802e-05,
      "whitening_loss": 7.564837142126635e-05,
      "predictor_loss": 1.8309648036956787
    },
    {
      "step": 2300,
      "reconstruction_loss": 0.00020850264991167933,
      "whitening_loss": 0.005683479364961386,
      "predictor_loss": 1.822058916091919
    },
    {
      "step": 2400,
      "reconstruction_loss": 2.0745503206853755e-05,
      "whitening_loss": 8.849342520989012e-07,
      "predictor_loss": 1.8361434936523438
    },
    {
      "step": 2500,
      "reconstruction_loss": 1.0664540241123177e-05,
      "whitening_loss": 0.0009145042859017849,
      "predictor_loss": 1.7433021068572998
    }
  ]
}

Backed-up artifact tree

• README.md
• __pycache__/app.cpython-311.pyc
• __pycache__/data.cpython-311.pyc
• __pycache__/model.cpython-311.pyc
• app.py
• artifacts/factorial-code-forge/codec.safetensors
• artifacts/factorial-code-forge/evaluation.json
• artifacts/factorial-code-forge/latent_comparison.npz
• artifacts/factorial-code-forge/predictor_01.safetensors
• artifacts/factorial-code-forge/predictor_10.safetensors
• data.py
• data/factorial_sources.parquet
• model.py
• requirements.txt
• train.py