571 of 618 runs done, 2 jobs running

Dynamic compositional representations · October 4, 2026

571 runs are finished, 8 are running, 39 not started. Grids grid1, grid1b and ctx1 are complete; grid2-role and fold1 are still running.

What ran

• Job 7104: the fold line, chunk 1 of 4, running for 3 h 22 min on 1 node with 256 CPU cores, 4 NVIDIA H100 graphics processors, 600 GB memory.
• Job 7099: grid2-role chunk 2 of 3, running for 2 h 25 min on 1 node with 256 CPU cores, 4 NVIDIA H100 graphics processors, 600 GB memory.
• Job 7098: grid2-role chunk 1 of 3, ran 4 h, stopped early due to time limit, on 1 node with 256 CPU cores, 600 GB memory.
• Job 7103: the fold line, timing, ran 8 min, finished normally.
• Job 3449021: the context line, chunk 1 of 2, ran 1 h 52 min, finished normally.
• Job 3449022: the context line, chunk 2 of 2, ran under a minute, finished normally.
• Job 7090: beta01, ran 2 h 6 min, finished normally.
• Job 7086: PC1, ran 2 h 46 min, finished normally.
• Job 7084: trial run, ran 5 min, finished normally.
• Job 3448912: the context line, timing, ran 1 min, finished normally.

What finished

• Job 3449022: the context line, chunk 2 of 2, ended at 2026-10-04 22:54 UTC.
• Job 3449021: the context line, chunk 1 of 2, ended at 2026-10-04 22:53 UTC.
• Job 7103: the fold line, timing, ended at 2026-10-04 21:00 UTC.
• Job 3448912: the context line, timing, ended at 2026-10-04 20:53 UTC.
• Job 7090: beta01, ended at 2026-10-04 09:16 UTC.
• Job 7086: PC1, ended at 2026-10-04 03:58 UTC.
• Job 7084: trial run, ended at 2026-10-04 00:43 UTC.

The numbers

• Total runs: 618
• Runs finished: 571
• Runs running: 8
• Runs not started: 39
• Finished share: 92.4%
• Node-hours used so far: 15.18
• Core-hours used so far: 578.1 (grid1), 322.3 (grid1b), 243.3 (ctx1), 1189.1 (grid2-role), 410.7 (fold1)
• Mean run time: 5.2 minutes

Failures and what was done

No failures occurred today. All jobs either finished normally or are still running.

What was learned

• The control run (job 7086) did not reproduce the published result: best checkpoint 0.7607 vs 0.9963, last 0.2110 vs 0.0000.
• Changing the Adam betas to (0.1, 0.98) did not close the gap: best checkpoint mean 0.7825, last 0.2449.
• The training curves show validation swinging wildly while training loss is at floor, suggesting instability in the control.
• The context line timing and chunks completed, allowing pass line checks for ctx1.

What was decided

• The pass lines for grid1, grid1b and ctx1 can be read now that those grids are complete.
• The pass lines for grid2-role and fold1 are not read yet because those grids are not finished.
• The baseline control does not reproduce; add-jump is withheld until the issue is resolved.

What is next

• The fold line, chunk 1 of 4 (job 7104) is running now.
• Grid2-role chunk 2 of 3 (job 7099) is running now.
• Grid2-role chunk 3 of 3 (job 7100) is waiting: waits for the job before it to finish.
• The fold line, chunk 2 of 4 (job 7105) is waiting: waits for the job before it to finish.
• The fold line, chunk 3 of 4 (job 7106) is waiting: waits for the job before it to finish.
• The fold line, chunk 4 of 4 (job 7107) is waiting: waits for the job before it to finish.