Goal: Submit to ICML 2026 (abstract deadline Jan 23, paper deadline Jan 28; 2 weeks left)

Agenda

  1. sdxl with special515 test set ( @Cesar Kadir Torrico Villanueva )

    1. using CLIP ViT-H/14, do retrieval using last hidden layer (higher dimensionality) and reconstruction using last layer (lower dimensionality) from SDXL Turbo. train on session 1, test session 3 using offline processing and compare with existing results from CLIP ViT-bigG.

  2. amount of training data vs. performance ( @Anish and @Cesar Kadir Torrico Villanueva )

    1. create new checkpoints trained on sessions 1 and 2 with varying amounts of training data. increments between 0-100% of data, where session 1 only=50% and both sessions together=100% ( @Anish )
    2. using those ckpts, test each on session 3 ( @Cesar Kadir Torrico Villanueva )

    train on session 1+2, test session 3 rt-processing, delay=0. lr=1e-4 (not converging at 3e-4). using session 1 reliability mask. after 50%, session 2 gets added.

    use glmsingle betas separate for sessions 1 and 2, with lr=3e-4

    image.png

    image.png

    image.png

  3. benchmark latency ( @abhi )

  4. clean up realtime code to make more modular and remove monkey patching ( @Rishab Iyer )

    1. add torch compile and measure speed improvement ( @Rishab Iyer )

  5. make icml roadmap + interim deadlines ( @Rishab Iyer )

  6. quantify difference in image statistics of NSD vs. MST images ( @Mira and @cindyhfls )

    1. FID and/or density measures ( @Mira )
    2. cosine distance and dissimilarity matrix to compare reconstruction evals ( @cindyhfls )

  7. glmsingle reliability

    1. do the various optimizations included in glmsingle still improve reliability of beta estimates even if the input data is real-time preprocessed (as opposed to fmriprep)? to test this, generate glmsingle reliability plots as seen here in cells 52, 66, 69, 70. we already have this for fmriprep preprocessing; run this for real-time preprocessed data ( @Ernest Lo )

  8. downsampling real-time data

    1. take sub-005_ses-01 data, downsample with trilinear interpolation from 2.0 → 2.5mm voxels ( @SaberToaster )

      image.png

    2. fine-tune (also need to update the reliability mask correspondingly) on ses-01 ( @duineeya. )

    3. test on ses-03 while doing same downsampling in real-time. is it quick enough to apply this within a couple of seconds? (open)

Next steps

  1. sdxl with special515 test set ( @Cesar Kadir Torrico Villanueva )
    1. using CLIP ViT-H/14, do retrieval using last hidden layer (higher dimensionality) and reconstruction using last layer (lower dimensionality) from SDXL Turbo. train on session 1, test session 3 using offline processing and compare with existing results from CLIP ViT-bigG.
  2. amount of training data vs. performance ( @Anish )
    1. create new checkpoints trained on sessions 1 and 2 with varying amounts of training data, test on session 3. increments between 0-100% of data, where session 1 only=50% and both sessions together=100%
      1. due to issues discussed last time, use glmsingle betas separate for sessions 1 and 2, with lr=3e-4 to match the lr used in the prior single session tests. 50% data here should therefore exactly match the 100% data from single session.
  3. benchmark latency ( @abhi )
    1. identify how much each part of the pipeline takes: data stream from RT-Cloud, motion correction (mcflirt), registration (flirt), GLM (nilearn), mindeye forward pass, display on psychopy
    2. for now, just focus on the parts within mindeye.ipynb (ignore data stream from RT-Cloud and display on psychopy, just the preprocessing, GLM, inference)
  4. glmsingle reliability
    1. do the various optimizations included in glmsingle still improve reliability of beta estimates even if the input data is real-time preprocessed (as opposed to fmriprep)? to test this, generate glmsingle reliability plots as seen here in cells 52, 66, 69, 70. we already have this for fmriprep preprocessing; run this for real-time preprocessed data ( @Ernest Lo )
  5. downsampling in 3T
    1. take sub-005_ses-01 data, downsample with trilinear interpolation from 2.0 → 2.5mm voxels as well as corresponding brain/reliability masks. fine-tune on ses-01, compare results with/without downsampling for identical special515 train/test split within ses-01. should first be done with offline pipeline, not realtime, to determine feasibility ( @SaberToaster and @duineeya )
    2. if results from above look good, test on ses-03 while doing same downsampling in real-time. is it quick enough to apply this within a couple of seconds?
  6. clean up realtime code to make more modular and remove monkey patching ( @Rishab Iyer )
    1. add torch compile and measure speed improvement ( @Rishab Iyer )
  7. make new union mask based on sessions 1-2 only (not peeking at session 3 data) ( @Rishab Iyer )
    1. after this, re-run training data vs. performance analysis using this mask. compare to results using session 1 reliability mask; this should be consistently better ( @anish )
  8. Draft abstract/introduction for icml paper ( @Rishab Iyer )