https://docs.google.com/presentation/d/1CE3y51ShyvE3U_Bs3KjqRttfk1oweDXNV4HEK8JYbCY/edit?usp=sharing
Summary
The fMRI Foundation Model project aims to create a foundation model for functional MRI data that can extract meaningful neural embeddings from raw, noisy fMRI data. This is the first public meeting for the recently relaunched MedArc community project, which operates independently from Sofont (the medical AI startup founded by Paul and Tanishq). The project has been under development for about a year and builds on MedArc's previous success with the MindEye project that reconstructed scene images from fMRI activity.
The team's approach involves:
This flat map approach balances structure and flexibility - it preserves more information than traditional parcellation approaches while being more structured than raw 4D volume approaches.
The project welcomes contributors with various backgrounds, though it's not designed as a mentoring service. Participants are expected to work independently on research tasks. The codebase is organized with a flat structure to make it accessible, and the workflow involves forking the repository and submitting pull requests. Weekly meetings will be used to catch up on progress and allocate tasks.
Notes
Transcript
Hey y'all.
What's up?
Hey Cesar, hey Morgan, everyone else. Maybe I can just start by giving a little introduction about MedArc. And then we can maybe do a little bit of intros and then Connor has a slideshow that he can share about the project and we can sort of take it from there.
If it all sounds good. So first of all, thanks for coming to the first public meeting for this project. We successfully relaunched MedArc. I'm very excited to kick things off with this fMRI Foundation Models meeting, specifically the fMRI Foundation Model Project.
has been kind of under development for a while now, maybe a year or so. But it's kind of been off and on because, you know, the history of METARC was that... We started like three years ago as a Discord server doing various medical AI projects. The most noteworthy was the MindEye project where we were reconstructing scene images from fMRI activity.
And so we had a lot of success there, being able to work with the community, being able to publish top tier papers, and showing that it actually is quite useful to work publicly in the open, sharing compute with volunteers, benefiting a lot from the crowdsourced approach to doing science, and I think that's still a sort of underutilized asset.
that even existing other Discord servers are not quite embracing to maybe the best of their abilities. We had some ups and downs because we lost access to compute for a while. Me and Tanish left Stability AI, which is where we were previously working at.
But now we founded our own medical AI startup called Sofant, doing medical foundation models. MedArc, by the way, is independent from Sofant's sort of commercial side. applications of fine-tuning things for medical, like pharma R&D sort of things. So, MedArc is the sort of open science
server to do all of these research projects in the open, release models openly, publish papers with the community, and all of that stuff. It's very similar to how we were We've been working for the past few years, but now we're trying to, you know, fully embrace this again and put some more sort of standards in place to ensure that everybody is benefiting.
To make sure that people can use the same sort of like shared file system, have compute, have regular meetings, consistency, project leads that are keeping everyone up to date on what's going on. These projects aren't going to. It is allowed due to various reasons of not having a project lead who can fully support the project or lack of consistency for how people are.
getting up to date with the project. So we put a lot of thought into how to sort of get this Discord-based approach to work. And so I really appreciate everyone. Being interested in this kind of, you know, non-traditional way of doing science together. And I hope that it benefits us all in terms of like sharing ideas, experience.
We're working together to make some really cool and interesting models and putting it out there publicly. So in terms of volunteering, right, so we are not meant to like be a mentoring kind of thing. Service, just to put that out there initially, like, we do expect people are coming in here to, like, work together on research, like, independently learn how to do things and collaborate. That's not to say that you have to be like an expert. You know, we've worked with high schoolers, undergrads.
PhD students, postdocs, industry professionals, clinicians, et cetera, across all these projects. And I think that everyone has something to offer, but we are not able to provide a mentoring service, if that makes sense. So, we will have tasks and things that we would like to work together on, and people are expected to independently be able to make
to that, the sort of big picture of how we're operating, like the details and, you know, if you're interested in the specifics of this. Take a look through our Notion website at medarc.notion.site, and you can get a lot more information about this. So introductions, I'll go ahead. So I'm Paul, CTO at Sofont.
I'm trying to do my best to support all this MedOrg stuff. I was previously a postdoc at Princeton, which is where I was initially doing computational neuroimaging projects, and that's how I eventually transitioned to... And stability. Connor, if you want to give an intro and then we can sort of round Robin style it. Yeah, so I'm Connor. I'm a visual scientist at so far.
I joined the team recently, like in June, but I've been working with Paul and with MedArc for a while, like at least maybe around two years. I've been working on various projects, mostly around doing machine learning on fMRI data, which has been my interest for a super long time.
So I'm now leading this project, which is the MRI foundation model. This is basically a project that I've been begging every place that I worked at to do. for like the past several years, and so I'm really excited to be able to work on it finally, and I'm hopeful to try to, you know, wrangle the energy and get us all, you know,
You know, collaborating on this effectively will be sort of my first time leading a large community project so I'm hoping to, you know, respond to feedback, figure out how things what things are working well what things are. Hopefully we'll all figure out how to do this together and have some good papers come out of it.
Great. I'm here. You want to give an intro?
Sure. Hey guys, I'm here and I'm a postgraduate researcher here at Baylor College of Medicine and I've been working with Paul and the nation. I've been working with Cesar and the entire MADARC team for like two years. Started working with them on MindEye 2 and then the Algonauts project. Yeah, really happy to be working on the foundation model for fMRI.
Yeah, thanks for being here. Cesar?
Hi, everyone. I'm Spencer. And I've been working on my life for two years, maybe a little bit more. And. Yeah, it's been great working here and apart from that I've been doing kind of machinery engineering and software engineering in different startups.
So, yeah, happy to collaborate in this project. Yeah, appreciate it. Morgan. Yeah, hi, I'm Morgan. I work on psychiatric neuroimaging. I'm also community manager at NeurotechX and do a researcher at Biopunk Lab. Cool. Yeah. John.
Hey everyone, I'm a machine learning engineer based out in DC, and I work on surgical video. Yeah, nice to meet you. Omar.
Hi, my name is Omar Perez Torres. I am a medical student with a strong interest in deep learning. I I research brain-computer interface. I would like to collaborate with the project.
Um, Tanish.
Hello everyone. I'm Tanishq. I am the CEO of Sofod. I previously also founded MedArk while I was a research director at Stability. So yeah, really excited to be here now that we've kind of relaunched MedArk. Yeah, it's great to see this happen. So I'm glad to see everyone joining us for this project.
Thanks.
Rishabh?
Sorry about that. I'm Rishabh. I'm a research assistant at Princeton. I'm reading the real-time MindEye. It's a project here that hopefully some of you guys can come to later, but just hoping to listen in to this meeting. Yeah, come by in, in two hours, and Richard will tell us all about the real time MindEye.
I'm sorry to solicit during your meeting. I'm coming. I'm going to stay up. It's gonna be so late for me. Yeah, geez. Daniel. Hello, everyone. My name is Daniel. I'm the other engineer, researcher, whatever, at Saffont. I lead the pathology projects.
Come by tomorrow at 4 p.m. Eastern to hear about the pathology projects. Yes, yes, it'll be exciting stuff, so. Sam?
Hi, everyone. It was nice to meet you recently. So I'm myself, I'm a postdoc in Berlin, associated with Shayate and Hertia AI from Tübingen, and working on some pre-training ideas for EEG, MRI, and multimodal models. So Connor gave me a heads up about this meeting.
I was curious to hear what you're up to and looking forward. Yeah, glad you could come. Ria?
Hello, y'all. I'm Rhea. I recently graduated from UT Dallas with a bachelor's in cognitive science, and I joined the Discord group like two days ago, so I was kind of curious. So, yeah, that's why I joined the meeting.
Um, Clara?
Hi, I'm Clara Fontenot. I'm a research scientist currently based in France. I'm actually working with multimodal imaging, so PET and fMRI, and I've been looking at similar approaches currently on my own. And I actually just found this project I was actually super interested in seeing how we can collaborate based on that. Super cool. Awesome.
Manish Manish. Hey everyone, nice to meet you guys. I'm undergrad from University of Washington. Currently, I'm working on proofing design and synthesis with Upgrader UMAPs, and I'm excited to co- Thanks for joining Leon. I'll just read your message. PhD candidate writing thesis.
Research focuses on deep learning on raw mass spec data. Cool. And I hope you feel better. Thanks for joining, even though you're sick. Omar? No, you already talked. I didn't go in order, so I have to make sure I'm not skipping anyone here. Sharwan?
Hey everyone, I'm Shravan. I'm one of the technical co-founders of Axon Health from India, Bangalore. We are trying to basically build a system to get the longitudinal We have like different problems here and would love to contribute and be a part of this journey.
Ragnar?
Hi, everyone. I'm Ratna. I'm a machine learning engineer with experience in medical image analysis. I'm interested in foundation models, so I'm here to have a look at the different projects and see how I can contribute to. Right. Rishikesh.
Hi, I'm Rishikesh Zavar. I'm a research engineer currently and I'm also pretty interested in like AI multimodal models and like in the medical side as well. So looking to contribute to you. Thanks for coming. Shaqnazar.
Hello, I can't turn on my camera, but I'm a robotics engineering student in Kazakhstan at Nazarbayev University. Currently, I work with like... Visual multimodal reasoning for coronary angiography images or syntax score estimation, if you know this problem. And before that, I was working on...
Applying some sort of image processing prior to masked out encoders to make like efficient representation learning from those images. I'm currently interested in both visual reasoning and self-supervised learning, so maybe I can contribute in those domains somewhere.
Hey guys, I am a co-founder at this company that uses EEG signals to decode speech. Our goal is to decode your inner speech so you wouldn't have to speak at all while wearing a non-invasive wearable and Eventually get to open vocabulary detection so that you can say whatever you want and control any robot or computer you want with just a cap.
Thanks, Amadip. Also to mention, you were co-first author on the MindEye paper, so that's also a big, noteworthy accomplishment. All right, so I think if anybody hasn't yet introduced themselves, speak now, because I might have missed somebody because I wasn't going in order.
Okay. That said, I guess Connor, you can go ahead and screen share your slides.
All right, everyone sit down.
Let's see if this works.
You don't see the slides? Yep.
Okay. Nice. All right, perfect. Windows all set up. Okay. So this will be an overview presentation. Mostly covering motivation and like our current approach. Basically, we've been working on this FMRI foundation model project off and on, as Paul said, for a while.
Most recently we started it back up in June and it's been mostly just us so far working on it. Now we're opening it up for everyone to get involved. So I'm just going to be trying to catch everybody up with where things are at right now. Okay, so some background on what an fMRI foundation model is even. So it's pretty simple. The idea is
You take this raw, noisy, complicated fMRI data, which is structured like a 4D time series of 3D volumes. And we're passing it through a big over-parameterized neural network. And the goal is to get out these neuro-embeddings. So basically just like high-dimensional embedding vectors that capture important information.
that can distinguish people along important and clinical dimensions. For example, you might have clusters or subspaces that emerge in the embedding dimension that differentiates people with neurodegenerative disabilities. The raw fMRI data is complicated and noisy and you can't make sense of it directly, but we want to pass it through a known network and get out a structured embedding that we can then make sense of and use for downstream.
applications. Okay.
So why do we want to do this? Why aren't we satisfied with our current approaches in translational neuroscience? So people have been trying to apply brain science in general to impacting people's mental health or impacting the status of treatment for neurodegenerative disease or neurodevelopmental disease.
It isn't really working. This is a widespread observation in the general field. This is some quote and figures from a recent book, which is pretty good, called Elusive Cures, from a prominent researcher in And she makes this figure in her book, which shows on the bottom left, there's lots of papers being written every year.
on new interesting neuroscience findings, but these are not really being translated into new treatments or new drugs to actually impact people's brain and mental health. So, like, why is this happening? What's wrong with the current approach in neuroscience, and why isn't that translating into actual clinical application?
To take another perspective, this is a quote from the director from the NIMH, who was the director from 2002 to 2015, Thomas Insel. This quote is also taken from the same book. Basically, he's just saying that they funded a ton of stuff. They spent a ton of money on neuroscience, basic neuroscience, and it didn't really convert into important, like clinical.
And so if this isn't working, obviously we should try some other different things. And people are going to be trying lots of different things as the field looks at this issue. And tries to make progress. Obviously, we don't have all the answers. We're not going to pretend like we know the answer to this big problem, but we can try some stuff. And the thing that we're trying is...
This foundation model strategy and the gist of the foundation model strategy. I assume everybody has kind of heard the term foundation model, but the gist of it is pretty simple. You take a giant amount of data from some domain, you take a giant GPU compute cluster, and you merge them together to train a big neural network.
using some often self-supervised training strategy. And what you get out as a result is this, you know, embedding machine like I was talking about. This sort of strategy has been applied in all sorts of domains. First, it was applied in classic machine learning, deep learning type domains. And then now, more recently, it's been applied in scientific domains, like we're seeing here, three examples from the past three years.
where the foundation model strategy was first applied to retinal images, to great scientific success, digital pathology images, which as it turns out we'll hear about tomorrow from In each of these cases, the strategy is working. Where you take the data, large amount of data from some domain, combine it with the compute and some appropriate model and training strategy. And you get out something that's at least scientifically useful and we hope will be ultimately like practically useful.
In our case, that means. Useful for clinical, mental health, brain health applications. And so this is the goal and this is why we think it might work.
With that motivation on what we're doing and why we're doing it and why it might work, let's switch and talk about functional MRI data. Sort of more specifically and what the challenge of modeling it is so I would guess like there's a handful of people here that know function MRI data very well and don't need to review the basic review, but some people probably aren't so familiar. So I'll just go over it anyway.
I always like showing videos of data. This is a video of raw fMRI data. You see three slices, like a slice down the middle of the brain on the left, a slice going this way, and then a slice parallel. to the horizontal plane. And if you can pay close attention, you can see tiny variation in the pixel values as the movie plays.
And this tiny variation is the fMRI signal that we're interested in. And this signal is specifically measuring fluctuation in blood oxygen level. And the way the brain happens to work is when neurons fire, the body oversupplies oxygenated blood to those firing cells. And as a result, you see this increase in blood oxygen signal.
And this is what fMRI measures. And the most important point is that the fMRI signal is a tiny fluctuation in the relative magnitude of the overall. The overall data, so it's only 1 to 2% of the of the overall variation that you can see.
Okay, so how do we go about modeling these data? So like I said, it has some like critical challenges that we should be aware of and like Key features that we should keep in mind when we're trying to think about how to model fMRI data. So first of all, the signal-to-noise ratio is very low. Like I said, the fluctuations are tiny, 1-2% changes.
Most of the contrast that you see is actually due to differences in tissue type and not really related to the actual neural. The signal is spatiotemporally smooth because the vascular system is kind of smooth and brain in general. You know, nearby areas are correlated in their activity and then probably sort of the most subtle issue that people coming from outside FMRI don't often appreciate is that the actual
signal that we're interested in, the blood oxygen signal, is concentrated in the thin ribbon of cortex. So like you can see in this figure on the left, bottom left, the cortex highlighted in white. Like this thin ribbon is where all the neuron cell bodies are and this is where all the blood is being supplied and where all the bold signals being measured.
And so it's really only in this area, as well as subcortical structures, which are not shown, where the key data that we're trying to model lives. The rest of it in the interior of the brain and definitely outside of the brain is not really of interest.
That is one key feature. Finally, another important feature is the brain has a standard map where Roughly the same, each person has roughly the same areas and roughly the same positions that do roughly the same things like modulo some wiggling around. Mostly the brain has a consistent map.
Sort of similar to how the earth has a consistent map of continents always in the same place. The brain has a consistent functional anatomical map. So we should really leverage these properties when we go about trying to model the data.