Columbia University use case

How Neuroscientists Are Using AI To Understand Behavior

The Project

Dan Biderman is a Ph.D. candidate At Columbia University’s Center for Theoretical Neuroscience, advised by Statistics professors  Liam Paninski and John Cunningham (both  are also affiliated with the Grossman Center for the Statistics of Mind)

Dan is currently focused on a project modeling animal behavior in videos, working to better understand how different brain regions (such as the cerebellum or the motor cortex) control natural movements. We connected with Dan to learn more about this project and how utilizing Grid has enhanced their research.

Columbia researchers started by tackling the well-known problem of pose estimation: how can we detect an animal’s joints from a raw video? With this information in hand, they hope to ask: What are the mechanical forces acting on each joint? What control strategies do biological agents use? 

Pose estimation is a crowded area of computer vision. The research relies mostly on supervised learning: researchers collect a large amount of manually annotated images and train a neural network to predict joint positions from those images. Dan’s team is taking a new semi-supervised approach: they observed that raw unannotated videos already contain a rich spatiotemporal structure that is ignored by standard approaches. They are developing a video-centric software package called Lightning Pose, that combines PyTorch Lightning and Nvidia DALI to efficiently load video sequences to GPU and use them to train convolutional neural networks. As of now, they focus on videos collected by neuroscience labs that track whole-body movements, arm reaches, and eye movements. And Dan used his various Grid sessions to develop and test new statistical tricks to allow the network to make sense of unannotated video frames. Dan and his team hope that in the coming months, their package could be used also on top of Grid.  This way, researchers in the neuroscience community could reproduce their results, contribute new ideas to their package, and perform scalable pose estimation neuroscience projects without worrying about infrastructure.


The Challenge

Before discovering Grid, the team managed all their cloud experiments themselves on AWS instances using boto3 scripts and vanilla PyTorch. They used S3 buckets to store large amounts of videos and images and ran into challenges setting up environments and managing SSH and Git. “90% of our time was invested in just making sure code runs,” said Dan.


The Solution

Dan learned of in the PyTorch Lightning Documentation and became curious about the workflow, and how Grid could help solve their pipeline management problem. Once implementing Grid, it allowed him to prototype new models for pose estimation. They now have less code than existing algorithms due to a reduction in boilerplate using PyTorch Lightning, an integration with Nvidia DALI, and a solid State-Of-The-Art pose estimation package to help with future neuroscience projects.



As their Grid Sessions evolve, Dan is looking forward to having a more pre-configured environment. He is currently enjoying the VS code integration, and also loves the Grid Community; as it provides the best support in an accessible timeframe, and when stuff goes wrong there are other humans they can work with.

Grid exceeded Dan’s expectations through the Jupyter Lab Community of developers, SSH login logout, and the ability to see the status of the compute. The interaction with Instances from Grid CLI is also more pleasant to work with than AWS, and it is very clear to see the status.

“If someone’s use case is prototyping models, where there is a lot of uncertainty around model architecture, getting started with Grid will be much faster than doing it yourself on a cloud provider. In contrast to a university cluster, being able to explicitly control which hardware you are using is beneficial.” – Dan Biderman, Ph.D. Student


With Grid, longer-term plans will evolve as more users in the lab use Grid for their projects. The ability to share model results, simplify data loading, and have one platform to capture results will become more streamlined.


Getting Started with Grid

If you found this customer story informative and want to see how Grid can help you manage deep learning model development for your next project, get started with Grid’s free community tier account by clicking here. Also, explore our documentation and join our Slack community to learn more about what the Grid platform can do for you.