💻 🧠 Brainhack school is taking a break, we'll be back in 2022! 🧠 💻:

Combine EEG/MRI/Behavioral data-sets to learn more about Music/Auditory system

By Marcel Farres Franch
Published on June 11, 2020

"In this project I aim to combine data from different modalities (fMRI, EEG, and behavioral) to understand more about sound and music processing. My main focus in this project was to try to reproduce some of the results from a published paper starting form raw data."

Combine EEG/MRI/Behavioral data-sets to learn more about Music/Auditory system



I'm currently a PhD student of the IPN at McGill University.

In this project I aim to combine data from different modalities (fMRI, EEG, and behavioral) to understand more about sound and music processing.

My main focus in this project was to try to reproduce some of the results from a published paper starting form raw data.

The overall goal of the current project is to be able to organize, pre-process and do some basic analyses form a fMRI study.

Project definition


In my current PhD project one of the end results should be a open multimodal behavioural and neuroimaging dataset characterizing healthy human auditory processing. It aims to allow researchers address individual differences in auditory cognitive skills across brain functions and structures, and it will serve as a baseline for comparison with clinical populations. To achieve that, our core objectives are to create a standardized framework with which to administer a battery of curated tasks. After acquiring the data from 70 young adults and we intend to share our framework, analysis pipelines, stimuli with linked descriptors, and metadata with the community through open data repositories. The dataset contains cognitive and psychophysical tasks, as well as questionnaires designed to assess musical abilities, speech, and general auditory perception. It also includes EEG and fMRI recorded during resting state, as well as naturalistic listening to musical stimuli and speech.

During this BrainHanks School project I wanted to understand what are the needs as a researcher to easily make use of public available data and learn the basics of pre-processing raw fMRI data.

Learning Goals

I have good experience analyzing highly process data… but how you get there?

  • Raw data → BIDS formatted raw data

  • BIDS formatted raw data → BIDS formatted preprocess data

  • Basic Data quality control

  • Basic analysis

  • Implement other people analysis and reproduce results

(but it is a 2 and a half week project…)

Estimating Time


The project will rely on the following technologies:

  • fmriprep

  • pandas

  • nipype

  • bids

  • bids-validator

  • nilearn

  • numpy

  • pathlib

  • jupiter lab

  • OpenNeuro


The first step was to search for candidates open datasets. I prioritized music/auditory related, as it is closer to my PhD project.

Next there is a list of interesting datasets I found, which I choose 2 to work during this course:


  • Forrest Gump: A very interesting study that use naturalistic stimuli to understand more about how our brain interacts with the real complex world. More information in the dedicated web-side. Here you can find the list of 29 publications that use the data made publicly available.

    I was particularly interested in the first 20 subjects that perform a music task (Listening to music (7T fMRI, cardiac & respiratory trace).

    I wanted to replicate the analyisis and findings of this paper.

    37bold, T1w, T2w, angio, dwi, fieldmapForrest Gump, objectcategories, movielocalizer, retmapccw, retmapcon, retmapexp, movie, retmapclw, coverage, orientation, auditory perceptionMaybe the most promising example


  • Neural Processing of Emotional Musical and Nonmusical Stimuli in Depression: Study looking how people with depression process differently musical stimuli.

    Same idea as the previous study, replicate results reported in this paper using the publicly available data.

    39T1w, boldMusic, Non-Music listeningmissing stims


Other Options


NOT Accessible

I also find some very interesting datsets that I was not able to access directly and in the period of 3 weeks I was still waiting for them.

I was a little frustrated with the process, making me realize how important is real Open Data.

  • DEAP: A Database for Emotion Analysis Using Physiological Signals:

    website: [https://www.eecs.qmul.ac.uk/mmv/datasets/deap/]

    paper: [https://www.eecs.qmul.ac.uk/mmv/datasets/deap/doc/tac_special_issue_2011.pdf]

    (Not Accessible without explicit permission, which we are missing for no)


At the end of this project, we have:

  • Scripts to download the dataset (HERE and HERE).

  • Scrips to pre-process the data using fmriprep in a cluster (HERE and HERE).

  • fmriprep report on the pre-processing (HERE).

  • Basic processing of 1 subject (HERE) with some need to be improve analysis.

Project plan / Objectives

Installation instructions

  1. Clone the repo to your computer:

    git clone https://github.com/brainhack-school2020/BHS-AuditoryMultimodal.git

  2. Install fMRI prep using this instructions

  3. Download and preprocess the data using these scripts (you can grab a coffee or two… these may take a while).

  4. Create a virtual-env python3 -m venv bhs-auditory

  5. Install requirements pip install -r requirements.txt

  6. Open the notebook jupyter lab BHS_AuditoryMultimodal-ds000171.ipynb


After the multiple problems I found trying to process the data, reproduce the analyses, and limitations imposed by the missing of information form the dataset I am more aware of what I will need to do to efficiency share my data/analyses in the near future.

Don't be that researcher…


See also these similar projects

fMRIPrep 101 - Pre-processing fMRI data and extracting connectivity matrices

This project aimed to understand how to pre-process fMRI data using fMRIPrep. Through this learning experience, a tutorial was created.

Discover this project

Diffusion MRI- From raw data to mapping brain connectomes.

The focus of this project was to combine, use and present a set of tools to organize, preprocess, analyze and visualize diffusion MRI data. The …

Discover this project

Revealing similarities between deep learning models and brain EEG representations

Do artificial neural networks process visual images similarly to our brain? If so, how? In this project, we bridge deep learning and brain EEG signals …

Discover this project