Dear Prof. Cohen,
I have a question concerning source separation for steady state responses in MEG using RESS.
MEG (at least Neuromag systems) have two different sensors types (with two different sensitivity profiles). Does this affect the way we want to calculate our generalized eigendecomposition?
More in detail, would you advise to run GED on both sensors type or would you keep them separate, thus calculating GED first on e.g. magnetometers and then on planar gradiometers?
Dear Prof. Cohen,
Hi Lorenzo. Please call me Mike
One way to proceed is to normalize all the sensors (e.g., z-transform over time) and then combine them into the same GED. This is useful because it incorporates all of the information you have across all the sensors. When visualizing the spatial filter forward models, you’ll need to separate the gradiometers and magnetometers, thus making two topoplots for each filter.
I think I managed to perform RESS on MEG data.
Here some preliminary results obtained on one experimental condition and four participants.
Visual stimuli were presented at 6 Hz, with one every 5 stimuli being an ‘odd’ stimulus. Thus what we would expect (assuming odd stimuli are processed) is a peak at 1.2 Hz and a peak at 6 Hz. 4 Hz is a frequency of no interest that I used to check for overfitting.
In red the results of RESS on the real data time series.
In black instead are the results of a permutation test where I randomly permute the data time series for 500 times.
RESS nicely separates the 6Hz peak.
4Hz, if we look at the SNR, it looks like an overfitting problem (there was no 4Hz stimulation). This is confirmed by the eigenspectrum, which does not seem to differ between real and permuted data.
1.2 Hz regrettably doesn’t reach significance, although we can see maybe a trend in the eigenspectrum… maybe with more participants…
This procedure however, took a lot of time. Few days of computation for only four participants. Permuting the data, filtering and new GED takes a lot of time (306 channels, 5 trials per participant [if we consider only one condition, I have more], each trial lasted 60 seconds).
Do you have any suggestion in this respect?
Do leave-one-out procedures take care of the overfitting problem as, in this case, permutation testing?
Also, would you change filters parameters between 1.2 and 6Hz?
Hi Lorenzo. Looks nice overall. “Overfitting” is a difficult term here, because there probably is real (non-SSVEP-related) brain activity at 4 Hz. It’s therefore also useful to look at the topographies of the filters (the eigenvector times the S matrix) – you might find that the 6 Hz component is posterior/visual whereas the 4 Hz component is anterior.
But days to run the analysis!?!?! Yikes, sounds like something went wrong or could be optimized. If you would like to share your code (you can also do this off-forum), then I can take a look and see if something can be improved. In theory, RESS should take tens of seconds to minutes. Unless you’re running this on one of these classics.
OK, based on the code you sent me, there are a few issues:
- The loop over channels doesn’t permute anything. Also, one of the advantages of eigendecomposition is that it isn’t based on any physical locations. So shuffling the channel order doesn’t do anything, unless the channel order is different on each trial (but this isn’t a good idea, see below).
- Randomly shuffling time points also isn’t a great idea. That’s changing the temporal autocorrelation structure of the signal, including the spectral structure.
In general, the idea of permutation shuffling is to keep the data as intact as possible, and change only the one key factor that is directly related to the hypothesis. In this case, the null hypothesis is that S=R. Thus, the appropriate way to shuffle is to randomly sort data into S vs. R. Here’s how I would do that:
- Filter the data in the three bands (SSVEP frequency and the two neighbors). Keep all of the filtered data. This will give you 3x the number of trials.
- Concatenate all filtered signals into one matrix that is chans X time X [3 x trials].
- To create the S matrix, randomly draw from 1/3 of those “trials”. Same for R_lower and R_upper.
- Then repeat the GED.
Notice that you’re not actually re-filtering the data on each iteration. You only need to filter all the data once, but the assignment of a given signal to S and R is randomized 1000 (or whatever) times.
Also keep in mind that for permutation testing, you don’t actually need to store the time series or maps on each iteration. You just need the sorted eigenvalues. That should also save computation time.
Hi Mike, thank you for your prompt response. I did as you suggested and is indeed much faster in terms of computational time.
I obtain however some puzzling results.
In black the average of the eigenvalues from the permutation test (500 permutations).
In red the eigenvalues, SNR and power without permutation.
Why would the average of the permuted eigenvalues be higher than that of the data where I don’t permute the filtered data from S and R?
hmm, yes that is strange. The null hypothesis values should be around 1, which you can see for the red line after the first few components. My first guess is that the shuffling isn’t quite correct. Please double-check that the data are roughly evenly randomly distributed into the S and R matrices. Feel free to email me the code again if you’re unsure.
Hi Lorenzo. Thanks for sending me your code and sample data. Really nice data!
I don’t think you’ve done anything wrong in the code. With only 5 trials, sorting the data randomly into S and R matrices gives N=10 for k=2 groups. That means that there’s only 2^9=512 unique ways of grouping the data. But that includes a lot of suboptimal groupings, e.g., when 4/5 single-trial covariance matrices are in the correct S or R matrix.
So in conclusion, I don’t think you can do permutation testing with such a small N. Indeed, the estimation error of the empirical p-value is roughly 1/N (ref), which means your p-value cannot be trusted to greater than a precision of .1
We discussed in the RESS paper several ways of doing statistics, only one of which was based on permutation testing for eigenvalues. Do you have multiple conditions with the same stimulus frequency? Then you can include all trials and look at power differences between conditions.
Hope that all helps,
hi Mike thank you for your help!
I indeed have multiple conditions with the same stimulus frequency. However, I expect a marginally different topographical distribution of the effects in the different conditions. Wouldn’t this be a problem?
Nah, I don’t think so. The thing is that you have light stimulation at 1.2 Hz, so no one will be surprised that you get a strong brain response at 1.2 Hz. You don’t need a p-value for everything.