Multiple comparison correction in an eploratory study

Hello EEG Community,

i’d like to hear your opinion about multiple comparison correction (e.g., Bonferroni correction) in an exploratory resting-state EEG study.

In my case I want to compare the mean power spectrum of different frequency bands (theta, alpha, beta, gamma) from 19 electrodes, at different conditions (3 conditions) and on different days (with several measurement sessions each day).

In such a design, a rigorous correction for the multiplicity of tests is hardly possible. In my opinion most of the correction methods are too strict and might underestimate effects. One way could be to summarize the data: e.g. to group the electrodes into areas to reduce the number of tests. But this comes with the disadvantage of maybe losing important information.

It is an issue that I have discussed with my colleagues and we have not yet reached an acceptable solution.

I would like to hear your opinion about it and how you would deal with it.

Thanks in advance!

Niko

Clearly the solution is to simpler research :wink:

Just kidding, of course. You will definitely need to do some kind of dimension reduction. You can do that statistically on the channels (e.g., taking ICA or GED of the channel data to get to one component instead of 19 channels) or by selecting channels, as you mention. You can also reduce dimensionality by averaging and differencing. For example, perhaps it’s reasonable to average the recordings with each recording day, and by computing the condition difference instead of treating each condition separately.

Yes, this comes with the potential cost of losing information, but I think you will need to do that in order to avoid having to test at p<.0000000001.

Alternatively, if you can justify strong a priori hypotheses, then you can test those hypotheses at p=.05 and not worry about the billion other possible comparisons.

I think a combination of dimension-reduction and stricter hypotheses will be a good solution overall.

1 Like

Clearly the solution is to simpler research :wink:

these were actually the last words in the discussion with my colleagues :smiley:

We also came to the same solution that we have to do some kind dimension reduction by grouping electrodes into areas and also to do some averaging and differencing.
I was going through the literature on how to deal with multiple comparison corrections in EEG and i found nearly everything from loose to strict… :wink: but I think the above solution is good overall!

Thanks for your opinion, Mike!