I tried to implement the non-parametric permutation testing described in “Analyzing Neural Time Series Data” in Chapter 33 (AMAZING book by the way!) for the test statistic spearmans roh. I used 5000 iterations where I permuted x and kept y as is. However the distribution of the p-values computed under H null is not normal (which it is in Figure 33.1B) but uniformly distributed.
Is that an issue?

Hi Erna. The distribution of H0 p-values is indeed not going to be normal. You probably want to be plotting (and using) the distribution of H0 correlation coefficients, not the p-values. I hope that wasn’t confusing from the book.

Hi Mike,
Thank you so much!
I have a follow up question. I have 10 variables for which I created a correlation matrix containing all possible parings (45 correlations of interest). For the permutation I used 5000 iterations, but needed to do 10 different permutation tests - permute x1 then calculate the correlation between permuted x1 with original x2, x3,… x10, then permute x2 and calculate the correlation with x3, x4,… x10 and so on. Do I have to adjust the p-values for the amount of permutation tests (e.g using something like bonferroni)?

Just a thought: If you’re correlating 10 pairs of variables, then it’s probably better to do a PCA or factor analysis of the correlation matrix to look for patterns. Correcting for multiple comparisons on 45 individual correlations is quite tricky. Bonferroni is probably too conservative, because I guess there are some meaningful relationships amongst the correlations. Anyway, p<.05/45 is ~.001, and even true correlations can fail to be significant at that threshold.

Yes, I tried that before using categorical PCA as the variables are ordinal and not normally distributed. However, the best fitting model where two components and they didn´t discriminate the data very well (most of the variables loaded on component 1). I thought I could use permutation testing as a method to control for multiple comparisons?

I see. Unfortunately, permutation testing alone is not a method to control for MC; it is simply a way to obtain an empirical null hypothesis distribution when an analytic distribution is unknown. Permutation testing can be easily integrated into cluster correction, which is a method for controlling MC.