0

I have some numerical measurements on two groups of people and I would like to compare means between these two groups. Just using a t-test for that purpose which gives me a confidence interval and p-value. Now, I'd like do a bootstrap analysis on this data to get a feel for the variability of both the CI and p-values.

I'm using R and the boot package. The data is stored in dataframe data. To calculate the statistics I have this function:

calculate <- function(formula, data, indices) {
    d <- data[indices,]
    m <- t.test(formula, data=d)
    return(c(m$conf.int, m$p.value))
}

Then I run the bootstrap as follows:

results <- boot(data=data, statistic=calculate, R=1000, formula=y ~ x)

Then I plot the p-values in "results" as follows:

hist(results$t[,3], breaks=32)

The histogram looks as shown below. I understand that the distribution of p-values is skewed because the p-value is constrained to be no smaller than zero. But I don't understand why the peak of the distribution is at zero as well, no matter how many breaks I display in the histogram.

Any insight would be greatly appreciated! Histogram of bootstrapped p-value distribution

StupidWolf
  • 45,075
  • 17
  • 40
  • 72
Ralph
  • 151
  • 11
  • From the topic tag - "R is a free software environment for statistical computing and graphics. This tag can be used for questions about running R software on macOS. Questions about programming in R should be asked on StackOverflow." – Tetsujin Mar 05 '20 at 16:52
  • Actually this probably should have been migrated to [stats.se]. Or at the very least include a [reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) here so we can run it to see what's going on. Otherwise Cross Validated can help with a more theoretical statistical answer. – MrFlick Mar 05 '20 at 18:46
  • @Tetsujin It's not a question about R or MacOS but about data. – Ralph Mar 07 '20 at 07:24
  • @MrFlick Thanks, I will repost the question in CV and try and provide an example that can be reproduced – Ralph Mar 07 '20 at 07:25
  • 1
    i think you are confusing bootstrap with permutation. if the original observation has a low pvalue, meaning there are indications of difference between groups, when you bootstrap you should something like this, because the difference is large enough for most of the bootstraps – StupidWolf Mar 07 '20 at 08:38
  • 1
    when you bootstrap, you simply resample your data.. so if the majority of your observations do agree with this difference, this is what you expect – StupidWolf Mar 07 '20 at 08:39

1 Answers1

0

We can try with an example dataset:

library(boot)
set.seed(111)
data = data.frame(y = rnorm(60,rep(0:1,each=30),1),x=rep(1:2,each=30))

results <- boot(data=data, statistic=calculate, R=1000, formula=y ~ x)

The original observation itself is significant:

t.test(y~x,data=data)

    Welch Two Sample t-test

data:  y by x
t = -4.0621, df = 55.339, p-value = 0.0001547
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.8262396 -0.6197054
sample estimates:
mean in group 1 mean in group 2 
     -0.1881403       1.0348322

And even if you sample it you, most of the bootstraps will show a significant value, but we can see how this varies on the -log10 scale, actual observed value in blue:

hist(-log10(results$t[,3]),br=20,xlab="Bootstrap -log 10 p values")
abline(v= -log10(results$t0[3]),lty=8,col="blue")

enter image description here

StupidWolf
  • 45,075
  • 17
  • 40
  • 72