Lecture 9: Tests of significance

Outline:
  1. A few R preliminaries
  2. What is a P value?
  3. Somethings about hypothesis testing.
Resources:
       Elementary statistics with R:  http://www.r-tutor.com/elementary-statistics
       Summary of R statistics functions covered in class: R statistics
       Minimal examples of different markups with explanation: http://yihui.name/knitr/demo/minimal/

Some preliminaries:
  • Be sure to have installed MikTex (instructions from last time). This is done outside of R.
  • Use the knitr library:   library('knitr')
  • Spin from a .R to a .Rnw file:  spin('my.R', format='Rnw')
  • A button should appear in R studio when you select the resulting file.
Sidelights on generating random numbers (discussed for N(0, 1)) but other distributions are similar
  • pnorm(x)  gives cumulative probability for values drawn from the distribution.  For example pnorm(3) is the probability that a value drawn at random from N(0, 1) is <= 3.
  • qnorm(q)  gives the data value of quantile q. For example, qnorm(0.25) which is -0.67449 is the data value such that 25% of the population values are below it.
  • dnorm(x)  gives the height of the probability histogram at point x.
  • rnorm(n)  generates n random numbers from the normal distribution 

P-values or tests of significance

P-value -- the probability of the result being observed by chance (random draw). If the result was unlikely to be observed by chance, then we can conclude that the observation is "significant" or "interesting". The probability is usually by evaluating a test statistic and determining how likely it was to have observed a value of the test statistic that was at least that extreme. 
 
P-values come up in many different situations in which a scientist or statistician wants to establish the significance of a result. Sometimes p-values are determined empirically, by shuffling and simulating "random draw". In hypothesis testing, the researcher formulates hypotheses and a statistical test to determine whether to accept or reject the hypothesis.

Example 1: We make a single measurement and want to know how likely it was to have come from N(0, 1). That is, we want to know if this sample was drawn for a population described by the normal distribution with 0 mean and standard deviation 1.  What we can say about this will be determined by the value of the point drawn. If the value is far away from the center of the bell-curve, it was not very likely that it came from N(0,1) (though it is still possible).

Standard normal distributions from wikipedia

Suppose the value we draw is 3.  The test statistic is just the probability that we would draw a value more extreme than 3 from the normal distribution. How we calculate this depends on whether we are doing a one-sided or a two-sided test.   The probability that x > 3 is 1 - pnorm(3) or 0.00135.  For a two-sided test we want the probability that |x| > 3 which is 0.0027. This is the p-value for our problem.  There is only a 0.3% probability that we could be wrong if we say x didn't come from N(0, 1). We play the odds.

On the hand, if the value we drew was 0.5, then it could very well have come from N(0, 1). 38% of the values are in [-0.5,0.5] and 62% are outside this interval.  On the other hand, the value could have come from another distribution such as N(0.5, 1) or N(0, 1.5). We can't say much of anything.

Usually, we have a neutral hypothesis (the null hypothesis) and a test hypothesis (the alternative hypothesis). We try to determine how likely the test hypothesis could have happened by "luck of the draw". 

To formalize:
  1. Test hypothesis (alternative hypothesis) is that the number did not come from N(0,1)
  2. Neutral hypothesis (null hypothesis) is that the number did come from the N(0, 1)
  3. The test statistic is just the probability of a value from the normal distribution satisfies |x| > a.
  4. Set the significance level --- if there is less than a 5% chance of making, we'll accept the test hypothesis.
For Example 1:  
  1. Test or alternative hypothesis:   value a does not come from N(0, 1)
  2. Neutral or null hypothesis:    value a comes from N(0, 1)
  3. Test significance level  Pr(|x| > a) < 0.05
Using qnorm(0.025) we find that a = 1.96.  In other words, we have less than a 5% chance of picking a value from the Normal distribution that is more than 1.96 standard deviations away from the mean.

Example 2:
Look at t.test and the examples from  http://www.r-tutor.com/elementary-statistics.