Image by Gerd Altmann from Pixabay

Inference: Estimation and Confidence Intervals

Inference, trying to make sense of the target population from the sample data, is perhaps the most difficult topic in most introductory statistics courses. The Null Hypothesis Significance Tests (NHST) is commonly taught as the only way to make a proper inference. While that approach does work many times, other times the conditions and simplifying assumptions needed to use the classical NHST approach break down. Further, that approach requires students to learn that a number of different tests are needed in your NHST toolbox. There are z-tests, t-tests, F-tests, and Chi-square tests, to name the most common groups of tests.

Here is a typical decision tree/concept map (Grosofsky, 2013) intended to simplify the inference process.

Thankfully, there is a much simpler approach to inference tests. Professor Allen Downey stated in a blog post in 2011: “There is only one test!”

“All tests try to answer the same question: ‘Is the apparent effect real, or is it due to chance?’ To answer that question, we formulate two hypotheses: the null hypothesis, Ho, is a model of the system if the effect is due to chance; the alternate hypothesis, Ha, is a model where the effect is real.” (Downey, 2011)

Downey updated the 2011 blog post here in 2016: There is still only one test

We will use Professor Downey’s approach in this course:

  1. Calculate a sample statistic which we will call “delta”. This is the main measure you care about: the difference in means, the average, the median, the proportion, the difference in proportions, etc.
  2. Use simulation to invent a world where delta is null. Simulate what the world would look like if there was no difference between two groups, or if there was no difference in proportions, or where the average value is a specific number.
  3. Look at delta in the null world. Put the sample statistic delta in the null world and see if it fits well.
  4. Calculate the probability that delta could exist in the null world. This is the p-value, or the probability that you’d see a delta at least that high in a world where there’s no difference.
  5. Decide if delta is statistically significant. Compare the p-value against an evidentiary standard - the significance level (typically 5%). If the p-value is less than the significance level, we reject the Null hypothesis of no difference.

Of course, we will always begin our analysis by looking at the data we have, exploratory research, to better understand it and how we should use it in our process of inference. And we will include more visualizations in our final analysis.

Seeing leads to better understanding.

Let’s look at some examples as we work through the two rehearses for Lab 5:

When you work through each Rehearse session, you will use a worksheet to capture your progress as you follow the instructions in the Rehearse web pages. You can keep the Rehearse page open on one browser tab and the appropriate worksheet in your RStudio account in another.

Let’s get started with the first tutorial - Bootstrapping and Confidence Intervals.

Next:Lab 5 Rehearse 1 right

Lab Manual Homepage

Creative Commons License
This work was created by Dawn Wright and is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

V2.0.1, Date 7/8/24

Last Compiled 2024-07-08