Comparing Proportions and Analyzing Cross-Tabulations

Calculating observed and expected counts

All statistical significance tests start with a null hypothesis () that asserts that no real effect is present in the population, and any effect you think you see in your sample is due only to random fluctuations. (See Chapter 3 for more information.) The for the chi-square test asserts that there’s no association between the levels of the row variable and the levels of the column variable, so you should expect the relative spread of cell counts across the columns to be the same for each row.

Figure 12-1 shows how this works out for the observed data taken from the example in this chapter’s introduction. You can see from the marginal “Total” row that the overall rate of pain relief (for both groups combined) is 43/100, or 43 percent.

Figure 12-1 presents the actual data you observed from your survey, where the observed counts are placed in each of the four cells. As part of the chi-square test statistic calculation, you now need to calculate an expected count for each cell. This is done by taking the product of the row and column marginals and dividing them by the total. So, to determine the expected count in the CBD/pain relief cell, you would multiply 43 (row marginal) by 60 (column marginal), then divide this by 100 (total) which comes out 25.8. Figure 12-2 presents the fourfold table with the expected counts in the cells.

The reason you need these expected counts is that they represent what would happen under the null hypothesis (meaning if the null hypothesis were true). If the null hypothesis were true:

• In the CBD-treated group, you’d expect about 25.8 participants to experience pain relief (43 percent of 60), with the remaining 34.2 reporting no pain relief.
• In the NSAIDs-treated group, you’d expect about 17.2 participants to feel pain relief (43 percent of 40) with the remaining 22.8 reporting no pain relief.

As you can see, this expected table assumes that you still have the overall pain relief rate of 43 percent, but that you also have the pain relief rates in each group equal to 43 percent. This is what would happen under the null hypothesis.

Now that you have observed and expected counts, you’re no doubt curious as to how each cell in the observed table differs from its companion cell in the expected table. To get these numbers, you can subtract each expected count from the observed count in each cell to get a difference table (observed – expected), as shown in Figure 12-3.

As you review Figure 12-3, because you know the observed and expected tables in Figures 12-1 and 12-2 always have the same marginal totals by design, you should not be surprised to observe that the marginal totals in the difference table are all equal to zero. All four cells in the center of this difference table have the same absolute value (7.2), with a plus and a minus value in each row and each column.

The pattern just described is always the case for tables. For larger tables, the difference numbers aren’t all the same, but they always sum up to zero for each row and each column.

The values in the difference table in Figure 12-3 show how far off from your observed data are. The question remains: Are those difference values larger than what may have arisen from random fluctuations alone if is really true? You need some kind of measurement unit by which to judge how unlikely those difference values are. Recall from Chapter 10 that the standard error (SE) expresses the general magnitude of random sampling, so looking at the SE as a type of measurement unit is a good way for judging the size of the differences you may expect to see from random fluctuations alone. It turns out that it is easy to approximate the SE of the differences because this is approximately equal to the square root of the expected counts. The rigorous proof behind this is too complicated for most mathophobes (as well as some normal people) to understand. Nevertheless, a simple informal explanation is based on the idea that random event occurrences typically follow the Poisson distribution for which the SE of the event count equals the square root of the expected count (as discussed in Chapter 10).

Summarizing and combining scaled differences

For the upper-left cell in the cross-tab (CBD–treated participants who experience pain relief), you see the following:

• The observed count (Ob) is 33.
• The expected count (Ex) is 25.8.
• The difference (Diff) is , or .
• The SE of the difference is or

You can “scale” the Ob-Ex difference (in terms of unit of SE) by dividing it by the SE measurement unit, getting the ratio , or 1.42. This means that the difference between the observed number of CBD-treated participants who experience pain relief and the number you would have expected if the CBD had no effect on survival is about 1.42 times as large as you would have expected from random sampling fluctuations alone. You can do the same calculation for the other three cells and summarize these scaled differences. Figure 12-4 shows the differences between observed and expected cell counts, scaled according to the estimated standard errors of the differences.

The next step is to combine these individual scaled differences into an overall measure of the difference between what you observed and what you would have expected if the CBD or NSAID use really did not impact pain relief differentially. You can’t just add them up because the negative and positive differences would cancel each other out. You want all differences (positive and negative) to contribute to the overall measure of how far your observations are from what you expected under .

Instead of summing the differences, statisticians prefer to sum the squares of differences, because the squares are always positive. This is exactly what’s done in the chi-square test. Figure 12-5 shows the squared scaled differences, which are calculated from the observed and expected counts in Figures 12-1 and 12-2 using the formula (rather than by squaring the rounded-off numbers in Figure 12-4, which would be less accurate).

You then add up these squared scaled differences: to get the chi-square test statistic. This sum is an excellent test statistic to measure the overall departure of your data from the null hypothesis:

• If the null hypothesis is true (use of CBD or NSAID does not impact pain relief status), this statistic should be quite small.
• If one of the levels of treatment has a disproportionate association with the outcome (in either direction), it will affect the whole table, and the result will be a larger test statistic.

Determining the p value

Now that you calculated the test statistic, the only remaining task before interpretation is to determine the p value. The p value represents the probability that random fluctuations alone, in the absence of any true effect of CBD or NSAIDs on pain relief, could lead to a value of 8.81 or greater for this test statistic. (We introduce p values in Chapter 3.) Once again, the rigorous proof is very complicated, so we present an informal explanation:

When the expected cell counts are very large, the Poisson distribution becomes very close to a normal distribution (see Chapter 24 for more on the Poisson distribution). If the is true, each scaled difference should be an approximately normally distributed random variable with a mean of zero and a standard deviation of 1. The mean is zero because you subtract the expected value from the observed value, and the standard deviation is 1 because it is divided by the SE. The sum of the squares of one or more normally distributed random numbers is a number that follows the chi-square distribution (also covered in Chapter 24). So the test statistic from this test should follow the chi-square distribution. Now it is obvious why it is named the chi-square test! The next step is to obtain the p value for the test statistic. To do that manually, you would look up the test statistic (which is 8.81 in our case) in a chi-square table.

In actuality, the chi-square distribution refers to a family of distributions. Which chi-square distribution you are using depends upon a number called the degrees of freedom, abbreviated d.f. or df or by the Greek lowercase letter nu (v) (in this book we use df). The df is a measure of the probability of independence between the value of the predictor (row) variable and value of the column (outcome) variable.

How would you calculate the df for a chi-square test? The answer is it depends on the number of rows in the cross-tab. For the cross-tab (fourfold table) in this example, you added up the four values in Figure 12-5, so you may think that you should look up the 8.81 chi-square value with 4 df. But you’d be wrong. Note the italicized word independence in the preceding paragraph. And keep in mind that the differences () in any row or column always add up to zero. The four terms making up the 8.81 total aren’t independent of each other. It turns out that the chi-square test statistic for a fourfold table has only 1 df, not 4. In general, an N-by-M table, with N rows, M columns, and therefore cells, has only df because of the constraints on the row and column sums. In our case, N — which is the number of rows — is 2, so N-1 is 1. Also, M — which is the number of columns — is 2, so M-1 is 1 also (and 1 times 1 is 1). Don’t feel bad if this wrinkle caught you by surprise — even Karl Pearson who invented the chi-square test got that part wrong!

So, if you were to manually look up the chi-square test statistic of 8.81 in a chi-square table, you would have to look under the distribution for 1 df to find out the p value. Alternatively, if you got this far and you wanted to use the statistical software R to look up the p value, you would use the following code: pchisq(8.81, 1, lower.tail = FALSE). Either way, the p value for chi-square = 8.81, with 1 df, is 0.003. This means that there’s only a 0.003 probability that random fluctuations could produce the effect seen, where CBD performs so differently than NSAIDs with respect to pain relief in chronic arthritis patients. A 0.003 probability is the same as 1 chance in 333 (because ), meaning very unlikely, but not impossible. So, if you set α = 0.05, because 0.003 < 0.05, your conclusion would be that in the chronic arthritis patients in our sample, whether the participant took CBD or NSAIDs was statistically significantly associated with whether or not they felt pain relief.

Putting it all together with some notation and formulas

The calculations of the Pearson chi-square test can be summarized concisely using the cell-naming conventions shown in Figure 12-6, along with the standard summation notation described in Chapter 2.

Using these conventions, the basic formulas for the Pearson chi-square test are as follows:

• Expected values:
• Chi-square statistic:
• Degrees of freedom:

where i and j are array indices that indicate the row and column, respectively, of each cell.