Holes in Medical Statistics Part 1: Understanding the P value through Poker

Is there a misunderstanding about the use and interpretation of P values and hypothesis testing? An excellent set of articles written by SN Goodman makes this case. This is the first in a series of blog entries to help illustrate the points in these papers.

To help clarify the concept of the P value, I will start with a contrived poker example:

Let's say you're in Las Vegas hoping for a little enjoyment and to win some cash. You enter one of the seedier downtown casinos and find a new game. It's called "Huslyall". The dealer is rather unkempt, wears an eyepatch and has a hook replacing his left hand.

"Here's the game", he says. "Poker. 52 card deck. The bet is $100. I shuffle and lay down 5 cards. If I get a straight flush, I win and take your $100. Anything less than that, you win and I give you $100."

You are suspicious, but interested. "This will be easy money if this is a normal deck," you think. "On the other hand if the deck is fixed, I'm out $100." Against your best instincts you decide to play the game (after all it seems less risky than the stock market.)

The dealer shuffles the deck (which is amazing given the hook) and lays down 5 cards. He then arranges them in order: 6,7,8,9,10 - all hearts.

Question: What is the probability of drawing a straight flush from an ordinary deck of cards? Answer: prob = 0.0000139.

Question: What is the P value for this particular occurrence?

Answer: First the definition of the P value -
"The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed."

An important point about the P value is that one assumes that the null hypothesis is correct. The P value is then calculated by the probability of the observed effect or a more extreme effect.

Getting back to our poker example, we assume first that the dealer's deck is a normal deck. We then calculate the probability of obtaining this ordinary straight flush (prob = 0.0000139) + the more extreme effect which would be a royal straight flush (prob = 0.00000154). The P value in this case is:
P = 0.0000139 + 0.00000154 = 0.00001544.

So this means that we reject the null hypothesis, right? Wrong.

Key Point: As stated previously in order to calculate the P value we actually accept the null hypothesis. We then calculate the odds of the observed effect or a more extreme effect. You cannot reject the null hypothesis that you just accepted.

So when do we get to reject the null hypothesis? By hypothesis testing which, as we will see, is a completely separate statistical idea. It has however become entangled together with the P value. This has brought us to our current state of confusion. More on this to come.

Should you knock the dealer in the jaw and take back your money? That's up to you...but that hook looks awfully sharp. Better consider discussing your case with casino management first.

TNT - Any benefit to atorvostatin 80 mg versus 10 mg in patient's with CHD?

From TNT study:
N Engl J Med. 2005 Apr 7;352(14):1425-35

Research Design and Methods: A total of 10,001 patients with clinically evident CHD and LDL cholesterol levels of less than 130 mg per deciliter (3.4 mmol per liter) were randomly assigned to double-blind therapy and received either 10 mg or 80 mg of atorvastatin per day. Patients were followed for a median of 4.9 years. The primary end point was the occurrence of a first major cardiovascular event, defined as death from CHD, nonfatal non-procedure-related myocardial infarction, resuscitation after cardiac arrest, or fatal or nonfatal stroke.





Primary efficacy outcome measure: Time to first occurrence of a major cardiovascular event.

Defined as: CHD death, nonfatal non–procedure-related myocardial infarction, resuscitated cardiac arrest, and fatal or nonfatal stroke.

Article's Conclusions: Intensive lipid-lowering therapy with 80 mg of atorvastatin per day in patients with stable CHD provides significant clinical benefit beyond that afforded by treatment with 10 mg of atorvastatin per day. This occurred with a greater incidence of elevated aminotransferase levels.

Analysis
Figure 2 appears to show a significant reduction in mean levels of LDL, Total cholesterol, and Triglycerides. In fact only total cholesterol (P<0.001) and triglycerides(P<0.001) were significantly reduced. There is no P value given for LDL. In the absence of this, the assumption is that the reduction was not significant. There was no significant change in mean HDL values.

There was a 22% relative reduction in the primary composite measure. Looking at the individual components we have:
CHD Death: HR 0.80[0.61-1.03] P=0.09
Non-fatal non-procedure-related myocardial infarction: HR 0.78[0.66-0.93] P=0.004
Resuscitated cardiac arrest: HR 0.96[0.56-1.67] P=0.89
Fatal/Non-fatal stroke: HR 0.75[0.59-0.96] P=0.02

There is a significant reduction with regards to Fatal/non-fatal stroke and non-fatal non-procedure-related MI. There is no significant change in CHD death. In fact there is no signifcant change in Death from any cause: HR 1.01 (0.85-1.19) P=0.92.

What about adverse events? 80 mg (8.1%) vs 10 mg (5.8%) P<0.001. The bulk of this appears to be due to a persistent increase in AST/ALT. This translates into a roughly 40% increase in adverse events.

What are we left with? A significant reduction in fatal/non-fatal stroke and non-fatal MI. However there is no improvement in all-cause mortality or CHD mortality specifically (This is in contrast to a study such as the Jupiter Study in which there appears to be a definite improvement in mortality with 20mg of statin versus placebo in paient's with a high CRP.) Furthermore, there is a 40% increase in adverse events.

I would say therefore that the data is weak in support using atorvostatin 80 mg a day for all patients with CHD. It may be useful for some at moderate or high risk of stroke. A review of this study as well as its substudy on diabetes points to the importance of examining specific outcomes (i.e. MI, stroke, death, PAD, etc.) versus composite outcomes (i.e. major cardiovascular events, cerebrovascular events, etc.), because when the outcomes are grouped together the true benefit can be misleading.

TNT and Diabetes - Any real benefit with Atorvostatin 80 vs 10 mg?

TNT study
Diabetes Care 29:1220-1226, 2006

RESEARCH DESIGN AND METHODS: A total of 1,501 patients with diabetes and CHD, with LDL cholesterol levels of <130 mg/dl, were randomized to double-blind therapy with either atorvastatin 10 (n = 753) or 80 (n = 748) mg/day. Patients were followed for a median of 4.9 years. The primary end point was the time to first major cardiovascular event, defined as death from CHD, nonfatal non-procedure-related myocardial infarction, resuscitated cardiac arrest, or fatal or nonfatal stroke.



Figure 1—Kaplan-Meier estimates of the incidence of major cardiovascular events in patients with diabetes. *Composite of CHD death, nonfatal non– procedure-related myocardial infarction, resuscitated cardiac arrest, and fatal or nonfatal stroke.

Figure 2

Primary efficacy outcome measure: Time to first occurrence of a major cardiovascular event.

Defined as: CHD death, nonfatal non–procedure-related myocardial infarction, resuscitated cardiac arrest, and fatal or nonfatal stroke.

Article's conclusions: Among patients with clinically evident CHD and diabetes, intensive therapy with atorvastatin 80 mg significantly reduced the rate of major cardiovascular events by 25% compared with atorvastatin 10 mg.

Analysis:

Another case of how you slice it? The article claims that there is a reduction of major cardiovascular events by 25% with atorvostatin 80 compared to 10 mg among patients with clinically evident CHD and diabetes. But look at each individual measure that defines "major cardiovascular event":

CHD Death: HR 0.74[0.47-1.18] P=0.203
Non-fatal non-procedure-related myocardial infarction: HR 0.79[0.55-1.14] P=0.202
Resuscitated cardiac arrest: no data provided
Fatal/Non-fatal stroke: HR 0.67[0.43-1.04] P=0.075

Actually none of these measures that have data provided reach significance. I would assume that resuscitated cardiac arrest must, though there is no data provided.

There is no difference in all cause mortality as seen in figure 2. There was a reduction in cardiovascular mortality with 80 mg(5.2%) versus 10 mg (6.5%), but an increase in noncardiovascular mortality in 80 mg(5.6%) versus 10 mg(3.3%). "However, the study was not powered to detect a significant difference between the treatment groups for mortality."

Not powered to detect a significant difference for mortality? Isn't that the ultimate point?

Given the above, I see little if any benefit in automatically prescribing atorvastatin 80 mg versus 10 mg for diabetics as the industry is implying we should do. One wonders as well if there is any benefit to intensive lowering of LDL cholesterol in diabetics.