Is it Really True that Only 8% of People Who Buy Herbalife (HLF) Products are Herbalife Distributors?

A reporter emailed me the other day with a question about a case I’d never heard of before, a company called Herbalife (HLF) that is being accused of being a pyramid scheme. The reporter pointed me to this document which describes a survey conducted by “a third party firm called Lieberman Research”:

Two independent studies took place using real time (aka “river”) sampling, in which respondents
were intercepted across a wide array of websites

Sample size of 2,000 adults 18+ matched to U.S. census on age, gender, income, region and ethnicity

“River sampling” in this case appears to mean, according to the reporter, that “people were invited into it through online ads.” The survey found that 5% of U.S. households had purchased Herbalife products during the past three months (with a “0.8% margin of error,” ha ha ha).

They they did a multiplication and a division to estimate that only 8% of households who bought these products were Herbalife distributors: 480,000 active distributorships / (0.05 * 114 million U.S. households) = 0.08.

The reporter asked what I thought about that 8%. Here are some of my reactions (again, recall that I had not heard about this case before, and these reactions are based on the information provided to me):

I think there are some serious problems here.

1. Generalizing from 2000 to the whole population is not a problem—as long as the sample is a representative sample, and as long as the survey responses are correct. I did not look at the details of the survey so I don’t know about the issue of representativeness. I wonder, though, exactly how they “fished” for respondents in their river sampling. It could be that they were more likely to get the sort of person who was interested in dietary supplements. Or maybe not, I don’t know.

2. However the sampling was done, I worry about the survey responses. It’s a well-known problem that rate events get overestimated. For example (from an article by David Hemenway from 1997): “The National Rifle reports 3 million dues-paying members, or about 1.5% of American adults. In national random telephone surveys, however, 4-10% of respondents claim that they are dues-paying NRA members. Similarly, although Sports Illustrated reports that fewer than 3% of American households purchase the magazine, in national surveys 15% of respondents claim that they are current subscribers.” The mathematics is that if there is a small rate of error, it can show up as a large error in the estimate of a small population. So, just because 5% of respondents say they used Herbalife products in the previous three months, that doesn’t mean that 5% of respondents actually used Herbalife products in the previous three months. I’d guess it’s a lot less.

3. I assume that “distributors” buy a lot more Herbalife products than non-distributors. Hence, even if 8% of the Herbalife consumers are distributors, I’d guess that a lot more than 8% of Herbalife products are bought by distributors. This doesn’t address Herbalife’s claims (they talk about the number of households, not the amount of products) but it seems relevant to the discussion.

P.S. Amusingly, one of Herbalife’s points is “Fact: Majority of Former Distributors Would Recommend Herbalife to Friends and Family.” But that’s exactly what you’d expect of a still-active pyramid scheme, no? Existing members want new people below them on the pyramid. I’m not saying this means it is a pyramid scheme, but it doesn’t seem like evidence against the hypothesis!

P.P.S. I tried to choose the most appropriate category for this one.

About Andrew Gelman 26 Articles

Affiliation: Columbia University

Andrew Gelman is a professor of statistics and political science and director of the Applied Statistics Center at Columbia University. He has received the Outstanding Statistical Application award from the American Statistical Association, the award for best article published in the American Political Science Review, and the Council of Presidents of Statistical Societies award for outstanding contributions by a person under the age of 40.

His books include Bayesian Data Analysis (with John Carlin, Hal Stern, and Don Rubin), Teaching Statistics: A Bag of Tricks (with Deb Nolan), Data Analysis Using Regression and Multilevel/Hierarchical Models (with Jennifer Hill), and, most recently, Red State, Blue State, Rich State, Poor State: Why Americans Vote the Way They Do (with David Park, Boris Shor, Joe Bafumi, and Jeronimo Cortina).

Andrew has done research on a wide range of topics, including: why it is rational to vote; why campaign polls are so variable when elections are so predictable; why redistricting is good for democracy; reversals of death sentences; police stops in New York City, the statistical challenges of estimating small effects; the probability that your vote will be decisive; seats and votes in Congress; social network structure; arsenic in Bangladesh; radon in your basement; toxicology; medical imaging; and methods in surveys, experimental design, statistical inference, computation, and graphics.

Visit: Andrew Gelman's Website

Be the first to comment

Leave a Reply

Your email address will not be published.