Probability is an illusion

"Random samples according to the scientific definition are an illusion"

Unfortunately, Prof. Schnell's position does not lead any further in the practical work of market and social research. Because there is indeed a lot to be said for considering the random sample as the gold standard. But Prof. Schnell will also have to admit that this gold standard is practically no longer achievable in market and social research.

Prof. Schnell correctly defines that, in the case of random samples, one must be able to calculate the selection probability for each element of the population before the sample is drawn. Perhaps we still have to clarify what is meant by “probability of selection”. That's not the likelihood that I will choose a person and ask for an interview. With a systematic approach, this probability can indeed be calculated very well. Rather, it is the probability that the data of a person from the population will be included in the pool of collected data. And this probability can no longer be calculated at all. Because this not only requires that a person be selected, but also that they are ready to be interviewed. And unfortunately that is quite often not the case if you are not called the Federal Statistical Office and have been authorized by the legislature to force the answers by imposing fines if necessary.

If a person refuses the interview, then the previously calculated probability turns out to be wrong. But because the sample size is to be reached despite the refusal, other people are interviewed. The probability of the other people willing to provide information to be included in the sample increases. Since the response rate is well below 50 percent even in very high-quality studies such as the Allbus, it can be stated that the sample only consists of people whose previously calculated selection probabilities are all wrong. Random samples according to the scientific definition are therefore an illusion.

In addition: Since nobody really knows why a person is ready for the interview or why not, these failures are not random, but rather "missing not at random" and thus - as Prof. Schnell also correctly writes - lead to almost uncorrectable errors . Under these circumstances, is it really still possible to declare the random sample to be the gold standard?

His assessment of online samples is also only comprehensible in theory, not in practice. According to "Best 4 Planning 2018", 84 percent of the population aged 14 and over have been internet users in the last three months. If you limit yourself to the target group 14 to 69 years, then it is 94 percent. It is true that people who are not online are different from those who use the Internet. And, yes, the 16 and 6 percent are “missing not at random”. Nevertheless, the proportion is so small, at least in the age group 14 to 69, that online samples cannot be blamed for general problems with the population. Telephone samples have similar problems.

Against this background, the “fall from grace” of the quota samples is to be seen differently than Prof. Schnell does. Quota samples are in fact little treated in science, if only because a theoretical investigation is difficult or impossible and universities have no access to it. Good quota samples, however, are by no means arbitrary, but finely controlled handicrafts, whereby the number and the distribution of characteristics of the interviewers as well as the quota plan are important. In addition, good quota samples have a specific advantage: Because interviews with acquaintances or friends of acquaintances are refused less than with strangers, quota interviewers often come to the interview where a random selection leads to a refusal.

In addition: consumer and retail panel samples are quota samples because random samples are simply not possible. However, the panel numbers are continuously and systematically compared with the sales figures of the manufacturers. If the result were as bad as Prof. Schnell said it should be, panels would have been dead for a long time.

What we need is a return to what market and social research is for. Ultimately, the purpose of the event is to enable managers and politicians to make better decisions. The representativeness of the study is an important quality feature. Because, of course, a survey is not only useless, it is even harmful if its results are so wrong that they lead to wrong decisions. Another important quality feature can also be the time required for a study to deliver the results. Because a survey, the result of which is provided after the decision had to be made, is also useless. It can make sense to reduce the representativeness of the target group 14 to 69 years and to receive the results in good time.