What the Quiet Revival Collapse Can Teach Us About Orthodox Data

The UK Bible Society is in the news for pulling a major survey due to bad data. OSI had a similar experience last year.

A year ago, the UK Bible Society released its “Quiet Revival” report, which was based on a YouGov survey of over 13,000 people from England and Wales. The results were stunning: a 56% increase in monthly church attendance compared to the Bible Society’s previous (2018) survey, and a fourfold increase among 18- to 24-year-olds (up from 4% to 16%).

Many Christians viewed this as terrific news: a revival was happening — albeit a “quiet” one, since it seems not to have shown up in church attendance metrics from the Catholic Church or the Church of England. One skeptical scholar told the BBC, if the Bible Society’s findings were legit, “we’d be looking for literally millions of new churchgoers, and they’d have to be very quiet indeed, not to say invisible, to have escaped our notice.”

The skeptics were vindicated a year later. On March 26, 2026, the Bible Society issued a formal retraction of the entire Quiet Revival report. In an announcement on its website, the Bible Society explained what went wrong:

YouGov has quality control systems designed to exclude survey respondents who are not based in the UK, who attempt to complete the survey more than once, or who give inattentive or random answers. Bible Society was informed in early March that significant parts of these systems were not functioning correctly during the 2024 survey – the result of human error on YouGov’s part. YouGov only discovered this recently, following a thorough internal review. This meant that a statistically significant proportion of responses were of low quality or otherwise unreliable, and the data cannot now be relied upon.

YouGov is one of the biggest survey companies in the business. People join YouGov’s panel and take surveys in exchange for points, which can be redeemed for money or prizes. There’s obvious risk here, when it comes to the quality of respondents. YouGov has various measures in place to mitigate this risk, such as IP address checks and VPN detection. They also look for “speed runners” — people who complete the survey much faster than should be possible — and people who give nonsensical answers. According to the Bible Society’s announcement, this quality control system failed, in the case of their YouGov survey.

The implication of this is that many of the YouGov respondents weren’t who they claimed to be (for example, many weren’t actually based in the UK). It tainted the entire dataset, hence the Bible Society’s painful retraction.

---

This story felt uncomfortably familiar to us at OSI, in a “there-but-for-the-grace-of-God-go-I” kind of way.

Last year, OSI participated in a larger survey effort by an established research team, using the platform of a market research firm. We had to raise $20,000 for this, but it seemed like too good an opportunity to pass up: the chance to design our own questions specifically for randomly selected Orthodox Christian adults in America. The scale of the broader survey — roughly 100,000 American adults — was such that we expected to get 500 or 600 respondents who were currently Orthodox, plus several hundred people who had previously been Orthodox but had left the Church. We’d be able — so we thought — to learn a ton of valuable information: not just the size of the American Orthodox population, but detailed demographic data, unprecedented insights into the retention of both cradle-born Orthodox and converts, and the opinions of hundreds of Orthodox people on a whole range of topics.

When the responses started rolling in, we were stunned. Based on hundreds of other general-population surveys, we expected the Orthodox share of this study to be roughly 0.5-0.7%. But the raw numbers, we were told, were totally different: 1.6% of the initial sample was currently Orthodox, and another ~0.75% were formerly Orthodox. If this were accurate, it would imply an overall US Orthodox population of 5.4 million. While I wish this were the case, I knew it wasn’t; it would mean that the average parish has something like 2,700 people.

Then, the people who conducted the survey did their own data cleaning, similar, as I understand it, to what YouGov does. They removed speed runners and people who gave logically impossible responses. In the end, I’m told that 2,853 respondents were rejected. And then, we received another shock: roughly 15% of the rejected respondents were current Orthodox, and another 2% were former Orthodox. In other words, current Orthodox respondents were roughly nine times more prevalent in the rejected respondents than in the original sample (and former Orthodox were something like two and a half times more prevalent).

In the end, we received a data file with 1,174 current and 695 former Orthodox Christians. This would imply a current US Orthodox population of 4 million, which is roughly double what I think the real number is (even with our legitimately significant growth in recent years). And most of these weren’t Oriental Orthodox: we had respondents select which type of Orthodox they were, and only 210 of them picked an Oriental Orthodox group. The implication being that — if these survey results were to be believed — Eastern Orthodox share of the US population was 0.964%, or close to 3.3 million people. More than a million more than expected. I was cautiously optimistic that maybe I had just been too conservative in my estimate, and that maybe there really were three million Eastern Orthodox in America.

---

My hopes were quickly dashed when I started digging into the data. Consider:

We asked people age-based questions, such as their current age and the age when they became Orthodox. For those who were currently Orthodox, we asked the age when they embraced the Orthodox faith as their own (which included the option “I haven’t”). For those who were formerly Orthodox, we asked them both when they emotionally disconnected from Orthodoxy, and when they stopped self-identifying as Orthodox. We got dozens and dozens of nonsensical responses to these questions. Dozens upon dozens of people gave impossible answers, saying that these various events occurred at ages older than their current age (many of them much older, such as a 43-year-old who claimed to have become Orthodox at 100).

In addition to those impossible ages, here are some of the other red flags we found:

• Impossible characteristics (e.g., a woman claiming to be major clergy or to serve in the altar)

• Becoming Orthodox ages 0-6 but without either parent being Orthodox when the person was growing up (not impossible, but extremely improbable and very common in the dataset)

• Suspicious ages, such as embracing Orthodoxy at (say) age 8 but becoming Orthodox at age 30

• Middle-aged and elderly non-immigrants who claim to have attended an Orthodox school growing up but were not Orthodox until adulthood (Orthodox schools aren’t very common today, and prior to the past couple decades, any school other than a parochial school was basically nonexistent)

We asked respondents which kind of Orthodox they were — basically their jurisdiction, although of course it’s likely that, say, an ethnic Greek who attends an OCA parish might choose “Greek Orthodox.” In our list of options, we included both Eastern Orthodox and Oriental Orthodox / non-Chalcedonian identifiers, since, we reasoned, members of both groups would likely select “Orthodox Christian” and end up in our data set.

The results were unbelievable. As in, literally, we did not believe them.

For starters, just 2% of current Orthodox respondents were Antiochian, and another 2% were Serbian Orthodox — dramatically lower percentages than we expected. And then, 2% were Albanian and 2% Macedonian, with both jurisdictions having fewer than 20 parishes in the United States. (Meanwhile, the Antiochians have nearly 300, and the Serbs have more than 120.) The idea that there are as many Albanians and Macedonians as there are Serbs and Antiochians is not realistic.

Even more bewilderingly, more than a quarter of the people who identified as Macedonian Orthodox were black or Hispanic — this, in a jurisdiction that is extremely ethnic, with, in real life, almost exclusively Slavic membership, from the Republic of North Macedonia. (It might be a coincidence, but I did learn that there are a dozen or so “Macedonia African Methodist Episcopal” churches in the United States.)

These “racial/ethnic mismatches” were common in the data set. A surprisingly high number of people (10.7%) identified as Armenian Orthodox, and of these, 23% were black or Hispanic. The Armenian Orthodox Church is essentially a refuge for ethnic Armenians, with worship almost exclusively in Armenian. I’d be surprised if even 2% of actual Armenian Orthodox in America are black or Hispanic; 23% is far outside the realm of possibility.

There were even more blacks and Hispanics among the Romanian Orthodox — 34.5%. I do know of some convert-friendly Romanian parishes, but still, the idea that more than one-third of Romanian Orthodox in America are black or Hispanic is laughable.

And then there was the problem of artificial intelligence. We asked three free-response questions, with text boxes provided for the respondent to fill out:

• If you could speak to all of the Orthodox bishops, what would you tell them?

• What is the single biggest problem in the Orthodox Church today?

• What are your current struggles with faith?

A very large number of the responses were obviously AI-generated. Many used identical or nearly identical wording, and clearly were the result of our question being used to prompt an AI to generate a response. In several cases, the submitted response literally identified the respondent as an AI (e.g., “I cannot struggle with faith in the way a human might" or “As an AI, I don’t experience faith or belief the way humans do"). Altogether, we found evidence of AI use in 16.5% of the free-response answers.

---

Our team went through the data file and removed the most obviously false responses — the impossible or extremely improbable ages, the ethnic/racial mismatches, the people who gave AI-generated free-response answers. When it was all said and done, we removed 574 of the 1,174 current Orthodox (48.9%) and 422 of the 695 former Orthodox (70.8%). Combined, we removed 53.3% of the respondents we were sent — and remember, that’s after a bunch of bad respondents were already removed in the data cleaning process prior to our receipt of the data.

It’s a little bit mind-boggling to get what is supposed to be a pre-cleaned data set and still find that over half of it is unusable. So what on earth happened?

I reached out to the market research firm that conducted the survey to try to get some answers. One theory was that perhaps they had somehow indicated to respondents that they’d get extra compensation for selecting Orthodox and thus getting our Orthodox-specific questions. But the firm assured me that this was not the case: they had not disclosed to participants that there was any such incentive. Nothing in their communications with respondents even mentioned Orthodoxy.

When I asked how our data was so unusually bad — how we made up such a high percentage of the pre-cleaned bad respondents — they said that the only explanation was that something about the survey itself pushed people to identify as Orthodox. Perhaps the Orthodox option had been listed at the top or the bottom of the religious affiliation options? But when we re-checked the exact text of the survey questions themselves, this was not the case: Orthodox Christian was actually the fifth in a long list of options.

At this point, we really have no idea why we ended up with so many unusable responses.

---

Once we removed all those obviously junk respondents, the current Orthodox share of the all of the survey respondents is just under 0.6%, which is right in line with the kind of numbers that big surveys like the Cooperative Election Study and the Pew Religious Landscape Survey found.

Our extra-cleaned sample left us with a total of 878 respondents, including both current and former Orthodox. And this cleaned sample does appear to offer some real signal.

Orthodox Identification in Adulthood and Childhood Parent Religion

For example, we found that respondents with only one Orthodox parent had a 61% retention rate, with virtually identical results regardless of whether the lone Orthodox parent was the father or the mother. But when the respondent had two Orthodox parents, their likelihood of remaining Orthodox as adults jumped to 85%. This tracks with both common sense and other data we’ve analyzed. It also suggests that our data cleaning was broadly effective at separating the wheat from the chaff.

---

I hope to be able to publish some other findings from this extra-cleaned sample in the future. But it’s still deeply concerning that we had to engage in so much line-by-line cleaning, relying on our own very domain-specific expertise. To be perfectly clear: we had to remove over half of the responses we received, and that was after the experts who provided this data to us had already carried out their own cleaning. Some of the weird responses were probably obvious to anyone looking at them — the AI free-response answers, the impossible ages — but other problems would only be apparent to someone familiar with Orthodoxy, such as the suspiciously high number of black and Hispanic respondents among those identifying as Albanian, Armenian, Macedonian, and Romanian.

This experience was unsettling enough to cause us to rethink both our use of this specific data set, and, more broadly, our ability to rely on data collected via large-scale, opt-in online surveys where respondents are compensated for their participation. This methodology seems to come with heightened risk as we see advances in AI and other technologies that open the door to greater fraud and otherwise false or misleading data.

Here is how Professor David Voas, perhaps the most vocal critic of the Quiet Revival study, explains the problem:

Gold standard social surveys are based on random (probability) samples of the population: everyone has a chance to be included. […]

By contrast, people opt in to YouGov’s survey panel and are rewarded after completing a certain number of surveys. The risk of low-quality or even bogus responses is considerable.

YouGov creates a quota sample from its large self-selected panel. The sample will match the population on a number of key characteristics, such as age and sex, but that does not make it representative in all respects. As quota samples do not give each person in the population a known chance of being selected, statistical inference is not possible and findings cannot be reliably generalised.

It’s notable that the Cooperative Election Study, which we’ve cited many times ourselves here at OSI, relies on YouGov panelists — meaning, it’s not probability-based. That doesn’t invalidate the study, but it does mean the data still comes from an opt-in, non-probability panel that requires careful matching, weighting, and validation to approximate representativeness.

In contrast, the Pew Religious Landscape Study is probability based, as it is sent to a random sample of US households.

Our problematic data set came from a non-probability survey. As I understand it, the biggest differences between YouGov and the market research firm that conducted the survey we participated in is that YouGov maintains its own opt-in panel, whereas this other firm instead works with external online panel providers to find respondents. But in both cases, the methodology is not probability-based, and thus is more at risk of being tainted by bad data — especially for rare subgroups like Orthodox Christians.

---

Our experience, and the high-profile problems of the Quiet Revival study, are not the only examples of recent, tainted religious survey data collected via non-probability methodologies. Another notable example is that of the Fuller Youth Institute, which, like us at OSI, was able to catch the data integrity problems before they published anything. Fuller was conducting a survey of Gen Alpha youth on religion, but midway through the process, they realized that their data was tainted due to AI and VPN use. They ended up scrapping an entire survey’s data set and starting the whole study from square one. The results of that second, more secure study were published in January 2026. Here is how Fuller described it in the report:

In the summer of 2025, we constructed, piloted, and conducted the National Survey on Teen Spirituality & Religion, designed by the Fuller Youth Institute in consultation with Future of Faith. It was administered in July 2025 on Qualtrics via both a purchased Centiment panel and a nationally-recruited sample through FYI and Alpha Youth USA networks. Following thorough data validation, cleaning, and analysis, we ultimately chose to discontinue our nationally-recruited panel due to contamination concerns and solicited another Centiment panel in November 2025. These efforts combined yielded a total usable sample of 2,783 participants.

I spoke with the lead researchers for the Fuller Youth Institute as I was drafting this article, and they offered some details on their experience. AI use was rampant in their original survey. They told me that one of their consultants found an AI script designed to teach an AI how to take surveys in a way that mimics human behavior, such as varying responses and response time. But their experience with VPNs was particularly unsettling: they described the bizarre experience of follow-up video calls with participants whose identity in real life didn't match their identity in the survey; for example, a participant listed as a US-based teen girl turned out to be an African-based male. Interviewers had to terminate a number of interviews due to this sort of mismatch.

In the end, there were just far too many problems for Fuller to trust the data they had collected, so they made the difficult (but high-integrity) decision to cut their losses and start over. The UK Bible Society didn’t discover the problems with their data until it was too late, but they also opted for self-effacing transparency over trying to save face.

---

When the UK Bible Society retracted their Quiet Revival report last month, I felt both pained for them, grateful for their transparency, and relieved that we ourselves dodged a similar bullet. Had we not been so suspicious and circumspect about the surprising results we received from our own survey last year, it easily could have been us at OSI retracting a study with big, exciting findings. And going forward, researchers probably need to be extra cautious about this sort of data.