The Vital Impact Assessment church survey cannot provide a perfect or fully objective evaluation of the church’s health and impact. However, it can provide a sufficient and trustworthy evaluation of the church’s health and impact. When assessing validity and reliability, it is important to remember the survey’s primary purpose: to measure the church’s health and impact so that the church can better fulfill its calling.
One of the most important issues to consider when interpreting the survey results is the relationship between the sample and population. “Does the survey sample accurately reflect the church?” “To what degree do these responses reflect the broader church?” Fortunately, we can put concrete numbers to the question of ideal sample size and suggest some best practices for ensuring an unbiased sample.
Ideal response rates
We recommend a target sample size of 70% of average adult weekend church attendance for the survey. In general, the smaller the church, the larger the percentage you need taking the church survey to accurately reflect the population. If you would like to identify a more specific target sample size, you can use a sample size calculator online. The numbers in the chart below are based on a 95% confidence level.
Sample size calculator:
http://www.openepi.com/SampleSize/SSPropor.htm
Select: ‘Enter new data’
Population size: Average adult weekend attendance
Anticipated frequency: ‘50’
Confidence limits: ‘5’
Design effect: ‘1’
Church size (avg. adult att.) Necessary response level Percentage of population
100 80 80%
200 132 66%
300 169 56%
400 197 49%
500 218 44%
1000 278 28%
2000 323 16%
Does the sample reflect the church?
One key question related to the confidence level is how well the sample reflects the church.
Providing the sample size is sufficient for the church size at a 95% confidence level (see the chart above), we can address the question of bias. Anonymity, the ability to skip any or all questions and survey analysis methods help to mitigate bias in the results. How you conduct the survey also significantly impacts the potential level of bias in the results. It’s important to carefully consider how to obtain a broad representation of individuals across all levels of church participation.
Four ways to conduct the church survey
Option 1:
Broadly distribute the church survey through email and social media. Send out reminders with links for a few consecutive days.
Option 2:
Personally invite all small groups, Sunday school classes, ministry teams, and other groups to take 10 minutes during their meeting that week to fill out the church survey. Have paper copies available for those who are unable to take complete the survey on their phone.
Option 3:
Enlist volunteers to randomly invite attendees of weekend services to complete the survey on a tablet or phone. Ensure that the survey has been properly introduced and volunteers are clearly identified before using this option. Volunteers will need to protect respondents’ anonymity with this option as well.
Option 4:
Conduct the survey at the end of a worship gathering and invite everyone to participate, regardless of background or level of involvement. The survey should take between 8-12 minutes. Place the survey link where it can be visible to everyone in the sanctuary. You can also email it to everyone immediately beforehand, so they can directly follow the link. Ushers can distribute paper copies to anyone who needs one.
The positive side of this approach is that it provides results from a cross-section of the church. The drawback is that it has the potential of compromising anonymity since people are likely sitting close to others filling out the survey. For example, a rating of marriage quality can be biased when a spouse can potentially view the response. This drawback can be mitigated somewhat by reinforcing the importance of anonymity and encouraging honesty before conducting the survey. Talking points could include: “Please, no peaking at the person next to you.” “The only wrong answer is the inaccurate one. This survey examines present experience and perspective, not ideals or hopes.” These options can be joined and blended based on the culture and need of the church. Often, churches will launch the survey with a combination of options 1-3 during week one and use option 4 the following weekend if response rates are low or leaders sense a need for a more random cross-section in the sample.