Why do a poll?
Polling is one way, among many, to actively listen to the people we serve – in this case a representative sample of all Coloradans. By having 20-minute conversations with a sample of people who represent the state, we can better understand the priorities and attitudes of a wide range of Coloradans. The Colorado Health Foundation’s annual poll aims to take the pulse of Coloradans on a range of important health issues each year to inform policy far into the future.
Through Pulse: The Colorado Health Foundation Poll, we interview more than 2,200 Colorado adults, a large sample that allows us to dig into differences in perspectives based on race, ethnicity, income, geography, and a host of other factors. In fact, one of the upsides of this kind of quantitative research is that it allows us to discern patterns in what different groups of Coloradans are thinking or experiencing.
Through the poll, we learn a lot about how people are feeling and what priorities they have – and we share those with policymakers, community leaders, the media and anyone else who is interested. These findings also inform our work to advance health equity.
Polling has its limitations, too – namely that we only have about 20 minutes with each Coloradan we interview. That’s why there are other research methods we use to understand even more about Coloradans’ views and what drives those. For example, we’ve used qualitative research like in-person or online focus groups to dive deeper into topics with smaller groups of people so we can really understand the mindsets of Coloradans.
How were people chosen to participate in the poll?
Pulse: The Colorado Health Foundation Poll uses an address-based sampling methodology. That means that participants were randomly selected from a list of Coloradans with addresses.
Addresses were matched to phone numbers and email addresses, and participants were
contacted by phone call, text message, email and mail. In 2020’s poll, we were able to interview 2,275 Coloradans, which gives us the ability to analyze and report data on many more groups of Coloradans than would be possible with a smaller sample size. We use data from the U.S. Census Bureau to ensure that the sample of people we interviewed is representative of who calls the state home.
Of course, 2,275 people is just a sliver of the statewide population of more than 5.75 million. However, because of this rigorous sampling methodology, we are confident the poll accurately reflects the views of Coloradans as a whole and of many subgroups within our overall population (e.g. Coloradans living on low incomes).
We specifically chose to conduct this research with adults – not just voters – living in Colorado. This decision follows our commitment to listen to all Coloradans, not just those who are registered or likely to vote. It also distinguishes our approach from much of the other polling that is conducted in our state on behalf of candidates or issue campaigns. Our purpose is not necessarily to predict the outcome of elections, but rather to listen to the people of our state and accurately reflect what Coloradans are thinking and experiencing.
What about people experiencing homelessness who don’t have addresses or people who are unstably housed and move frequently?
People experiencing homelessness and housing insecurity are among some of the most challenging folks to include in quantitative research projects. With an address-based sample, it’s likely we didn’t reach many people who have been chronically homeless. This is a limitation of our methodology, and it’s one that challenges us to use other methodologies to listen to people experiencing homelessness in order to understand their opinions and experiences. Even with the limitations of this methodology, Pulse found that a significant number of Coloradans are struggling to afford housing – with 22% of Coloradans worried they might lose their home in the next year because they can’t afford the rent or mortgage. That number jumps to 52% for Coloradans living on low incomes – a reminder that concerns about housing affordability and stability are not limited to those currently experiencing homelessness.
What are oversamples, and why do you use them?
Through Pulse: The Colorado Health Foundation Poll, we seek to understand all Coloradans, even population groups that make up only a small percentage of residents of our state overall. That’s why, through our sampling, we sought to increase the number of Black/African American and Asian American and Pacific Islander (AAPI) adults interviewed than we would achieve if we only selected respondents by chance.
Here’s how it works: Black people make up around 4% of Colorado’s population; in a purely representative sample, that means they’d make up roughly 4% of all respondents. Given our commitment to improving the health of communities of color, we interviewed more Black Coloradans than just 4% of our sample so that we’d better understand their experiences, priorities and opinions. We worked to do the same with Asian American and Pacific Islander adults. In all, our oversamples allowed us to interview 130 Black/African American and 76 AAPI people for the 2020 poll.
It can be confusing to understand how oversamples like this don’t skew a poll’s overall results away from being representative of Colorado’s population as a whole. But they don’t – and that’s because of an additional step to the analysis process called “weighting.” According to the American Association for Public Opinion Research, “Weighting adjusts the poll data in an attempt to ensure that the sample more accurately reflects the characteristics of the population from which it was drawn. Weighting is used to adjust the relative contribution of the respondents, but it does not involve any changes to the actual answers to survey questions.”
In short, we do more interviews with populations that are mission-centric for us so that we’re better able to understand their opinions when we look at how they responded. This allows us to have greater confidence that we’re accurately reflecting the views of these groups of Coloradans when we report data about their opinions and experiences. Then when we’re reporting data about Coloradans overall, the weighting process balances everything out so that the topline numbers aren’t skewed by our oversamples.
Are you really able to better understand the Asian American and Pacific Islander community with only 76 AAPI respondents in 2020?
According to the U.S. Census Bureau, Asian American and Pacific Islander (AAPI) Coloradans currently make up a little more than 3% of the state’s population. We hoped that Pulse would be able to report data on this population of Coloradans, so we decided to oversample AAPI Coloradans, which is something that few polls in Colorado attempt to do. We set a goal of achieving 100 interviews of AAPI Coloradans because that sample size would yield a margin of error of +/- 9.8% for data reporting AAPI Coloradans’ perspectives and experiences.
Unfortunately, while 76 interviews represent an oversample of the AAPI population in Colorado, we were not able to achieve our goal of 100 total responses. Our AAPI sample has a margin of error of +/-11.5%.
As a Foundation committed to continual learning and improvement, we immediately started digging into the reasons why we weren’t able to achieve 100 interviews in our first attempt at an AAPI oversample. There are likely several reasons why we weren’t able to reach our goal.
Because our sample is drawn randomly from postal service addresses, we have no way of knowing the ethnicity of the residents of each individual address in advance. That means that to efficiently target interviews to ethnic groups that make up small proportions of the overall population, we have to employ one of a number of techniques to increase our likelihood of reaching households with the ethnicity we’re seeking to include.
One technique is to target geographies that, based on Census data, have a high density of residents that share that ethnic background; however, there are relatively few areas in Colorado with extremely high densities of AAPI residents. Another technique is to match the address file to a variety of public databases to identify names associated with that address, and then use surnames to identify residents who may be AAPI. But this also has limited utilities, as surnames are not always strong indicators of ethnicity across all AAPI subgroups. A third option is to use models built from commercial databases (magazine and newspaper subscriptions, purchases, etc.) to identify households with a likely AAPI resident, but these also have only limited accuracy.
We applied all of these techniques, but after extensive calls, emails and texts, they did not produce the 100 interviews we hoped they would. We are committed to trying again in 2021, with new approaches and techniques in hopes of achieving 100 interviews and deepening our understanding of the experiences, priorities and opinions of AAPI Coloradans.
Does the poll give us information about the opinions and experiences of Native Americans in Colorado?
Pulse was able to interview 37 Native American/Indigenous people. Unfortunately, that’s too small a number to feel confident in reporting out their responses in comparison to other subgroups.
According to the U.S. Census Bureau, 2% of Colorado’s population is American Indian or Alaska Native. The population is difficult to reach through research methods based on random sampling for reasons such as poverty level, educational attainment, housing insecurity, age and remoteness.
At The Colorado Health Foundation, when a population is difficult to reach through one methodology like a poll, we look for other ways to listen to these community members. This fall, we’ll be conducting online focus groups with Native American/Indigenous Coloradans as a part of another public opinion research project. This qualitative methodology will allow us to have 90-minute conversations with 20 people as opposed to 20-minute conversations with 90 people (what a poll might seek to achieve). We’re hopeful that these conversations will help us better understand the mindsets of Native American/Indigenous Coloradans.
Is there a way to look at the responses of people who are undocumented?
Pulse: The Colorado Health Foundation Poll asks participants if they were born inside of or outside of the United States. The full results show how those two sub-groups compare in their answers to each question. The poll does not ask about documentation status for people not born in the U.S. In our current political environment, immigrants – both documented and undocumented – may live in fear of deportation or other consequences. Asking about immigration status could increase that fear and likely make many immigrants decline to participate in the poll. While we would love to know the priorities and experiences of undocumented immigrants living in Colorado, we do not want to create additional fear by asking immigration status in our poll. We are challenging ourselves to listen to and explore the perspectives of people who are undocumented in other ways.
How can the data be broken down geographically? Can I get data specific to a county/region of the state?
A poll the size of Pulse – 2,275 respondents – allows for some geographic breakdowns. By reviewing the crosstabs, you can see data broken out by media market and by region.
- For media markets, you can see the Denver media market, the Colorado Springs media market, and “other,” which encompasses all respondents from other markets.
- For region, respondents have been broken into 5 regions: Eastern Plains, Colorado Springs/Pueblo, Larimer/Weld, Denver Metro Area, and West Slope.
In addition to regions, you can see data broken out by the counties with the greatest numbers of respondents. Given the population centers of our state, these counties are all on the Front Range and include Adams, Arapahoe, Boulder, Denver, Douglas, El Paso, Jefferson and Larimer counties.
What’s a “margin of error” and why does it change depending on what part of a poll we’re exploring?
When conducting a poll, we only talk to a sample of the population, and while the sample represents the population, we’d get a truer result if we interviewed every single person in the population. The margin of error describes how close we can reasonably expect a survey result to be to the true population value if we were to hear from everyone. A margin of error of plus or minus 3 percentage points at the 95% confidence level means that if we fielded the same survey
100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times.
In a poll, the most commonly cited margin of error applies to the entire sample (for Pulse, that’s 2,275 adults living in Colorado with a margin of error of +/-2.83%). When we look at sub-groups within the population – such as Hispanic/Latinx adults living in Colorado – the margin of error gets bigger because the sample size gets smaller. While there are a number of factors that influence the margin of error, the predominant factor is sample size – the bigger it is (i.e., the more people you hear from), the smaller the margin of error (i.e., the closer you’ll be to representing the results if you talked to every single person).
Didn’t the 2016 presidential election prove that polling isn’t reliable?
Pulse: The Colorado Health Foundation Poll doesn’t aim to predict the results of any election. Interviews are conducted with Colorado adults, not just voters, because our goal is to know more about the experiences and feelings of all Coloradans to help us bring health closer in reach.
We understand that the experiences of recent polling trying to predict election outcomes and failing to successfully do so has caused a lot of people to feel skeptical. The American Association for Public Opinion Research (AAPOR) commissioned an evaluation of the accuracy of 2016 pre-election polling to better understand what happened. The evaluation is available here.
In short, 2016 wasn’t a perfect year for election polling, but a spotty year for election polls is not an indictment of all survey research or even all polling. Well-designed and rigorously executed polls are still able to produce valuable, accurate information about the attitudes and experiences of the public.
What about people who don’t have landlines or refuse to answer the phone?
To be sure, one of the most significant challenges facing the polling industry is the reality that people often don’t answer their phones. Pulse seeks to mitigate this challenge by using an address-based sampling methodology and contacting Coloradans via multiple modes. We send emails, text messages and postcards – in addition to calling – in an effort to reach a representative sample of our state.
Can we really trust these results? Does The Colorado Health Foundation want to bias the responses to show support for specific policies we are advocating for?
Pulse: The Colorado Health Foundation Poll is committed to transparency; that’s why we’ve released everything – every question we asked and all the results we collected. The poll is also conducted as a partnership with both Democratic and Republican pollsters, and we worked hard to make sure our questions were presented in a balanced way. The poll has no predetermined outcome, and the data speaks for itself.
What languages was the poll conducted in?
Respondents were able to complete the poll in two languages: English and Spanish. According to U.S. Census Bureau data, English is the most spoken language in the state, and Spanish is the second most spoken language with more than 600,000 speakers. We conducted the poll in these two languages to ensure we could reach a representative sample of adults in Colorado.
The 2020 results have been released in English only; our aim is to be able to release results in both English and Spanish starting in 2021.
Although our sample is representative, we know that we weren’t able to include Coloradans who don’t speak English or Spanish. German and Chinese (including Mandarin, Cantonese) are the next two most-spoken languages in the state – each with fewer than 24,000 speakers. Because we want to ensure that Pulse reflects the priorities of all Coloradans, we are considering what additional languages need to be integrated in future iterations of the poll.
You said you’re doing this poll annually. How are you approaching that?
Pulse: The Colorado Health Foundation Poll is an annual commitment because advocates, policymakers, foundations and countless others have told us how useful it is to have consistently updated, moment-in-time data available to help them better understand the people of our state. And it’s helpful for us, too, because we hope to track trends and changes in public opinion over time.
In 2020, we devoted nearly half of the poll to questions about coronavirus as the state continues to wrestle with myriad impacts from the pandemic. It’s possible that we’ll repeat these questions in future years, but we’ll determine that as we continue to monitor COVID-19’s impact on our lives and communities. The remaining questions were developed to be repeated year after year so that we can track what’s on the minds of Coloradans over time. Going forward, we hope each year’s poll will contain a mix of questions that are especially timely and relevant to that year and questions that are repeated over time.