Quantitative online surveys are powerful vehicles for collecting information to solve business problems. As we’ve touched on previously in this blog series, a well-designed survey with clearly defined research objectives, a robust screener, and thoughtful survey questions will yield high-quality data. Similar to any other vehicle, however, surveys require maintenance to keep performance at its best. Implementing high performing data quality checks can be an effective tool to make the most of your research investment by protecting your data from undesirable participants and keeping your survey performing in top shape.

Data Quality Checks – A Survey Service Manual

A list of our recommended data quality checks, when you need them, and how to apply them.

You wouldn’t take your brand new car to the service center and ask for everything on the automotive service menu – instead, you would ask the technician to determine what type of maintenance is required. Survey maintenance also comes in various levels. While a myriad of options exist, a skillful researcher is selective in how they leverage data quality checks – by determining the appropriate level of survey maintenance, you will keep your survey running in peak condition, ensuring powerful insights to help you outperform the competition.

Preventative Maintenance – Basic checks to make sure your survey runs smoothly

Even if, like me, you know very little about cars, you recognize the importance of basic vehicle maintenance, and a great mechanic whose recommendations you can trust. Research Now recommends measuring three key behaviors to ensure high-quality data in every survey: Speeding, straightlining, and nonsensical answers to open-end or fill-in-the-blank questions. Survey participants who exhibit these behaviors should be identified, or flagged, in the dataset.

Straightlining

Straightlining is a term that describes participants who answer similarly in several questions or in the same pattern on scaled questions. Participants straightline for two main reasons:

1. Legitimately having the same response for all options
2. Inattention to the survey question

Participants may feel strongly about certain products and attributes like politics, for example. On other topics, they may feel neutral (e.g. millennials identifying their level of interest for household products). As such, it is important to consider question structure and content to ensure a flag is appropriate – penalizing straightlining may not be appropriate for all questions.

Speeding

The industry standard is to remove participants who finish in less than 1/3 the median length of interview (LOI). In surveys where every participant doesn’t see every question, it is important to account for the shortest survey path when determining if someone is speeding. Many companies utilize platforms that have sophisticated speeding checks, monitoring engagement on every page and instances of speeding throughout the survey. Utilizing this approach is quite useful when the platform can account for skip patterns and different paths through the survey and also adapt for participants quickly answering easy recall questions, such as demographics.

Research Now considers participants taking the survey at an extreme pace to be the one instance where a singular infraction can be enough to remove a participant from the dataset. No one trusts a participant who never hits the brakes!

Open-end gibberish

In some circumstances, it is beneficial to ask open-end or fill-in-the-blank questions to obtain participants unprompted responses. When they are leveraged, open-end responses should not be mandatory as not everyone will want to elaborate. In a mandatory open-ended question, participants may enter gibberish (i.e., ‘asdfhjkl’) as a way to indicate a lack of a response.

If open-end questions are optional, flag cases with nonsensical answers for a review of the survey participant’s full set of responses to determine whether they should be removed from the dataset. In some cases, they may not need to be removed. For example, a participant who provides thoughtful answers throughout a survey is not flagged for straightlining or speeding, but provides gibberish in an open-ended question may just be a poor written communicator – but still useful for your data.

Advanced Checks – For optimized performance

A deeper knowledge or validation check is a useful additional data quality measure in situations where the audience must have a particular skill or specific topic expertise to truly provide insight. Knowledge checks can be used in combination with the above three measures, but researchers should be cognizant of overburdening the participant and interrupting the cognitive thought process the questionnaire is supposed to be measuring.

Knowledge checks

When you are interviewing in a very technical space, for example, you may need to validate that participants are knowledgeable in the space, as opposed to simply holding the title that implies they are. For instance, a participant that identifies as a Vehicle Maintenance Technician but can’t identify the function of basic car parts is not demonstrating they have the baseline knowledge needed to answer survey questions, and should be terminated from the survey (and will hopefully never work on your car!)

Attention Checks – The indicator lights of survey maintenance

Many times, concern centers on bad actors in the online space, perhaps due to the element of the unseen with digital data collection. While this is the exception, the “noise” undesirable participants contribute can create overall uncertainty. Pressure check attention by leveraging the following approaches:

Red herrings

Include red herrings, or fake brands, in a larger list of brands. One essential step? Creating a red herring that is a truly ‘fake’ brand, not similar to anything legitimately operating in the space or well-known in other spaces. For example, even if instructions ask participants if they have ‘heard of any of the following brands of auto parts,’ the human brain has a hard time distinguishing that McDonald’s is not an auto part brand but rather a yummy place to eat fries on a cheat day. It feels unnatural not to select something I know I have heard of. However, if a participant selects that they have actually used 2 or more fake brands you can reasonably assume they are not paying close attention. I might not be able to keep my brain from identifying that I have awareness of a brand called McDonalds, but I can certainly tell you I didn’t buy auto parts there.

Low incidence occurrences

Have you earned your pilot’s license and bought an expensive car in the same year? No? Me either! Most people probably haven’t, and including multiple low-incidence occurrences like these in a larger list can be a valuable flag for identifying discreditable behavior. There are people who have, however, and if we were to target high net worth participants we might even discover a higher incidence of them. To validate participant attention by including low-incidence occurrences in a list of larger options, add two (or more) un-correlated options.

Front/back validation

Validation checks ask participants two related questions at different locations in the survey such as zip code and state of residence. Responses to the second question should match the first question. It is important to recognize, however, apparent contradictions may actually be truthful answers. For example, a person may own a car but not service the car, may buy healthy food and never eat it, or have a gym membership but not use it. In any given time period, we might all be that person :).

Check your alignment

Data quality measures should identify participants who display discreditable behavior and not remove participants who are answering in good faith and might be experiencing a brief distraction (i.e. a child running through the room). Be discerning. Leverage effective data quality checks that produce high-quality data but don’t steer you off path.

Use quality measures that are congruent with the survey topic and do not take away from the design of the survey. Participants are sick of being asked to ‘select 2 on this row’ after being provided instructions in the question to select their level of agreement on a 1-5 scale. It is confusing and can be frustrating, especially for diligent participants. Provide your participants with clear directions or they may react differently than expected when presented with a fork in the road. The right service level

Research Now’s best practice is to either terminate immediately or remove participants from the dataset at the conclusion of fielding if they are flagged for at least 2+ or 3+ data quality measures. This way we aren’t penalizing someone who read a single question incorrectly but are removing participants who exhibit bad behavior throughout the survey. The result is typically a 4-5% removal for quality, which can vary based on the survey design, audience, and subject matter – which we will explore more in the final blog of our series.

While quality checks are important to ensuring proper survey maintenance, implementing too many or too stringent quality checks can lead to over cleaning the data or bias the data to support a hypothesis (e.g. “I don’t think a responder should be answering this way”). Excessive quality checks are the equivalent of taking in a brand new car for service and asking for the 200,000-mile checkup, when all you really need is an oil change.

Sometimes you need a specialist.

It’s easy to put air in your tires, but if you need to remove your engine, you might want a little help with that. Think of Research Now as your trusted survey specialist. If you are implementing data quality checks for the first time, or are a tenured researcher with a tricky subject matter – Research Now is here to help. We will get you out on the road to high-quality data in no time!

Stay tuned for the final installment of our high-quality data series: validating the data.

This blog is part 3 of a 4-part blog series by Research Now. To check out parts 1, 2, and 4, click the hyperlinks.