Panel Quality: Our Values–Answers to ESOMAR’s 28 Questions
Research Now SSI is a proud member of the ESOMAR community and is pleased to provide our responses to the ESOMAR 28 Questions. The primary aim of these 28 Questions is to increase transparency and raise awareness of key issues when considering online sampling. A related aim is to ensure that what clients receive meets their expectations.
01 What experience does your company have in providing online samples for market research?
With almost 60 years of combined leadership experience and innovation, the Research Now SSI corporate family (“RN SSI”) is uniquely positioned to deliver world class market research sample and data services. RN SSI was the first to make random sample available to researchers and were the inventors of online and B2B panels with the advent of online research. RN SSI is the only provider to offer sample across the full range of modes and can therefore recommend the best methodology for each research project. We are an independent single–source for permission-based data collection across the globe. We manage thousands of projects each month for which we provide sample, survey programming, data processing and other field services. These are executed by industry-leading programming and project management teams. We are experienced, research-literate, multi-lingual data collection specialists, servicing clients from across 41 offices worldwide. RN SSI uses proprietary technology to deliver consistent data, products and services no matter how complex the project or where in the world clients choose to do business with us. Research Now SSI adheres to the highest standards of sampling science across every aspect of every project and sponsors extensive “research on research” revealing critical insights into better sampling and survey design. We are active contributors to 24 market research associations around the world.
02 Please describe and explain the type(s) of online sample sources from which you get respondents. Are these databases? Actively managed research panels? Direct marketing lists? Social networks? Web intercept (also known as river) samples?
Research Now SSI has a variety of sample sources such as panel, web intercept sample, and specialty lists available to meet our clients’ unique project requirements. All panels are actively managed, online access panels built from two decades of experience. All our panels are localized – not just translated – with native language panel support and country-specific reward choices. We run “open enrollment” and “by-invitation-only” ® recruitment campaigns, via direct email and through online marketing channels, utilizing hundreds of diverse, online affiliate partners and targeted websites. “By-Invitation-Only” is a proprietary method of exclusively inviting pre-validated individuals, or individuals who share known characteristics, to enroll into our market research panels. We achieve “By-Invitation-Only” by partnering with a diverse set of globally recognized consumer and business-facing brands.
Sample sources and recruitment
03 If you provide more than one type of sample source: How are the different sample sources blended together to ensure validity? How can this be replicated over time to provide reliability? How do you deal with the possibility of duplication of respondents across sources?
Research Now SSI works to optimally blend our proprietary sample sources by conducting comparability tests and modeling the blend that will achieve the closest match to census and social benchmarks. To ensure reliability over time, we control the blend of multiple sample sources based upon our client’s research requirements. We ensure full transparency with our clients regarding sample sources used, including times when an external panel partner may be required. To prevent duplication, we use third-party digital fingerprint technology.
04 Are your sample source(s) used solely for market research? If not, what other purposes are they used for?
No marketing action is directed at any panelists by Research Now SSI as a result of their participation in research activities.
05 How do you source groups that may be hard to reach on the internet?
We design recruitment campaigns to target hard-to-reach population segments by selecting unique sources and applying tailored campaigns. To guarantee detailed knowledge of the specifics of panelists, we employ hundreds of profiling attributes on our panels. Due to the size of our panels, low incidence groups are accessible. To facilitate this process, we deeply target our panels and deploy pre-screeners to collect information for niche sample targets such as finance, IT decision-makers and health ailments. Research Now SSI recognizes that online may not be the only solution to reach rare groups and can assist clients in understanding the tradeoffs of different data collection modes and recommend the best methodology to achieve the research goal.
06 If, on a particular project, you need to supplement your samples with samples from other providers, how do you select those partners? Is it your policy to notify a client in advance when using a third-party provider?
If we need to supplement sample and we need another provider an external, non-API, panel partner be required, we operate a policy of complete transparency with our clients. We only use trusted panel partners and maintain a preferred supplier list. We always communicate to our clients in the event of using a non-proprietary source. Our dedicated panel partner team has built, and maintains, an extensive global database of quality panel suppliers. Before employing third-party sample, external providers must comply with a pre-identified set of questions and provide credentials. For example, we would ask them whether they have specialty panels, which countries they can cover, what targets their panels include, whether they have a set minimum incidence or a maximum length of interview, etc.
07 What steps do you take to achieve a representative sample of the target population?
Sample selection is based on the sample needs and client requirements for each individual survey, driven by a study’s research objectives. Where possible, pre-targeted sample is used to minimize screen-outs and provide a better quality panelist experience. Customized sampling, e.g. nationally representative, is also available. Research Now SSI can balance sample on outbound invitations, surveys starts or completed interviews. We do this using a wide range of targeting criteria, from simple demographics to more complex behavioral and attitudinal targeting.
Sampling and project management
08 Do you employ a survey router?
Yes, Research Now SSI uses routing technology. Research Now SSI’s next generation technology has been carefully designed to improve both the research participant experience and feasibility while maintaining sampling quality. The amount of sample sourced through our routing technology can be customized per project to meet our clients’ unique research requirements.
09 If you use a router: Please describe the allocation process within your router. How do you decide which surveys might be considered for a respondent? On what priority basis are respondents allocated to surveys?
Research Now SSI’s routing technology uses weighted randomization to assign surveys to participants. Upon entry into the system, panelists are checked to ensure they have not exceeded survey participation limits. A list of potential survey matches is determined for each panelist based upon the information we know about them. Panelists may be asked additional screening questions within the system to ensure they meet the project criteria. Priority may be given to surveys that are behind schedule; however, this is kept to a minimum as survey randomization must remain in place as a key element for preventing bias.
10 If you use a router: What measures do you take to guard against, or mitigate, any bias arising from employing a router? How do you measure and report any bias?
From the outset, Research Now SSI’s routing technology was designed to ensure high–quality sampling and minimize bias. In the development of our technology, we believed an element of randomization was essential to minimize the risk of assignment bias. Research Now SSI has tested its technology settings using multiple data points to ensure that bias is limited. Research Now SSI has also partnered with external consultants to review and validate its routing technology design. Finally, we review reports of system statistics and conversion rates to ensure consistency.
11 If you use a router: Who in your company sets the parameters of the router? Is it a dedicated team or individual project managers?
No single person at Research Now SSI is authorized to unilaterally change technology settings. Our Research Science team is responsible for routing practices, configuration and settings. This team establishes, tests, and measures all aspects of system performance, including sampling rules, router settings, data quality, and panelist participation limits and guidelines, and works together to propose and approve any needed changes. Research Science leaders head a global team of sampling and research experts who assist with thought leadership and research on research best practices and methodologies.
12 What profiling data is held on respondents? How is it done? How does this differ across sample sources? How is it kept up-to-date? If no relevant profiling data is held, how are low incidence projects dealt with?
The collection and updating of information about our panelists is managed and controlled through various approaches. Basic demographic information (e.g. age, gender, region, household demographic) is collected at registration. A verification message and short survey are immediately sent to collect further information. Panelists also have the option to enter information about themselves via their member page at any point. Specific screener surveys are run when Research Now SSI is building a particular sub-group (e.g. automotive, mobile phone habits, financial services) or for a particular project. If the data collected as part of this process is reusable it becomes part of the member profile. Profiling data is consistently updated through Dynamic Profile Enrichment™ solution. Panelists are periodically re-asked profile questions both to qualify them for surveys and to refresh data as appropriate. For screening and targeting needs that are not part of our global profile library, questions are asked as part of the survey flow either through in-router refinement or as part of the programmed survey.
13 Please describe your survey invitation process. What is the proposition people are offered to take part in individual surveys? What information about the project itself is given in the process? Apart from direct invitations to specific surveys (or to a router), what other means of invitation to surveys are respondents exposed to? You should note that not all invitations to participate take the form
Research Now SSI uses invitations of all types including e-mail invitations, phone alerts, banners and messaging on panel community sites to include people with a diversity of motivations to take part in research. At the time of enrollment, new panelists are asked to join an online market research panel. At this point it is made clear that it is not part of a sales process. Our survey invitations provide only basic links and information that is non-leading. Panelists are rewarded for taking part in surveys according to a structured incentive scheme, with the incentive amount offered for a survey determined by the length and content of the survey, the type of data being collected, the nature of the task and the sample characteristics. Panelists are supported by a dedicated team and have the option to unsubscribe at any time. Our panel management is compliant with market research industry standards, data protection and privacy laws.
14 Please describe the (various) incentives that respondents are offered for taking part in your surveys. How does this differ by sample source, by interview length, by respondent characteristics?
Research Now SSI uses an incentive scale which is based on set time increments and panelist characteristics. For example,
a medical practitioner would generally be paid a significantly higher incentive per completed survey than the average consumer. The incentives to be awarded for these “specialist” opinions are discussed with the client so that they are attractive enough for time-poor/money-rich individuals to want to participate. All incentives are awarded only once the survey has been completed. The incentive options allow panelists to redeem from a large range of gift cards, points programs, charitable contributions, and partner products or services.
15 What information about a project do you need in order to give an accurate estimate of feasibility using your own resources?
We will need to know:
- Who you wish to speak to;
- What you will ask of them;
- What other types of data you will be collecting (for example, cookies, behavioral data etc.)
- How much time is available to gather the data.
The first item incorporates geography, demographics, incidence, quotas and the number of interviews required; the second covers the length of the survey, any special tasks required, the questionnaire design and completion difficulty for the participant; the third defines the data gathered, and the fourth the fielding period.
16 Do you measure respondent satisfaction? Is this made available to clients?
Yes, we conduct member satisfaction surveys and project feedback studies. We regularly measure panelist satisfaction on elements such as frequency of invitations, value and diversity of incentives and redemption choices, their willingness to complete various lengths of surveys, and our level of responsiveness to any questions or concerns they share with our Member Services team. Additionally, at the end of completed surveys, we gather feedback from participants on their experience. This data is available to clients in aggregate as a benchmark and for each of the surveys they run. Research Now SSI’s Engagement Team is dedicated to providing a positive participant experience. The team responds quickly to participant inquiries and takes immediate action to resolve any issues or dissatisfaction.
17 What information do you provide to debrief your client after the project has finished?
For projects where Research Now SSI provides programming and hosting services, we capture all participation history, the date of entry and panel recruitment source for each member. Provided it does not involve the disclosure of personally identifiable information, we can provide this to clients as a report upon request. For projects where Research Now SSI hosts the survey, it is also possible to see a full survey activity report for each project. The key statistics from such a report include:
- Fieldwork dates
- Total number of survey invitations sent
- Number of surveys started
- Number of screen-outs/quota-fulls and survey drop-outs
- Number of completes
Additionally, we collect our panelists’ feedback about their survey experience to measure their satisfaction. We can provide this data on demand. It can be helpful to spot strong points and weaknesses in the research design itself.
Data quality and validation
18 Who is responsible for data quality checks?If it is you, do you have in place procedures to reduce or eliminate undesired within survey behaviors, such as (a) random responding, (b) illogical or inconsistent responding, (c) overuse of item non-response
(e.g. “Don’t Know”), or (d) speeding (too rapid survey completion).
Please describe these procedures.
Research Now SSI believes that data quality is a shared responsibility, and that a well-designed survey is the best way to reduce data quality issues and reduce the chance of error from removing too much data. Research Now SSI can help clients and provide feedback based on our rich experience and industry best practices. Data quality is at the forefront of Research Now SSI’s role as a provider of digital data, so for surveys where Research Now SSI provides the hosting and programming, we will run a series of quality checks on the data collected. We monitor the quality of our data via the use of:
a. Random responding: Review of the data to ensure that answers are logical; we also have additional logic checks that can be built into the script to ensure participants cannot continue if they try to submit an illogical answer.
b. Illogical or inconsistent responding: This is monitored and detected by use of logic checks that can be programmed into the script. To ensure that these are fully activated, the project manager completes the survey and attempts to bypass the logic. The project team will also re-check the link to ensure that the programmed logic is operating correctly, prior to the survey going live.
c. Overuse of item non-response (e.g. ‘Don’t Know’): Research Now SSI refers to these participants as flat-liners, who can be identified and removed from the final data during our quality checks.
Responses where the completion time is less than 30% of the median length of the survey are identified across the entire sample. The project manager checks the route that the participant followed to ensure they have not bypassed a significant section of the survey, and completes a sense check of any verbatim. Based on these checks, fast responders are classified as speeders and are removed from the final data. We work closely with our clients to reduce occurrences of survey offending and monitor offenders, employing different techniques to address the behavior of participants who regularly provide poor quality data.
19 How often can any individual be contacted to take part in a survey whether they respond to the contact or not? How does this vary across your sample sources?
Each panelist is assigned an individual ID, so we can record their entire survey participation history. This means we can carefully select panel members for each survey to ensure that they are not only relevant, but also are not being over-contacted. We therefore have limits on how many survey invitations panelists are sent. These limits vary depending on country and sample source. Upon request, we can exclude re-invites to participants who have participated in either a survey of the same topic or for the same client. The timeframe on these limits can be stipulated by the client.
20 How often can any individual take part in a survey? How does this vary across your sample sources? How do you manage this within categories and/or time periods?
Participation limits vary across Research Now SSI’s sources. Some restrictions are hard limits, others soft guidelines. Diversity of participation level increases the diversity of the sample and improves its ability to reflect the target population. Restricting participation must be weighed against the risk of excluding certain people from a survey solely based on their previous participation.
21 Do you maintain individual level data, such as recent participation history, date of entry, source, etc., on your survey respondents?Are you able to supply your client with a project analysis of such individual level data?
We hold the complete survey participation history of every participant on our panel. Each panelist is assigned an individual ID number which stays with them throughout their entire panel membership and allows us to track their history, date of entry, source, etc. We can also provide a per-job analysis to clients.
22 Do you have a confirmation of respondent identity procedure? Do you have procedures to detect fraudulent respondents?Please describe these procedures as they are implemented at sample source registration or at the point of entry to a survey or router. If you offer B2B samples what are the procedures there, if any?
Yes, we have a series of defined processes in place to ensure the high quality of our participants. It includes checking for duplicate participants by evaluating variables such as email address, matches across several demographic data, and device-related data through our use of digital fingerprint technology. Additionally, the nature of our “By-Invitation Only”® panelist recruitment allows us to be confident that real people with valid identities are enrolling in our program. Research Now SSI supports a wide variety of other techniques to ensure sample quality such as external list matching.
Policies and compliance
23 Please describe the ‘opt-in for market research’ processes for all your online sample sources.
All our research activities are opt-in and permission-based. Initial enrollment messaging is clear about the purpose of their membership, with complete and detailed information provided at the time of enrollment. New panelists who click through an invitation to join our panels must complete enrollment at the panel registration website and opt-in to all terms and conditions
- Voluntary co-operation of panelists
- Protection of researchers’ and participants’ identities
- Terms & conditions and privacy policies compliant with local laws
- State-of-the-art data security policies and measures
- Reliable and validated data procedures
- Strict adherence to rules governing the interviewing of children and young people.
For examples of our privacy policies in individual markets, please refer to:
25 Please describe the measures you take to ensure data protection and data security.
Research Now SSI follows all local data protection regulations. Our training teams conduct extensive trainings on Data Protection with client-facing staff highlighting industry (ESOMAR, Insights Association, etc.) guidelines and legal recommendations. We have secure servers to carry out the collection of survey data. Sampling is undertaken with highly encrypted links to the database servers. Personal information is fully protected and can only be communicated following a strict procedure. We also use randomization procedures to ensure there is no preferential treatment of certain parts of the database. Our sampling teams do not have direct access to the database to reveal the identity of users. Survey data is linked to the panel database using numeric IDs so the identity of the end-user (panelist) is always protected. As a company with extensive panel data assets based on recruitment and a “permissioned” approach, we understand the importance of having a mutually beneficial relationship with our research participants – whether consumers or business professionals – and the value which such a relationship brings to enriching marketing and communications. To protect our respondents’ valuable personal information, RN SSI has a team dedicated to privacy law compliance, including the GDPR.
26 What practices do you follow to decide whether online research should be used to present commercially sensitive client data or materials to survey respondents?
Upon the commission of fieldwork, we will make sure the client is comfortable with any commercially-sensitive material being presented or discussed in the course of the project. During sign-up, our panelists agree that they will treat the information they come across in confidence. There are two levels of security features for scripting. Standard security is available by default on all projects, while premium security is only enabled on request and may incur an additional charge. For example, among the solutions developed by our scripting teams, we have tools to disable screenshots and copy-pasting on a computer and we use streaming video to circumvent video buffering and web browser storage of files. We can also watermark an image with the participant’s ID number.
27 Are you certified to any specific quality system ? If so, which one(s)?
Research Now SSI has established a defined project management process and associated systems for each of the thousands
of projects managed each month globally. Research Now SSI routinely reviews quality metrics to maintain high quality. Our Australian offices are certified and comply with ISO 20252 standards.
28 Do you conduct online surveys with children and young people? If so, do you adhere to the standards that ESOMAR provides? What other rules or standards, for example COPPA in the United States, do you comply with?
Yes, we conduct online surveys with children and young adults. In these instances, we adhere to ESOMAR standards and all applicable local regulatory and legal requirements, including COPPA in the U.S.