Darlena Jones, PhD., Director of Assessment and Research, Association for Institutional Research
March 8, 2019
Every department at an institution should have access to an online survey system for research and assessment purposes. There are a lot of variables to consider with selecting an online survey system and a lot of commercial solutions to exam.
Before we explore these products, you should ask your institutional research or effectiveness office if your institution has a contract established with a vendor that allows you to use their software as part of the existing institution package (e.g., Campus Labs, Qualtrics). If so, there might be considerable cost savings for your department. But, if you don’t have access to a software system, or you are unhappy with your current solution, this blog post is for you!
There are likely a hundred variables you could consider when choosing an online survey solution but, in my opinion, these are the most important:
How much will it cost? Some survey systems are free (e.g., Survey Monkey, Google Surveys) but more sophisticated systems will charge (e.g., SurveyGizmo, QuestionPro). These charges could be by individual survey or an annual contract, for example. You will need to talk with your department’s leader to understand your budget for survey data collection software.
How robust is the software provider’s data security? This can be a critical issue in protecting respondent privacy especially if you’re using this system to collect sensitive data. The online survey system you choose should have documents for you, and/or your technology team, to review to ensure that it meets your institution’s data security demands.
Is the online survey system Section 508 compliant? Section 508, an amendment to the United States Workforce Rehabilitation Act of 1973, is a federal law mandating that all electronic and information technology developed, procured, maintained, or used by the federal government be accessible to people with disabilities.
If your institution is publicly funded, you must ensure that your survey system meets these standards. And, if your institution is not publicly-funded, ensuring accessibility is the right thing to do. So, the survey system that you select should provide you with documentation that they meet these accessibility standards.
Is the survey set-up system robust? In other words, is the survey set up easy to use, does it provide a wide range of question types, and does it allow you to customize the look and feel of the survey (e.g., upload the institution’s logo, change the survey font, change the colors)?
How are respondents notified about the survey? Less sophisticated survey systems generate a form with a link that you copy and paste into an email that is sent to your participant list from your email address. Anyone who receives that email can respond to the survey. Unfortunately, participants can forward that email to others to respond to which can cause data integrity issues. In addition, these less-sophisticated systems can make your surveys public which means that anyone who randomly finds your survey can respond.
More sophisticated survey systems allow you to upload a participant list (the list typically includes the participant’s first name, last name, and email address) and the system emails the participant a unique, encrypted link that can only be used by that participant. Those surveys cannot be found by an internet search. This helps ensure that data quality is high.
Will the system send reminders to only non-respondents? You’ll be sending multiple reminders to help improve your response rates, but those reminders should only go to those who have not yet responded. If a survey system emails everyone, including respondents, respondents may become irritated that they continue to receive reminders for a survey they’ve already completed and may ignore future survey requests.
Does the online reporting meet your needs? Some systems have minimal reporting while others have sophisticated reporting functions. How much reporting is necessary? Do you intend to produce your analyses? If so, then a system with less advanced reporting may be adequate. But, if you intend to rely on the system to produce analyses, you’ll need to opt for a more sophisticated system.
Can you easily access your responses? Most systems allow you to download your responses data; but if it doesn’t, walk away. You should always have easy access to your responses data.
Does the data coding make sense? If you intend to perform your own analyses, the data must be coded in an easily-understood way and the system must be able to generate a downloadable data dictionary (a file that tells you how each survey question is coded). If it doesn’t, you may be faced spending several hours decrypting data or putting those data into formats that will make analyses easier.
How does its survey opt-out work? Ethical survey tools require the survey researcher to include a link in the invitation and reminder emails allowing the respondent to “opt-out” from the survey; this stops the system from sending them reminders to take the survey. If the survey system you’re evaluating does not allow for opt-out, then don’t choose that system.
In addition, some systems have an opt-out system that allows the survey participant to no longer receive any survey you send. In other words, besides opting-out of your current survey, they will never receive another survey invitation from you again! Avoid these systems since it will impact all future studies you conduct.
Can the system allow for branching/skip logic? Invariably, you’ll design a survey where some questions don’t apply to some participants and you’ll want to “branch” or “skip” those participants around those questions. Make sure that the system you choose has robust skip logic if you anticipate design advanced surveys.
Instead of buying access to a survey system where you design the survey, you could consider buying access to a commercially-produced survey which typically comes with a fee (e.g., Skyfactor, NSSE). The costs may appear prohibitive, but consider what comes with that fee:
Darlena Jones, Ph.D. serves as Director of Assessment and Research for AIR. Darlena directs assessment and research initiatives for AIR such as the annual conference, IPEDS training, and the National Data Institute. Darlena also leads two large initiatives: Development of AIR’s online educational opportunities and the National Survey of Institutional Research Offices. Darlena earned her doctorate in physics, has 10+ years of collegiate teaching experience, and has 20+ years in survey research and program assessment.
If you’re interested in learning more about AIR or accessing our resources, please visit our website, www.airweb.org.
STAY CONNECTED: to the AER KC by following us on twitter @NASPA_AERKC. Use #SAassess to follow conversations! This is a great way to interact with KC leadership and other KC members. You can also "like" us on Facebook for weekly updates. If you are interested in contributing to the blog e-mail [email protected] for more information.
Opinions expressed in this blog are those of the author(s) and do not necessarily reflect the opinions of NASPA. If you agree or disagree with the content of this post, we encourage you to dialogue in the comment section below. NASPA reserves the right to remove any blog that is inaccurate or offensive.