Evidence-Based Instructional Design
How many options should a multiple choice question have? Four. Obviously. One correct response and three distractors. End of story. Why four? Well…look around! Four is just the right number of options.
Back when dinosaurs roamed the earth and everyone wore flannel for some reason, I huddled in a dank, firelit cave with other Gen X children to learn my ABCs. As I recall, just about every test I took featured four-option multiple-choice questions. Four! Later, my ID instructors taught me that four was the correct number of options. The instructional standards I worked from as a young designer also specified four. Four! Even the quiz templates that come with popular eLearning development software are pre-loaded for four-option multiple-choice questions. Four!
But, I’ll humor you. On the ridiculously unlikely chance that I’m wrong about this, let’s take a look at the peer-reviewed research.
Research On Number Of Options In Multiple Choice Assessment Questions
“The findings suggest that three-option MCQs save time for covering more content and items in the test, thereby increasing test validity and reliability.” (A Comparison Between Three-and Four-Option Multiple Choice Questions)
“Students answered three-option MCQs an average of 5 seconds faster than they answered four‐ and five‐option MCQs (36 seconds versus 41 seconds; p = 0.008). There were no significant differences in item difficulty and discrimination or test reliability.” (Reducing the Number of Options on Multiple‐Choice Questions: Response Time, Psychometrics and Standard Setting)
“Data from public sector employment tests are presented that indicate that tests composed of multiple-choice items containing three alternatives have psychometric properties similar to those offered by tests composed of items containing five alternatives. Given the similarity of the psychometric properties and the likely reductions in the cost of development and administration time, three‐alternative multiple-choice items may be preferable to five‐alternative multiple-choice items for some testing purposes.” (Three-Alternative Multiple Choice Tests: An Attractive Alternative )
“The study revealed that [the number of options] did not significantly affect mean test scores, mean test difficulty, and test discrimination indices, but not internal consistency reliability coefficient. Generally, the findings provided more evidence to support the use of three options.” (The Number of Options in a Multiple-Choice Test Item and the Psychometric Characteristics )
A meta-analysis of 80 years of research on multiple-choice assessment construction and evaluation suggests that three is the optimal number of options. (Three Options Are Optimal for Multiple‐Choice Items: A Meta‐Analysis of 80 Years of Research )
Okay, what were we talking about again? Yes, as I was saying, three is the right number of options for multiple-choice questions. Oh, sure, some of the old-timers still talk about four, but that’s so five minutes ago, you know?
In all seriousness, the quantity and quality of peer-reviewed evidence in favor of three-option multiple-choice questions are remarkable. I summarized five studies above, but I found at least a dozen more, all of which supported the use of three-option questions. Based on all this high-octane nerdery, we can conclude the following, with confidence:
- Three-option questions are just as reliable and valid as four- or five-option questions. (In fact, some evidence suggests that three-option questions are more reliable and valid than four- or five-option questions.)
- Three-option questions do not reduce the difficulty of a test compared to four- or five-option questions.
- Scores for tests made up of three-option questions do not differ significantly from tests made up of four- or five-option questions.
- Students can complete three-option questions more quickly than four- or five-option questions.
It sure looks like three-option multiple-choice questions are just as good, or better, than four- or five-option questions in every way that a working ID might care about. In addition, because learners can complete three-option questions more quickly, using them will reduce seat time by increasing the efficiency of instruction.
How much seat time can you reasonably expect to save, though? Let’s use the figures provided in the research above as a rough guide: about 5 seconds per question. For a psychometrician developing standardized tests, that’s a big deal. It means more questions can be completed in a given timeframe, which will increase the validity and reliability of the test.
But how much difference will it make for a typical ID developing workplace training? It depends on the size of your audience. Let’s assume you’ve created a course with 10 multiple-choice questions. Using three-option questions should save you about 50 seconds/learner. If 100 people take your course, that’s going to result in about 83 minutes of saved productivity. If 10,000 people take your course, you’ve saved your organization about 139 hours of seat time.
Even if you’re not going to dramatically reduce seat time, though, there’s another very good reason to adopt the three-option standard: development time. The research I summarized above doesn’t deal with this directly, but it’s logical to assume that creating two distractors will take less time than creating three. Suppose it takes you about 5 minutes to create and validate a distractor. Switching to the three-item standard will save you 50 minutes in development time for a course with 10 multiple-choice questions.
If you’d like to see how much time you could potentially save your learners, or yourself, by switching from four-option multiple-choice questions to three-option MCQs, I’ve created a calculator you can try.
Long live three!
Note: Although I’ve converted to the First Church of Three, I’m mainly looking at it in terms of new development. I won’t be going back to revise old courses. I haven’t done the math, but I doubt the extra development and administrative effort needed to revise and re-deploy existing courses will be justified by savings in seat time.
Originally published at www.linkedin.com.