You’ve taken a multiple choice test before, right?
Of course, you have!
They’re everywhere, from the multiple choice question tests we take in schools to the latest addictive Buzzfeed questionnaire.
As an End-Point Assessment Organisation (EPAO) assessing apprentices, TQUK is responsible for creating a variety of assessment materials, including the test papers that apprentices will encounter. Multiple choice questions will make up, or will included in, many of these test papers.
Now, you’d think that creating an multiple choice questions test would be simple. After all, they’re everywhere; how hard can it be?
Well, as it turns out, it’s a complicated business.
It may be easy to create a Buzzfeed questionnaire — it’s a whole other order to create a quality multiple choice questions test for an apprentice’s end-point assessment (EPA).
Here are just a few factors that the EPA Team takes into account when creating one of our great test papers.
The Role of Multiple Choice Question Writers and Subject Experts
An intrepid EPAO beginning their journey towards creating an amazing multiple choice question test must gather some friends along the way.
After all, you can’t just pick up a pen and decide, willy-nilly, to write the 33rd question on an multiple choice question test for a Commis Chef EPA. Hold your horses!
In order to write appropriate questions for an assessment activity, you need expert help. You need professionals on your side who know what skills an apprentice should know, and when they need to know them, during their course of learning.
TQUK consistently commissions questions from our huge bank of question writers — subject experts, all, from healthcare and hospitality to management and retail — to ensure we are creating rich, high quality questions that populate our real and mock multiple choice question tests.
Language
Certainly, language is a key consideration when composing a test question. Clear phrasing and appropriate difficulty are paramount.
Seems straightforward, right?
Guess again!
Appropriate language levels are the subject of a tremendous amount of debate. What kind of language is appropriate for particular apprenticeships?
Let’s explain.
Say you’re in a level 2 apprenticeship in customer service. Most likely, you and most of your peers will between ages 16-19.
Any test you or they take must be reasonably written to the reading level of someone that age and, by extension, the level of that apprenticeship. You wouldn’t expect to find command verbs like “collate” and “tabulate” in those test questions — such complex words would be more appropriate for a higher level apprenticeship.
The questions we have to ask are, Which words and phrases belong at each level? What kind of sentence structure is too complex? Which words are appropriate for a level 4 but inappropriate for a level 3?
The natural place to start, we thought, was the Ofqual guidance on qualification and component levels. It is helpful at first, but perhaps can’t quite provide the level of detail required. After all, detailing alignment of language within new apprenticeships was not the aim of this document from Ofqual.
But there are models that could provide guidance.
For instance, Bloom’s Taxonomy offers a template on what type of questions and language can be used to assess low and high level thinking skills. These frameworks can be applied to lower and higher level apprenticeships.
In Bloom’s Taxonomy, lower level thinking skills can include knowledge recall, the demonstration of the basic understanding of facts and the application of those facts to particular situations. (ie “What are the different types of chicken meat?”, “Compare the taste profiles of white and dark meat”, or “Could eating undercooked chicken be a risk for contracting salmonella?”)
Higher level thinking skills could include knowledge dissection, synthesising compiled information in a different manner and the ability to evaluate and defend positions and judgments. (“List four ways free range chicken imparts more health benefits. Provide references.” “Take a sweet chicken recipe and convert it to a savoury chicken recipe. Explain how your recipe substitutions will work towards your goal.” “Would dark meat be better paired with red or white wine? Why?”)
As a general rule of thumb, TQUK uses the following criteria in creating level-appropriate test questions:
- Level 1 questions should be one simple sentence with no conjunctions;
- Level 2 and 3 questions should include only 12 words per sentence, and one conjunction per question. Language should be kept as simple as possible;
- Level 4 and 5 questions can be made up of several sentences but should not extend past a reading time of 20 seconds.
This is only one model, however. TQUK relishes the chance to work with our training providers to produce the best possible EPA questions.
Stems
The “stem” is the questions or problem presented at the beginning of the multiple choice question.
EPAOs and their question writers have the benefit of consulting the apprenticeship standards. These lay out all the knowledge, skills and abilities that apprentices need to learn over the course of the apprenticeship.
Stems should be
- Self-contained and meaningful on their own;
- Free of irrelevant material.
A great stem must focus on relevant subject material, should prompt the apprentice to complex thought processes and use scenarios that apprentices could likely encounter.
This is where the experience of our question writers comes in. They will be able to draw on their industry knowledge to create plausible and challenging “what if” scenarios.
Distractors
How difficult should a question be?
Creating the ideal test question requires a certain amount of empathy. You need to make sure the question is appropriately difficult for that apprenticeship while keeping in mind what the apprentice should know at that stage.
For instance, take the following stem question: What are the ingredients used in a white sauce?
The correct answer is: flour, butter and milk.
Distractors are the main source of difficulty with a multiple choice question.
When the other distractors are a) black pudding, milk and mint b) red wine, shallots and calvados or c) gravy granules, water and black food colouring, the correct answer should not be overly difficult to guess. This would be a bad test question.
However, when the distractors are a) flour, olive oil and milk b) rice flour, butter and cream c) flour, butter and cream, the correct answer would be more difficult to choose. This would be a better test question.
Choosing the best possible distractors is always a big consideration when writing test questions. All effective distractors should be:
- Plausible;
- Stated clearly and concisely;
- Mutually exclusive;
- Homogenous in content;
- Free from clues about which response is correct;
- Presented in a logical order.
*
It’s always fascinating how seemingly simple things can demand so much attention!
We’ll continue to do the little things well to ensure apprentices get the best possible assessment service in England today.
To keep up to date with the latest news from TQUK EPA, return to our blog or follow us on Twitter, Facebook and Instagram.
See you around The Hive!
