How to Write an UnGoogleable Exam Question – Part 1

Post COVID-19, the rules of assessment have changed. Online examinations are here to stay, which will typically involve students answering set questions and should really be thought of as time-limited coursework. With the whole of human knowledge just a fingertip away, the problem is how do you prevent the answer being looked up? In Part 1 of this blog series, I’ll look at a few examples that worked in the 2020/21 delivery of my modules.

A non-googleable question is one that cannot be easily answered through a single click in an internet search engine. When written well, they create intellectual challenge and require interpretation and inquiry.

(Is it wrong that I googled that?)

Writing these questions is not easy. You’re looking to assess what the students can do with a given set of information and there are many more examples than those that follow (case studies, data analysis, scenario, opinion pieces). However, the common theme is the application of knowledge, rather than recall.

Show me the knowledge

To get going, how do you stop the answer being looked up? The short reply is that without some fancy key login software you can’t (and even then students could use their phone). Embrace the change, alter your mindset and re-write your learning objectives.

LO1: Students will be able to find accurate and relevant information on a given subject.

There’s a skill in being able to look up and find relevant information. For first (and maybe second) year students, being able to report back a definition or meaning relevant to a prompt can be a learning objective. It involves the students understanding that what they are reading is relevant to the question asked. This is a good pass-level descriptor. Alternatively, do the googling for them, then ask the students to work with the most immediate form of information. Here, the question is asking the students to work with two sets of easily obtainable information and then process them in some way.

“A protein domain is a region of the protein’s polypeptide chain that is self-stabilizing and that folds independently from the rest. Each domain forms a compact folded three-dimensional structure. Many proteins consist of several domains. One domain may appear in a variety of different proteins.”

Wikipedia

Example Question 1: Using the definitions below, state the fundamental difference between a globular domain and a transmembrane domain.

Transmembrane domain usually denotes a transmembrane segment of single alpha helix of a transmembrane protein. More broadly, a transmembrane domain is any membrane-spanning protein domain.

Wikipedia

Question pools

MCQs are great. What’s even better in an online exam is they can mark themselves. When written well, those questions can access high-level thinking and allow numerical answers to be determined. I have written on this topic here.

LO2: Students will be able to demonstrate an understanding of key information and be able to analyse and apply that information.

Students love to talk. There is an entire global industry set up just for them to communicate in large groups in real time. So how do you stop answer swapping on WhatsApp? Enter the question pool! For each MCQ a number of variations are written. Your virtual learning environment can be set up such that one question is randomly pulled from the pool and presented to the students (Blackboard, Moodle and Canvas all do this). In a 30 question test, with each question having four variations, that’s 1.8 x1019 different exam papers. You can’t stop them asking a friend for the answer, but hopefully they’re too busy doing the test themselves to respond.

Questions can be written in such a way as to easily make the pool and keep consistency. Having the same or similar responses ensures that all questions are of equal difficulty. The factors in example two can be changed, leading to a different correct response. To create the pool, all I need to do is change the numerical values (pI and pH), however the answer set remains the same.

Example Question 2: A protein with a pI of 5.2 at a pH 8.0 would bind to which type of ion exchange column:

a) Cation exchange because the protein has a positive charge.
b) Cation exchange because the protein has a negative charge.
c) Anion ion exchange because the protein has a positive charge.
d) Anion ion exchange because the protein has a negative charge.

Spot/correct the error

In this question type, the idea is to take a statement, passage or image describing a given scenario or system and add in deliberate errors. The students are required to both identify and correct those errors. The other bonus to this type of question is that pools can be created and, by use of drop-down lists or fill in the blank style questions, the marking can be automated. The question style requires the students to analyse the text or image and demonstrate their understanding through the corrections. The VLE can also be setup with expected responses making the question self-mark. In the full answer there were 15 possible correct/incorrect responses. If you write the text or create the images yourself, then the information can not easily be googled.

LO3: Students will be able evaluate key information on a given topic.

Example Question 3 (image): The peptide below has deliberate errors added in. Use your knowledge of amino acids and peptides to identify the errors.

Peptide structure with deliberate errors

The answer can be free text if you wish or, if you want to automate the process, the students can select the error from a list of possibilities:

a) C-term and N-term labelled incorrectly. (correct answer)
b) Incorrect charge on terminal residues. (correct answer)
c) The glutamine side-chain cannot be ionized.
d) The C-terminal amino acid backbone has too many carbon atoms.
e) The side chain of Aspartate carries an incorrect charge. 

To make other questions with the same answer set, I just need to draw new images that match the other possible errors.

Example Question 4 (text): Read the passage below and correct any errors in the text. Possible errors are highlighted in bold.

Enzymes are macromolecules that catalyze and decrease the rate of chemical reactions. Enzymes do not affect the equilibrium (i.e. the thermodynamics) of a reaction, as they fundamentally alter the structure and energetics of the products and substrates, but rather they allow the reaction equilibrium to be attained more rapidly.  While most catalysts can act on a number of different types of reactions, a key feature of an enzyme is that it is specific……………..

Fill in the blanks to correct the errors where they occur.

Enzymes are macromolecules that catalyze and [A] the rate of chemical reactions. Enzymes [B] affect the equilibrium (i.e. the thermodynamics) of a reaction, as they [C] alter the structure and energetics of the products and substrates, but rather they allow the reaction equilibrium to be attained more rapidly.  While most catalysts can act on a number of different types of reactions, a key feature of an enzyme is that it is [D]……………

Get creative

The highest levels of learning require students to create. You will have spent a whole module teaching core concepts and ideas, now is the time to let them use this information. One way to do that is with open-ended questions – forget having one correct answer and set your marking to be flexible and adaptable, assessing the idea and feasibility of what’s being suggested. The challenge for the marker is that there is no predetermined correct answer and many possible solutions. Set real situations the students would face on graduation, or problems they are likely to have to solve.

LO4: Students will be able apply their knowledge and understanding to generate unique solutions to a given problem.

The example below is drawn from the final paragraphs of a research paper (I have also used the research interests of the teaching team). Typically within the Biosciences research papers, these paragraphs suggest an area of study or a new direction for the investigation. Students are required to suggest potential experiments to address the problem.

Collusion is the issue here in online exams, as students do talk to each other and there is the risk of similar answers being produced. However, you can argue that in practice people would work together to hew their ideas and thus this replicates the real world.

Example Question 5: Within the paper discussion (Limatola et al. 2018), the authors comment on the effects of post translational modifications on proteolytic processing of α-synuclein. Set out a detailed experimental plan to determine if phosphorylation of these Tyrosine residues is critical for the proteolytic processing of α-synuclein.

Assessment criteria out of ten: (I often show this to the students in advance so they know what I am looking for. Here, I am assessing if the methods picked are feasible, and show an understanding of how they would be used in practice. For higher marks, I look for a logical presentation and expected outcomes.)

Pass – attempt to answer question, relevant methods present with some form of description that may contain errors or misunderstanding.
5 – attempt to answer question, relevant methods present with some form of description that is correct.
6 – good attempt to answer question, relevant methods present with some form of detailed description that is correct.
7 – good attempt to answer question, relevant methods present with detailed description that is correct and presented in a logical work flow.
(note here the shift to the high end by the addition of expected outcomes)
8 – excellent attempt to answer question, fully integrated methods present with detailed description that is correct plus logical workflow and expected outcomes.
9 – excellent attempt to answer question, extensive methods present with detailed description that is correct plus logical workflow and expected outcomes.
10 – outstanding attempt to answer question, extensive methods present with fully detailed description that is correct plus logical workflow and expected outcomes.

Summary

Online or not, ungoogleble questions are testing the application of knowledge rather than recall. They can allow the student to be creative and offer the opportunity of authentic assessments. When writing the exam, you want it to be passable, meet the learning objectives and discriminate between students. To do this, use a mix of question types alongside some easy-to-get questions to reach the pass mark. Put yourself in the place of the student: have a go at completing your own exam to see how long it takes, googling your questions to see what comes up!

Look out for Part 2 which will cover discussions, and asking the students for their opinions on and evaluation of information (sourced from Google and beyond).

The page has been translated into French by Dr Philippe Dessus and his excellent blogs can be found here.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

5 thoughts on “How to Write an UnGoogleable Exam Question – Part 1

  1. Pingback: How to Write an UnGoogleable Exam Question – Part 2 | David's adventures in the classroom.

  2. Pingback: Preparing Final Assessments - TECHE

  3. Pingback: Can we use multiple-choice questions for assessment in any subject? | thesedablog

  4. Pingback: Weekly Resource Roundup – 23/11/2022 |

  5. Pingback: Crynodeb Wythnosol o Adnoddau – 23/11/2022 |

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s