AI 101 – a short guide to good prompts

Keep hearing about large language models (LLMs) such as OpenAI’s ChatGPT3.5 and GPT4, Google’s Bard and Microsoft’s Bing? Want to understand AI and even use it in your teaching but don’t know where to begin? Here’s the 101 that many academics have been asking me for, to help you harness AI’s power and utilise it more effectively!

What is a Large Language Model?

First, let’s set out what AI chatbots like GPT4 and Bard are. Each chatbot is a form of artificial intelligence known as a “large language model” (LLM). You can view it as supercharged predictive text. Given a series of words, it predicts what word is most likely to come next.

To do this, AIs are “trained” from a huge collection of text data sources such as internet books, websites, blogs, etc. to capture as much human language as possible. In the case of ChatGPT3.5 and GPT4, the training only includes text up to 2021, so it doesn’t have any knowledge or information from after this date.

Think about all the books and papers you’ve ever read, and conversations you’ve had – these influence how you think and what you’re going to say to someone in a conversation. In human terms, if you know someone well enough, you can probably predict what they will say next. This is what the AI is doing when you interact with it. 

After asking the LLM a question (or prompt), they use their training to predict which words to respond with based on all the words it has seen before. During an exchange, the AI can recall what’s been said earlier in that specific chat and can use that information to guide the next prediction, making the interactions feel like a conversation. By default LLMs don’t “remember” the exchanged details between one conversation and the next; once trained, the models don’t directly update or learn from their interactions. Once the session ends, that information is no longer available to the LLM.

If you want a deeper dive, then Google’s Introduction to Generative AI is a good start.

Prompting and asking questions

Interacting with an AI is like using a messaging app: you ask a question and the AI will respond based on the patterns it’s learned during its training. Asking a question is often referred to as ‘prompting’, and crafting effective questions is known as ‘prompt engineering’. We’ll delve deeper into that in a minute. But first, what kind of response can you expect if you ask a simple question?

Prompting can be anything from “What is the weather like today?” to “Tell me a joke” or “Explain how photosynthesis works”. Since LLMs like ChatGPT and GPT4 don’t have access to live information, they can’t tell you what the weather is like today, although AIs/plugins like Bing and Bard do have access to current information and can tell you if it’s likely to rain. If you ask it to tell you a joke without context, then it will tell you the most statistically likely joke from its training set. When prompting on subject specific knowledge, you get a response that is the most statistically likely string of words.

For the prompt, “Explain how photosynthesis works”, the AI does not actually understand photosynthesis but in its training set it will have encountered many explanations of photosynthesis, and so can generate a response that it predicts to be the most likely coherent and correct explanation.

This means that if you put the same prompt into the AI many times, you’ll generate text that’s similar in its content and structure. The results are predictable. Because it’s generating the answer from an algorithm, rather than truly understanding the topic, it can also make mistakes in its choice of words, leading to inaccuracies. This issue is less pronounced for topics that are well-represented in the training data, like fundamental knowledge and core concepts, because the model has a larger pool of information from which to draw its predictions.

Try asking the AI about a topic you know a lot about and see what is produced, then repeat for a topic you know only a little about. For the one you know well, what was missing or incorrect? For the topic you know only a little about, how could you instruct someone to verify the content?

Giving context leads to better responses

Providing context in your prompts leads to more accurate responses, as it supplies the AI with more information to work with. For instance, providing the AI with a ‘ROLE’, ‘SITUATION’ and ‘TASK’ will help guide and improve its answers.

Example: this prompt will tune the AI to create specific questions for an interview. 

ROLE: You are the laboratory manager for a small biotechnology firm.
SITUATION: You are interviewing for a graduate scientist with good practical skills and teamworking.
TASK: Suggest five interview questions that could be used to select the best candidate.

In a similar way, instead of simply asking for an essay on photosynthesis, you can slightly adjust the prompt to provide more context, guiding the AI to produce an essay structure.

You are writing a 2000-word undergraduate essay for a plant biology course. Suggest a structure for the essay as well as the key content and concepts that should be covered.

Here, I have told the AI the level I want it to write at, the length and the target. This is more effective than merely requesting ‘write me an essay’, as it now provides a structure that I can explore and expand on myself. When prompting,you might need to try many iterations of the initial prompt to refine and reshape to get the desired response. For example,the essay structure might have a topic missing that you know needs to be included or a suggestion for an area you don’t need to cover. In subsequent questions, you can ask the AI to remove or expand a given area.

Asking follow-up questions in response can help you further the conversation and explore the content more deeply. For example, in my photosynthesis prompt I gained an eight-point essay plan, where point 4 talked about “Photosynthesis and Energy Transfer”. By asking the question, “Can you give me some guidance and detail on point 4 to help me understand the topic”, I am provided with more information on that specific area.

AI like GPT4 can be a powerful tool, generating text that seems remarkably human. But remember, it’s not perfect. These AI models work on stats, not on true understanding. So, while they can write text on almost any subject, sometimes they might go off topic or use incorrect ideas. They might make a mistake, miss a nuance or even create something that just doesn’t make sense. You have to be the one checking that the AI is on track, making sense and not stepping over any lines.

Now try crafting your first prompt by giving a role, situation and task and see how it changes. Or ask it to draft an essay plan.

Putting new information into the AI

Joshua Sortino on Unsplash

One of the latest enhancements to GPT4 includes plugins that allow direct uploading of files, such as PDFs. Alternatively, you can manually input a body of text and signal the AI to process it, typically by prefacing it with a command like ‘READ’. This command signifies to the AI that the following text should be absorbed and used to guide subsequent responses.

For instance, you might say, “I am going to provide a body of text for you to process. Please read this text and confirm you’ve done so by responding with the word READ.” Once the text is introduced, you can then guide the AI’s use of that information with further prompts. So, if you’ve input the main body of an essay you’ve written, you could use follow-up prompts to instruct the AI, such as:

“Provide a short summary of the information in an abstract style”.

“Reword for clarity and remove redundancy in the text”.

“What are the key points of this text?”.

However, keep in mind that the AI can only use this new information for the duration of the current conversation. It does not ‘learn’ or retain this information for future conversations. Also, be mindful of ethical considerations when inputting personal or sensitive information.

Take a block of text and place in in the square brackets in the prompt below, then ask the AI to summarise, reword or change the text in some way.

“Please read this block of text, when you have read it say READ and wait for the next prompt [YOUR TEXT HERE]”

Adopting personas and having structured conversations

By crafting your prompts, you can instruct the AI to adopt a particular persona or respond in a specific style. Fun examples of this are to have the AI act as if it was a well-known person or to respond in a given manner for instance, as an academic.

“Take on the persona of the Guide from “The Hitchhikers Guide to the Galaxy” – we are going to have a conversation as if I was an intergalactic traveller to earth, are you ready?”

The AI can now converse with me in using that voice. A more academically related prompt might be:

“You are a University Professor writing a module descriptor for a new course, help me write the learning outcomes.”

These prompts can be expanded to include instructions on how the conversation will develop and the direction that it might take. In this example, direct instructions are given to the AI about the way I want it to respond:

“Your role in this conversation is to act as a personal tutor. You are taking on the persona of a University Professor. You will ask a simple question to start with and, if the response is correct, ask increasingly more complex questions. If the question is answered incorrectly, you will provide feedback and hints to help answer the question. Your questioning will be set at undergraduate levels of understanding.

The topic of conversation will be [your topic here].

When you are ready, ask your first question.”

Such a prompt works well on fundamental knowledge topics of established processes that are well represented in the training set. Other ideas might be to write a prompt that guides students through reflective writing or choosing the correct statistical models. 

Try writing a prompt to ask the AI to behave in a given way or take on the persona of your favourite character.

Summary

Harnessing the power of AI, particularly large language models (LLMs), like OpenAI’s GPT4, Google’s Bard and Microsoft’s Bing, can benefit your teaching. By using the tips in this 101, you’ll be able to get started with prompt writing and interact with/use AIs in your practice. The next stage is to consider how you can use this with your students as a tool for learning, and develop both your and their digital skills. One of the best ways to learn is to do, so get stuck in and experiment!

Thanks to this post’s co-author, Mel Green, for her input and editing.

AI 101 – a short guide to good prompts by David Smith is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

AI can do your written assignments, so what now for assessments?

Assessment in education is changing as AIs such as ChatGPT and other language model bots move rapidly into everyday life. Many bots are open access, easy to use and becoming integrated into everyday tools. Think what that means for a moment: students have access to software that can help them create essays, blogs, video transcripts and reflections, suggest workflows, summarise peer review publications, etc. When constructed well, prompt outputs can give the user targeted and relevant information (although not always accurate). At the time of writing, the outputs of such bots are comparable to first-year undergraduate essays and good online exam answers.

The world of assessment has changed, so here are some practical approaches for using of AI in written assessments, rather than fighting against it.

Embrace or displace?

Proponents of AI argue that it can improve efficiency, productivity and access to information. However, others worry that AI could lead to a loss of academic standards and greater inequality and bias in information sources. Outputs are only as good as the database they draw from. Detecting the use of AI in written assessments is possible – the written style can be noticeable and the outputs on the same prompt are similar. There’s even an AI to detect the use of AI. But do we really want to do that? What’s the benefit to the student for punitive action against the use of a tool that’s fast becoming deeply embedded in search engines and word processors, and could become a critical employability skill?

Is the process or the product more important in learning?

Consider how many undergraduate essays you wrote that you still refer to?  Now consider how often you use the skills you gained as an undergraduate in how to write and how to critically assess information. It was the act of creation that was important, not the product.

Spell checkers and Grammarly have already made the creation of error-free text an automated process. Online packages will perform basic statistics in an instant and give you the text to put in your figure legend. Citation managers such as RefWorks and Endnote will curate and produce your bibliography in whatever citation style you need. However, these tools are all procedural rule-following with defined prescriptive outputs. There is little critical thinking in the act of creation.  Higher-level thinking is then centred on how that product is used, the references picked and the interpretation of the data. The same is true for language models – the outputs are only as good as the input prompts used to generate them, and how they are used becomes key. In terms of AI and written assessments, we are interested in the process and the act of creation rather than the product.

Assessment of the process

Assessing the process of writing over the final product has many advantages, as the way a tool has been used becomes part of the output. There are also benefits around tackling academic integrity by having an assessment that builds and works to the final article, where you can track who created it and how. This approach can be used in an individual literature review assessment where each student is given a topic that interests them and is directly linked to their final research project. There are two learning objectives with this assessment:

1. To engage with literature relevant to the future research project.
2. To develop critical thinking skills required for the creation of a review article.

The assessment has a 3,000-word limit, with 2,000 of those words given to the process. Students complete a template detailing the search criteria they’ve used, the databases accessed and the prompts inputted into bots such as ChatGPT. They’re then required to critically evaluate the information gathered from the databases against inclusion and exclusion criteria, and detail how that information will be used in the final written product.

Which AI did you use?

– include a copy of your prompts here

Paste the original output(s) here

Comment on the quality, depth and rigour of the output(s)

– Consider the information (is it correct?) – Bias in the information (is there an alternative viewpoint?) – Identify omissions (is all the information present?)

Detail how you have fact-checked the output, including your peer-reviewed sources

How have you used these outputs to help write your literature review?

Justify your contribution to the final text

Template for use of Generative AI in written assessments

Assessment of the knowledge

Our role as educators is to open the door to the language and thinking needed to generate sensible questions. The information you gain from an AI is only as good as the prompts. If you ask a silly question, you get a silly answer. Asking the right questions is an invaluable professional and personal life skill. We should assess not the product of the AI, but train the students in prompt engineering so they can probe a given topic more deeply. At the early stages of a degree, AI tools could be used to generate expanded notes around the learning materials, through this approach and the design of well-crafted prompts. Further assessments or tutorials are then based on critiquing the outputs and documenting how this was done.

Top and tail with AI

Language models can assist in writing an essay by providing suggestions for structure, content and language. It removes that dreaded blank page and gives a framework to edit and expand. They will:

1. Provide a suggested outline, including an introduction, body paragraphs and conclusion.
2. Help develop statements and the main ideas for each body paragraph.
3. Give suggestions for supporting evidence and examples to include.
4. Make suggestions for grammar, punctuation and clarity.

It’s important to note that tools such as ChatGPT are language models and suggestions should be reviewed and edited, and iterative prompts used to refine the structure. This is where the skill of the individual comes in, through additional information, framing and personal observation. At later stages, AI can be used to get great copy, removing redundant words and suggesting improvements for overall clarity. Final ownership, however, should come from the individual to ensure that it is their voice, experience and understanding that is being represented in the text.

Assessment of skills development

Portfolios are a cornerstone of skills development. They are based on the collection of artefacts and reflections on the experience. AI tools can attempt to write a reflection against a given prompt, but the output is generic and lacks critical feedforward and actionable elements. However, the portfolio was never a valuable item, the act of reflection and action planning was. AI tools can suggest areas to reflect on, which students often struggle with, and help structure the process. The assessment then is in the collection of artefacts, the conceptualisation of individual experience and the personalised application.

Final thoughts

The introduction of generative AI tools in education has sparked a much-needed conversation on assessment practices. We mustn’t regress to using outdated and impersonal methods such as traditional in-person exams, which have proven to not only widen the achievement gap but are also inadequate when evaluating the skills required for today’s workplace. Instead, it is essential that we innovate, adapt and evolve our approach to assessments.

Thanks to this post’s co-authors – ChatGPT and Mel Green – for their input and editing.

---

How artificial intelligence has answered the UnGoogleable exam question

I started my UnGoogleable exam questions blogs (part 1 and part 2) by saying the robots have not won yet. A lot can change in a year and artificial intelligence (AI) has already found a way to beat most UnGoogleable assessments. If the old game was stopping answers from being looked up, the new game is stopping them from being answered by AIs. Platforms, such as ChatGPT, enable students to answer online exam questions, write essays, complete some MCQs, summarise papers and produce reflections for skills portfolios. (It can even write poetry, if you need that in your life!)

Don’t believe me?  Let’s ask the AI to tell you itself:

“AI systems can potentially be used to answer online exams, although this would depend on the specific nature of the exam and the capabilities of the AI system. Some exams may be easier for AI to answer than others, depending on the complexity of the content and the types of questions being asked. If the exam is open-book and allows the use of external resources, an AI system with access to a comprehensive knowledge base and advanced natural language processing capabilities might be able to answer some questions. However, if the exam is closed-book or requires higher-order thinking or problem-solving skills, it is less likely that an AI system would be able to provide accurate answers. Additionally, the use of AI to answer exams could potentially be detected by exam proctors or by the exam software itself and could lead to penalties or other consequences.”

If the AI was a student, what grade would it get? I tried a range of written assessment-style questions in ChatGPT and marked them against my assessment criteria. I then cross-checked the mark with other academics, effectively double-marking and moderating.

Traditional exam/essay style questions ask students to draw on knowledge and discuss against a title. They are a staple of both exams and essays, e.g.:

• Discuss the evidence linking mitochondrial dysfunction to neurodegenerative diseases.
• Compare data-dependent acquisition and data-independent acquisition for the analysis of proteomic data.
• Write a description of gene editing by CRISPR that would be understandable to an A-level Biology student.

The AI’s answers had a reasonable level of specific knowledge in them, the information was correct and multiple points were brought together. However, the text was vague in the way it was discussed and lacked depth in understanding. Given that this AI was trained with information up to 2021, current thinking was missing. Nevertheless, had its answers been presented in a time-limited online exam, I would happily have given them a low to mid 2:1.  As a coursework essay, it would get a 2:2. WOW!

Short answer questions probe knowledge and understanding but don’t always draw on analysis skills. They are found in exams and workbooks, e.g.:

• State how a fluorescently tagged protein can be introduced into a mammalian cell line.
• Write a short 100-word summary of the paper “Engagement with video content in the blended classroom” by Smith and Francis (2022).

In these more direct recall or summary questions, the answers were fully correct – all the details were present and would have been graded at high 2:1 to 1st level. The AI does a really good job of reporting back on existing knowledge, it can even answer most MCQs.

Problem-solving questions give students a situation to apply knowledge and develop a solution. They are designed such that the student needs to draw on what they know against a prompt. Here is an example from a recent two-part exam question where the student was asked to design a workflow and predict outcomes:

1. Describe a series of experiments to show that the induction of stress granules correlates with the activation of the Integrated Stress Response (ISR). 
2. The drug ISRIB is an ISR inhibiting molecule as it binds and promotes eIF2B activity. Describe the effect ISRIB treatment would have on stress granule formation if cells were exposed to oxidative stress and ISRIB. Discuss in your answer what impact this drug would have on the experiments described in part A.

The AI could write a decent experimental plan against a prompt and develop a valid response for part B, about what it had written in part A. The answers were again unfocused in places, and some of the information was not correctly applied or fully appropriate, but it would still easily have gained a 2:2 or low 2:1.

When a similar style of problem-solving question required a more interpretive element, such as using an image as a prompt or a rationale as to why the approach was appropriate, the AI fell over and was unable to answer. Without the text-based context, it had no means by which to work.

Reflections assessments simply take a reflective learning exercise and use it as a tool to assess the learning of the learner. 

When the AI was asked to write a reflective task with the prompt “write a reflection on lab work”, it drew on the generic skills both from personal development and employability which one would gain in that environment. The answer failed to come up with any personal examples though, next steps or future action planning, so it lacked creativity. However, again, it would still grade well.

Surely AI answers are easily detected?

The AI that I played with (ChatGPT) had a specific writing style that set it apart from the other student scripts I read. Spelling and grammatical errors were low, so if that was a notable change from someone’s past writing, your suspicions would be raised. However, with anonymous marking (which is good inclusive practice), and the volume of scripts that are typically marked, you would not spot it. A key red flag though is that any references created by the AI were random and not real papers.

In order to check how effective plagiarism detectors such as Turnitin and Grammarly would be, I ran the same question through the AI ten times. Although you would expect multiple answers to the same question to generate matching text, or to match a pre-existing source, the answers created by the AI each time were worded differently. When I put those ten responses through Turnitin, only two showed text matching above 30%, my normal flag for having a deeper look. So, even our go-to tool for academic integrity did not detect anything amiss!

What is the future for written assessments then?

Each of the assessment prompts used returned viable answers but they were vague, lacked depth and were limited in creative aspects. However, all would achieve a good grade close to the class average. The AI that I worked with could not complete any assessment requiring human subjective judgement, such as ethical or moral assessments. AI also could not complete assessments that require creativity or intuition, as these require a level of human cognition that their systems are not yet capable of.

You cannot stop the use of AI, the genie is out of the bottle and it’s only going to get smarter. While assessments that require physical actions or manipulation of objects (such as practical work), or where the individual is probed or questioned on their understanding (such as vivas or poster presentations) are all potential workarounds, they are not always possible or appropriate. We can try to fight the use of AI but it is better to think more deeply about what the new purpose of assessment should be and what role AI plays. I feel another blog coming on…

Assessment guides us
Gauge the student progress made
For better learning

[An AI generated haiku about the purpose of assessment, with a little bit of help]

How to Write an UnGoogleable Exam Question – Part 2

Google can’t tell you what to think (well I hope not, otherwise the robots have already won)! Forming and expressing your ideas in a way that’s clear, concise and accurate is a key skill, and one that cannot be looked up.

Part 1 of this blog series considered integrating knowledge, spot-the-error style questions and application of knowledge through problem solving. These are all ways of assessing understanding and applying information. But there are even more options for writing ungoogleable questions.

In this post, I share three additional examples of written questions that cannot easily be googled. The trick in this is that I’m asking for opinions and rationale from the students. The information is there, it’s their job to tell me what they think about it.

How to Write an UnGoogleable Exam Question – Part 1

Post COVID-19, the rules of assessment have changed. Online examinations are here to stay, which will typically involve students answering set questions and should really be thought of as time-limited coursework. With the whole of human knowledge just a fingertip away, the problem is how do you prevent the answer being looked up? In Part 1 of this blog series, I’ll look at a few examples that worked in the 2020/21 delivery of my modules.

A non-googleable question is one that cannot be easily answered through a single click in an internet search engine. When written well, they create intellectual challenge and require interpretation and inquiry.

(Is it wrong that I googled that?)

Writing these questions is not easy. You’re looking to assess what the students can do with a given set of information and there are many more examples than those that follow (case studies, data analysis, scenario, opinion pieces). However, the common theme is the application of knowledge, rather than recall.

Surviving and thriving as a dyslexic in academia

I would consider myself to have done quite well as an academic. After 22 years at University, I’ve made it to Reader, gained a national teaching fellowship and published over 45 research papers. But it’s not been easy, and I’ve not done it alone. 

Dyslexia is a broad brush that manifests itself to different degrees in different people. My dyslexia was first recognised when I was in primary school.  Although I can complete complex mathematics, I can’t hold a string of numbers in my head. It takes me longer to read and process information. Often what I write isn’t what I think I’ve written, some words I just can’t get. The phrase “I am certain it’s a curtain” required me to google the correct words. I’ve no idea what my module codes are and never will. Blessed with good teachers, I was taught coping strategies and ways of working that I still draw on today.

Peer group formation in a COVID19 world.

Our latest paper entitled “Self-selecting peer groups formed
within the laboratory environment have a lasting effect on individual student
attainment and working practices” has just been published in FEBS
Open Bio. We explore how peer groups form and the influence this has on the final
attainment of the students. The paper was researched and written long before
COVID19 was even a thing, yet the conclusions hold valuable lessons for new students and the start of a socially distanced teaching year.

Creating and Using a Rapid Feedback Generator

How can you generate rapid individual feedback and individual targeted emails?

Quality feedback is a cornerstone of assessment. What constitutes good feedback has been extensively covered in the literature and here on this blog. In this post, I’ll share how – through a few Excel commands, some mail merging and a macro – you can generate individual feedback rapidly and en masse. Once you have got to grips with the Excel commands and mail merging technique you can apply it in a number of different ways:

Engaging the Silent Majority

How can you ensure all students in a teaching session have the opportunity to engage, be involved and interact?

This is the question that I wanted to address when I set out to understand why students choose to sit in a given location within the lecture theatre.

Deeper thinking and creative classroom use of MCQs

Multiple choice questions (MCQs) or multiple response questions (MRQs) can be more than just knowledge recall questions. They can be used to analyse data or to access higher levels of learning. This blog covers what they are and links to further pages detailing how you can write more complex questions as well as offering a few examples of use and tips for good questions.

A key advantage of MCQs is that they are easy to mark giving the opportunity for instant feedback and can act as a useful learning tool. Blackboard quizzes, as well as Google Forms, have the option of individual feedback, based on the response selected which can also be presented as screencasts and videos. Writing good questions does take work as plausible responses for both correct and incorrect answers are needed. A well-designed MCQ assessment can cover a breadth of content providing an objective measurement of ability.

One criticism of MCQs is that they only assess memory, knowledge and understanding. They can, however, be written to evaluate higher order thinking skills, such as the ability to apply, analyse and evaluate information. It is worth noting that, when used for summative assessment, the highest orders of thinking such as “creation” are out of the reach of MCQs as the answers are predetermined. With only a single or limited answer being possible it is also impossible to demonstrate a breadth of understanding outside the scope of the question stem. So although they are useful, they should be thought of as part of a broader assessment strategy.

MCQs can be useful for the following aspects:

• Recall of facts
• Comprehension of text/graphics/data
• Numerical skills
• Deductive reasoning
• Anything that leads to a right/wrong statement of fact
• Prompted recall of the clearly stated arguments, points of view or opinions of the authors of key texts
• Evaluation of the validity of logical arguments and conclusions based on evidence

How then do you go about writing more complex questions? The links below take you to pages dedicated to each of the following topics. They can be read in order or you can dip in and out of any of them:

1. Higher order questioning with MCQs
2. Using MCQs during teaching sessions
3. Tips for writing good questions
4. Resource list and biography 

#big thanks to Mel Green for editing and proofing these pages and Danny Allwood for content (panic button and other ideas).