Topic outline

  • AI testing

    Collapse all Expand all

    • Here we are using the test AI tool to experiment with the best settings to allow the use of AI to support an ECR (Example Candidate Response).

      The idea here is that we:

      1. Present a Cambridge question
      2. Allow the user to respond
      3. Provide feedback on the quality of their answer, making use of the Cambridge mark scheme 


      This could be part of a larger ECR item where we make use of the existing ECR and include this AI-powered self-review tool.

      Importantly - here we are using the OpenAI.api - meaning that only queries are sent to OpenAI but no Cambridge data is retained beyond the 30-day abuse policy as in their terms & conditions.

  • Attempt #0

    • Cambridge question
    • markingmode - text-davinci-003
    • Question + list of 'Answers' (based on Mark Scheme), and 'Markers' (based on marking feedback/comments from mark scheme)
    • Query sent to OpenAI is: 
       
      You are a teacher grading an exam question.
Question: What particular challenges has Denise faced as an elite-level cyclist?
The student has given the following answer to the question:
Answer: I would suggest that Denise faces a range of challenges including concerns about her ability to compete with other riders, the mental challenge, and many more...
You use the following markers to grade the question:
Marker: A very effective response that demonstrates a thorough understanding of the requirements of the task. • Demonstrates understanding of a wide range of relevant ideas and is consistently well-focused. Points are skilfully selected to demonstrate an overview.
Marker: A relevant response that is expressed clearly, fluently and mostly with concision. The response is well organised. The response is in the candidate’s own words (where appropriate), using a range of well-chosen vocabulary which clarifies meaning. Spelling, punctuation and grammar are mostly accurate.
Marker: Should be between 100 and 120 words
Your grade and motivation:

    • Example output:

      I would give this answer an A grade. The student's response is very effective and demonstrates a thorough understanding of the requirements of the task. They have provided a clear, concise, and well-structured answer that covers a wide range of relevant ideas. The student has used appropriate vocabulary to clarify their meaning, and the spelling, punctuation, and grammar are mostly accurate. The length of the response is also within the specified range of 100-120 words.

  • Attempt #1

    • Cambridge question
    • markingmode - text-davinci-003
    • Question + list of 'Answers' (based on Mark Scheme), no 'Markers'
    • Query sent to OpenAI is: 
       
      You are a teacher grading an exam question.
Question: What particular challenges has Denise faced as an elite-level cyclist?
The student has given the following answer to the question:
Answer: I would suggest that Denise faces a range of challenges including concerns about her ability to compete with other riders, the mental challenge, and many more...
Your grade and motivation:

    • Example output:

      I would give this answer a grade of B. The student has provided an accurate and detailed explanation of the challenges Denise might face as an elite-level cyclist. However, the student could have provided more specific examples of the challenges Denise has faced to receive a higher grade. Well done!

  • Attempt #2

    • Cambridge question
    • Marking mode - text-davinci-003
    • Question + no 'Answers', 'Markers' (based on Mark Scheme)
    • Query sent to OpenAI is: 
      You are a teacher grading an exam question.
Question: What particular challenges has Denise faced as an elite-level cyclist?
The student has given the following answer to the question:
Answer: I would suggest that Denise faces a range of challenges including concerns about her ability to compete with other riders, the mental challenge, and many more...
You use the following markers to grade the question:
Marker: physical setbacks / effects of childhood accident
Marker: doubting it was possible
Marker: going against what others thought / challenging expectations
Marker: limiting social norms / low expectations of her due to stereotype(s)
Marker: (other people’s) worries that she might hurt herself / concerns for her physical health
Marker: feeling unattractive / daunted by society’s perception of the perfect body
Marker: having to break new ground / being the first female paracyclist to complete the mountain tour
Marker: extreme demands of tour(s) / 865 kilometres long ride(s) / near-vertical climbing in tours / extreme changes in weather in competition / hours of climbing in races
Marker: slow process to improve / many sessions of patient and individualised coaching / specialist training
Marker: learning to use the prosthetic leg on the bike / learning not to depend on the permanent leg to do the work
Marker: falling off the bike
Marker: concerned might not be able to afford bike / fear of being a financial burden to parents / adapted bikes are worryingly expensive
Marker: problems with finding funding / funding sources hard to find / having to wait for funding
Your grade and motivation:


    • Example output:

      I would give this answer an 8/10. The student has provided a comprehensive list of challenges Denise has faced as an elite-level cyclist. They have mentioned the physical setbacks and effects of her childhood accident, the doubts she may have had about her ability to compete with other riders, the mental challenge of going against what others thought, the low expectations of her due to stereotypes and other people's concerns for her physical health. The student has also mentioned feeling unattractive due to society's perception of the perfect body, the challenge of being the first female paracyclist to complete the mountain tour, the extreme demands of the tour, the slow process to improve, learning to use the prosthetic leg on the bike, falling off the bike, the financial burden of needing an adapted bike and the problems with finding funding.

  • Attempt #3

    • Cambridge question
    • aifeedback - text-davinci-003
    • Question + list of 'Answers' (based on Mark Scheme), no 'Markers'
    • Query sent to OpenAI is: 
      You are a teacher grading an exam question.
Question: What particular challenges has Denise faced as an elite-level cyclist?
The student has given the following answer to the question:
Answer: I would suggest that Denise faces a range of challenges including concerns about her ability to compete with other riders, the mental challenge, and many more...
State if the answer is correct and provide some feedback:

    • Example output:


      The answer is partially correct. Denise does face a range of challenges, including concerns about her ability to compete with other riders and the mental challenge. However, it would be helpful to provide some specific examples to illustrate these challenges.

  • Attempt #4

    • Cambridge question
    • aifeedback - text-davinci-003
    • Question + no 'Answers', 'Markers' (based on Mark Scheme)
    • Query sent to OpenAI is: 
      You are a teacher grading an exam question.
Question: What particular challenges has Denise faced as an elite-level cyclist?
The student has given the following answer to the question:
Answer: I would suggest that Denise faces a range of challenges including concerns about her ability to compete with other riders, the mental challenge, and many more...
State if the answer is correct and provide some feedback:


    • Output:

      The answer is partially correct. Denise may face challenges such as concerns about her ability to compete with other riders and the mental challenge, but there may be other challenges such as physical fitness, nutrition, and financial pressures. Additionally, the answer could be more detailed and provide specific examples.