🤖 Day 9: Evaluate prompt quality and try to improve it

[indres (Indre Stasiuniene)]

Hi,
I evaluated the generated responses according to these techniques:

Evaluation Techniques:

1. Relevance: Does the AI model’s response align with the intended purpose and context of the prompt?
2. Coherence: Is the AI model’s response logical, well-structured, and easy to understand?
3. Accuracy: Are the facts and information in the AI model’s response correct and up-to-date?
4. Bias: Does the AI model’s response exhibit any unintended biases or stereotypes?
5. Creativity: Does the AI model’s response offer novel or innovative ideas, solutions, or perspectives?
6. Efficiency: How quickly does the AI model generate a response, and how effectively does it utilize tokens?

I choose for evaluating prompt to generate detailed test scenarios for signing up to Ministry of testing website following test scenarios structure.

ChatGPT generate answer quit quickly and easy to understand, also they was good structured:

but there was a lack of test scenarios, so I corrected my prompt in order to get complete and more actual list of test scenarios. As result same of additionally given test scenarios excample was the same as previous, but same of them was useful and cover the part of missing scenarios.

[p.schrijver (Simon Schrijver)]

Based on the conversation with the AI, I adjusted my prompt several times with some feedback of the previous session. Every time I adjusted my prompt, I started a new session.
I must say it got better results when I mentioned the minimum/maximum length of the password in a separate requirement.

Question to ChatGPT:

You are an experienced tester. You are asked to generate a detailed test scenario for both expected and edge cases for a Password Validator

Use the set of requirements enclosed by triple quotes:

“””

Password must meet all conditions as mentioned below

• Password must contain one digit or more.
• And, Password must contain one upper case alphabet or more
• And, Password must contain one lower case alphabet or more.
• And, Password must contain one special character or more. The following are considered special characters, !@#$%^&*()
• And, It is not allowed to have a space in the password

The length of the Password is also a requirement. It must contain a minimum 8 of characters (8 included). The limit is a maximum of 20 characters (20 included).

“”"

Answer from ChatGPT:

Here’s a detailed test scenario for the Password Validator based on the provided requirements:

Test Scenario 1: Expected Cases

• Input: “Password123@”
  • Expected Output: Valid password.
• Input: “Test@1234”
  • Expected Output: Valid password.
• Input: “StrongPWD12#”
  • Expected Output: Valid password.
• Input: “PassWord$123”
  • Expected Output: Valid password.
• Input: “1234@Abcde”
  • Expected Output: Valid password.

Test Scenario 2: Edge Cases

• Minimum Length Boundary Testing:
  • Input: “Pwd@123” (7 characters)
  • Expected Output: Password length is below the minimum limit.
  • Input: “Pwd@1234” (8 characters)
  • Expected Output: Valid password.
• Maximum Length Boundary Testing:
  • Input: “LongPassword1234567890@” (21 characters)
  • Expected Output: Password length exceeds the maximum limit.
• Password Composition Testing:
  • Input: “onlylowercase"
  • Expected Output: Password does not meet the requirement for containing an upper case alphabet.
  • Input: “ONLYUPPERCASE”
  • Expected Output: Password does not meet the requirement for containing a lower case alphabet.
  • Input: “1234567890”
  • Expected Output: Password does not meet the requirement for containing a special character.
  • Input: “With space@”
  • Expected Output: Password contains a space, which is not allowed.
• Combination Testing:
  • Input: “Short@” (Shorter than minimum length)
  • Expected Output: Password length is below the minimum limit.
  • Input: “TooLongPasswordWith1234567890@” (Exceeds maximum length)
  • Expected Output: Password length exceeds the maximum limit.
  • Input: “ValidPassw0rd” (Missing special character)
  • Expected Output: Password does not meet the requirement for containing a special character.
  • Input: “Password!@123” (Valid)
  • Expected Output: Valid password.
• Special Characters Testing:
  • Input: “SpecialCharOnly#”
  • Expected Output: Valid password.
  • Input: “SpecialCharOnly@” (Different special character)
  • Expected Output: Valid password.
• Numeric Testing:
  • Input: “NoNumbers@”
  • Expected Output: Password does not meet the requirement for containing a digit.
  • Input: “1234567890” (Only numbers)
  • Expected Output: Password does not meet the requirement for containing an upper case alphabet, a lower case alphabet, and a special character.
• Mixed Testing:
  • Input: “MixedPass123@” (Valid)
  • Expected Output: Valid password.
  • Input: “MixedPass@” (Valid, but missing digit)
  • Expected Output: Password does not meet the requirement for containing a digit.
  • Input: “MixedPassword123” (Valid, but missing special character)
  • Expected Output: Password does not meet the requirement for containing a special character.

This set of test scenarios covers both the expected and edge cases, ensuring thorough testing of the Password Validator against the specified requirements.

I still see some hallucination effect in the results, but less than before

[ash_winter]

Being trained (or at least evaluated) by a tool on how to use the tool is definitely new! I suppose I requested the feedback though, so at least it was by invitation.

Humans don’t always wait until you ask for feedback…

[appu365 (Appu D)]

Thank you for detailed explanation of each task with prompts. As a beginner tester its useful for me , to improve my testing by considering different aspects of test cases(Positive, Negative, edge cases, exploratory, checklists to verify the test cases and so on)

[joyz (Joyz Ng)]

After some research and discussion with different bots, I try to aggregate a simple checklist for what we can do to improve the prompts when trying to generate manual test cases.

General:
Use complete sentences
Minimize ambiguity, be specific
Give instructions on the tone and lanugage
Give instructions on formatting

Context:
Set a role
Define the system under test
Provide diverse user scenarios for the feature to test
Provide functional specification for the feature to test
Define the data requirement of user input

Output:
Give example(s) of expected test case
Describe types of test cases needed
Set a proper number of test cases needed

Refinement:
Provide feedback and refine
Compare the result with other AI models’ responses

[markcole]

I left off my discussion with chatGPT with
“So what practical plan to implement the five philosophical insights user cases?”

The first user case was:

“Socratic Method: Crafting Effective Test Cases”

and the suggested plan was:

Plan:

1. Training and Familiarity:

• Dedicate time to deepen your understanding of the Socratic method.
• Explore resources on how it can be applied to software testing and quality assurance.

2. Integration into Testing Process:

• Identify specific test cases or scenarios where applying the Socratic method would be beneficial.
• Start incorporating open-ended questions into your test plans to encourage critical thinking.

3. Feedback and Iteration:

• Seek feedback from team members and stakeholders on the effectiveness of the Socratic approach in test cases.
• Iterate based on the feedback received, refining the questions to enhance their impact.

For sake of brevity, I will just deal with the first user case, although I am very interested in exploring the other ones such as #4, “reflective journaling” and #5, “seeking guidance”. If anyone wants to give me guidance on seeking guidance I would appreciate it!

So for #1 - Socratic Method: Crafting Effect Test Cases

After clarifying what the Socratic Method was and how it could be used to test a school discipline application (it gave me a list of questions to ask people) - I asked chatGPT:

What is the most important Socratic Question that needs to be answered to test a school discipline application?

Here is the answer:

The Socratic Method is a philosophical approach to inquiry and dialogue attributed to the ancient Greek philosopher Socrates. It involves asking probing questions to stimulate critical thinking, uncover underlying assumptions, and clarify concepts. Rather than providing direct answers, the Socratic Method encourages individuals to explore ideas, challenge their beliefs, and arrive at deeper insights through reasoned discourse.

The most important Socratic question to be answered when testing a school discipline application would likely focus on the application’s alignment with the underlying principles and objectives of discipline management in an educational setting. Here’s a suggested question:

Socratic Question: “How does the design and functionality of the discipline application support the overarching goals and values of disciplinary practices within the school?”

This question encourages stakeholders to critically evaluate the application’s effectiveness in meeting the broader objectives of maintaining a safe, respectful, and conducive learning environment. By exploring this question, testers can assess various aspects of the application, including its adherence to school policies, its ability to promote fairness and accountability, and its contribution to positive student behavior and academic success.

SO, my conclusions:

In my first session of asking questions, I came up with use cases to become a better tester using philosophical insights. ChatGPT gave me a five step plan with user cases.

In my second session I drilled down to what I specifically am working on and then I asked it to give me the most important Socratic Question to base my test on.

“What is the most important Socratic Question that needs to be answered to test a school discipline application?”

It provided a question that I can ask the stake holders:

Socratic Question: “How does the design and functionality of the discipline application support the overarching goals and values of disciplinary practices within the school?”

This is mainly a better drill down - my first prompt was very general and I kept on asking it more specific questions until I could get to something very practical, relevant to what I am doing.

[simon_tomes]

@manojk

By refining prompts through evaluation, we enhance the effectiveness of our testing processes, contributing to the advancement of prompt engineering practices in the testing community. Let’s continue to iterate and improve for better outcomes.

Nice one, @manojk. What a good way to frame things!

Defo something in us collecting prompts and sharing them with each other.

[blanchec (Blanche Carstens)]

Before: You are tasked with testing the sign-up functionality for an online platform similar to the Ministry of Testing (MoT). Create a set of test scenarios to ensure the sign-up process functions correctly and efficiently. Consider various aspects such as user experience, security, and error handling. Your scenarios should cover both typical and edge cases to thoroughly validate the sign-up process. Please evaluate my prompt for :Contextual Understanding and Correctness

After applying the Refine and evaluate technique with a focus on relevance and correctness:

You are assigned to test the sign-up functionality for an online platform, which closely resembles the Ministry of Testing (MoT). The aim is to ensure the sign-up process operates correctly, efficiently, and securely while providing an optimal user experience. Your task involves creating a set of comprehensive test scenarios covering various aspects, including user experience, security measures, and error handling, encompassing both typical and edge cases to thoroughly validate the sign-up process.

[sargam (Sargam Panwar)]

Hi all!
So in my previous section:
The prompt was: Give test cases for the forgot password screen.
Research Evaluation Techniques: The output that ChatGPT provided was pretty much, with the respective input and its expected output. But the only thing missing was the test cases regarding the UI or the look and feel of the ‘Forgot Password’ page. It would have been great if it could have given me generic cases for both UI and FN(Functional).
Apply and Share:

UI Test Cases:

1. Appearance and Layout:

• Verify that the “Forgot Password” page is visually appealing and consistent with the overall application design.
• Ensure that all elements (input fields, buttons, labels) are properly aligned and spaced.
• Check that the font size, color, and style are consistent with the application’s design guidelines.

2. Responsiveness:

• Test the page’s responsiveness on different devices and screen sizes (desktop, mobile, tablet).
• Ensure that the page layout adjusts dynamically to fit different screen resolutions.
• Verify that all elements remain functional and accessible across various devices.

3. Error Messages:

• Test the display of error messages for invalid inputs (e.g., invalid email address).
• Verify that error messages are displayed prominently and in a user-friendly manner.
• Ensure that error messages are displayed in real-time as the user interacts with the input fields.

4. Input Field Validation:

• Test input field validation for various scenarios (empty field, invalid email format, etc.).
• Verify that appropriate error messages are displayed when invalid input is entered.
• Ensure that input field validation occurs both on blur and on form submission.

5. UI Elements Accessibility:

• Test the accessibility of UI elements using screen readers and keyboard navigation.
• Verify that all UI elements have appropriate labels and attributes for accessibility.
• Ensure that users with disabilities can navigate and interact with the “Forgot Password” page effectively.

Functional Test Cases:

1. Valid Email Submission:

• Enter a valid email address and submit the form.
• Verify that the system accepts the email address and initiates the password reset process.
• Check that the user receives a password reset email containing a reset link.

2. Invalid Email Submission:

• Enter an invalid email address format (e.g., missing “@” symbol) and submit the form.
• Verify that the system displays an error message indicating the invalid email format.
• Ensure that the form submission is not processed until a valid email address is provided.

3. Non-Existent Email Submission:

• Enter an email address that is not registered in the system and submit the form.
• Verify that the system displays a message indicating that the email address is not associated with any account.
• Ensure that no password reset email is sent for non-existent email addresses.

AND so on…

[futurelady25 (Amruta Bhokare)]

Original BDD Scenario

Feature: Ministry of Testing Website Login

As a registered user of the Ministry of Testing website,
I want to be able to log in with my credentials,
So that I can access exclusive content and manage my account.

Scenario: Successful User Login
Given I am on the Ministry of Testing website login page
When I enter my valid username “testuser@example.com” and password “Password123”
And I click the “Login” button
Then I should be logged in to my account
And I should see a welcome message “Welcome, Test User!”

After
Peer Review
Pilot Testing
Stakeholder Consultation

Updated Prompt: Create a BDD scenario for a new user launching Ministry of Testing for first time. User registers on the website and logs in successfully on the website.

Feature: Ministry of Testing Website Registration and Login

As a new user of the Ministry of Testing website,
I want to register for an account,
And then log in successfully to access the website’s features.

Scenario: New User Registration and Successful Login
Given I am on the Ministry of Testing website homepage
When I click on the “Register” or “Sign Up” link
Then I should be directed to the registration page

When I enter my full name "John Doe", email "johndoe@example.com", and a password "Welcome123"
And I confirm the password "Welcome123"
And I click the "Register" or "Create Account" button
Then I should see a success message "Registration successful! Please log in to continue."

When I am redirected to the login page
And I enter my registered email "johndoe@example.com" and password "Welcome123"
And I click the "Login" button
Then I should be logged in to my new account
And I should see a welcome message "Welcome, John Doe!"