🤖 Day 8: Craft a detailed prompt to support test activities

[manojk (Manoj Kumar Barik)]

Welcome to Day 8 of 30 Days of AI in Testing! Today, I’m diving into the advanced challenges, ready to put my prompt engineering skills to the test!

Challenge Overview: Today’s challenges offer an opportunity to explore more complex scenarios in AI-assisted testing. From comparative feature analysis to evaluating test strategies, I’ll be engaging with Large Language Models (LLMs) to enhance my testing prowess. Let’s dive in!

Comparative Feature Analysis: In this challenge, I’ll task the LLM with summarizing changes between two sets of feature requirements and identifying areas needing testing. It’s all about managing feature evolution effectively!

Test Evaluation: Next up, I’ll present a set of test cases and feature requirements to the LLM and ask it to evaluate the quality and completeness of the tests. This will provide valuable insights into the effectiveness of my testing strategies.

LLMs Evaluating LLMs: Lastly, I’ll engage an LLM to generate scenarios for a feature and then prompt it to assess the quality of those scenarios based on the feature requirements. It’s like getting the AI to reflect on its own work!

Insights and Reflections: Through these challenges, I aim to refine my prompt engineering skills, experiment with different framing techniques, and learn how to elicit more useful responses from the LLMs. Let’s see where this journey takes me!

The following code defines a class AdvancedTestingChallenges with methods for each of the advanced-level challenges. You can easily create prompts by passing relevant inputs to these methods. The sample usage demonstrates how to use these methods to generate prompts for each challenge.

class AdvancedTestingChallenges:
def init(self):
pass

def comparative_feature_analysis(self, requirements_set_1, requirements_set_2):
    prompt = f"Given two sets of requirements representing different versions of a feature, please summarize the changes between them and highlight areas that need testing.\n\n"
    prompt += f"Sample Requirements Set 1:\n{requirements_set_1}\n\n"
    prompt += f"Sample Requirements Set 2:\n{requirements_set_2}"
    return prompt

def test_evaluation(self, test_cases, feature_requirements):
    prompt = "Given the following test cases and feature requirements, evaluate the completeness and quality of these tests. Provide insights into how well the tests cover the requirements.\n\n"
    prompt += "Sample Test Cases:\n"
    for i, test_case in enumerate(test_cases):
        prompt += f"- Test Case {i+1}: {test_case}\n"
    prompt += "\nSample Feature Requirements:\n"
    prompt += feature_requirements
    return prompt

def llm_evaluation(self, feature_requirements, generated_scenarios):
    prompt = "Using an LLM, generate a set of scenarios for the 'User Profile Management' feature based on the given requirements. Then, assess the quality, completeness, and accuracy of these scenarios in relation to the feature requirements.\n\n"
    prompt += f"Sample Generated Scenarios:\n"
    for i, scenario in enumerate(generated_scenarios):
        prompt += f"- Scenario {i+1}: {scenario}\n"
    return prompt

Sample usage:

challenges = AdvancedTestingChallenges()

requirements_set_1 = “”"

• Feature: User Profile Management
• Requirements:
  1. Users should be able to update their profile information.
  2. Users should have the option to upload a profile picture.
  3. Users can set privacy settings for their profile.
     “”"

requirements_set_2 = “”"

• Feature: Enhanced User Profile Management
• Requirements:
  1. Implement two-factor authentication for profile updates.
  2. Allow users to connect their profile with social media accounts.
  3. Enable users to customize profile themes.
     “”"

test_cases = [
“Verify that users can successfully update their profile information.”,
“Validate that users can upload a profile picture in supported formats.”,
“Test privacy settings functionality across different user roles.”
]

feature_requirements = “”"
Requirements:

1. Users should be able to update their profile information.
2. Users should have the option to upload a profile picture.
3. Users can set privacy settings for their profile.
   “”"

generated_scenarios = [
“User updates profile information successfully.”,
“User uploads a profile picture but encounters an unsupported format error.”,
“User adjusts privacy settings and experiences changes immediately.”
]

print(challenges.comparative_feature_analysis(requirements_set_1, requirements_set_2))
print(challenges.test_evaluation(test_cases, feature_requirements))
print(challenges.llm_evaluation(feature_requirements, generated_scenarios))

Here are the outputs generated by the AdvancedTestingChallenges class for the provided requirements:

Given two sets of requirements representing different versions of a feature, please summarize the changes between them and highlight areas that need testing.

Sample Requirements Set 1:

• Feature: User Profile Management
• Requirements:
  1. Users should be able to update their profile information.
  2. Users should have the option to upload a profile picture.
  3. Users can set privacy settings for their profile.

Sample Requirements Set 2:

• Feature: Enhanced User Profile Management
• Requirements:
  1. Implement two-factor authentication for profile updates.
  2. Allow users to connect their profile with social media accounts.
  3. Enable users to customize profile themes.

Given the following test cases and feature requirements, evaluate the completeness and quality of these tests. Provide insights into how well the tests cover the requirements.

Sample Test Cases:

• Test Case 1: Verify that users can successfully update their profile information.
• Test Case 2: Validate that users can upload a profile picture in supported formats.
• Test Case 3: Test privacy settings functionality across different user roles.

Sample Feature Requirements:
Requirements:

1. Users should be able to update their profile information.
2. Users should have the option to upload a profile picture.
3. Users can set privacy settings for their profile.

Using an LLM, generate a set of scenarios for the ‘User Profile Management’ feature based on the given requirements. Then, assess the quality, completeness, and accuracy of these scenarios in relation to the feature requirements.

Sample Generated Scenarios:

• Scenario 1: User updates profile information successfully.
• Scenario 2: User uploads a profile picture but encounters an unsupported format error.
• Scenario 3: User adjusts privacy settings and experiences changes immediately.

With excitement and determination, I’m ready to tackle these advanced challenges and continue my exploration of AI in testing!

Cheers All!!!