🤖 Day 11: Generate test data using AI and evaluate its efficacy

📆 30 Days of Testing
, , ,
42

I asked ChatGPT to return a subset of data of its choice

ChatGPT
Certainly! Here’s a subset of data containing information about a fictional company’s employees:

Employee ID Name Department Salary
001 John Doe Engineering $80,000
002 Jane Smith Marketing $75,000
003 David Johnson HR $70,000
004 Emily Brown Finance $85,000
005 Michael Lee Engineering $82,000

This subset includes basic information such as Employee ID, Name, Department, and Salary for five employees in different departments.

Experimenting to learn how the data was generated can be an insightful exercise in understanding data generation processes and possibly uncovering patterns or algorithms used. Here’s a step-by-step guide on how you can approach this:

  1. Understand the Data Structure: Examine the structure of the data provided. In this case, it’s a table with columns such as Employee ID, Name, Department, and Salary.

  2. Analyze the Data Patterns: Look for patterns or relationships within the data. For example, notice that each employee has a unique Employee ID, and there are different departments listed along with corresponding salaries.

  3. Hypothesize Data Generation Process: Based on your observations, make hypotheses about how the data might have been generated. For instance, it seems like the data might have been created randomly or semi-randomly to simulate a small workforce.

  4. Experiment with Random Generation: Try generating similar data using random or semi-random processes. For example, you can use programming languages like Python to generate random numbers for Employee IDs and salaries within a certain range. You can also randomly select names and departments from predefined lists.

  5. Compare Results: Compare the data you generate with the original subset. Look for similarities and differences. Adjust your generation process based on how closely it matches the original data.

  6. Iterate and Refine: Refine your generation process based on your observations and comparison. Try different algorithms or approaches to see how they affect the generated data.

  7. Validate Against Additional Data: If possible, validate your generation process against additional data. This can help ensure that your approach is robust and can generalize to other datasets.

  8. Document Your Findings: Document your experimentation process, including your hypotheses, methods, and observations. This documentation can serve as a reference for future experiments or analyses.

  9. Evaluate the generated data: Review the quality and completeness of the data generated. Since i gave it a blank canvas its immediate table response was pretty impressive and with a summary explanation
    a. How easy was it to generate the data? - Quick fast and super easy
    b. How flexible is the data generation? - Super flexible, a good contextual tweak to your prompt yields the best results
    c. Did the generated data meet your needs? Was it realistic? Yes and Yes

  10. Share your findings: As always, share your findings with the community so they can benefit from your insights. Consider sharing: - shared above
    a. The data problem you were trying to solve and how well you think the tool performed. - Pretty good given that i kept it super basic, i read some of the limations based on other experience where they requested specific data and it came short, giving general data rather than specific data.
    b. Your perceptions about what was positive and negative about the tool and the data generated. - Super fast and easy vs Context without context the value is lost
    c. How the generated data might be improved. - Good context added to the prompt
    d. How might the tool help with your day-to-day testing activities? Fast and easy way to create data, save time and effort.

43

Hi there!
In my opinion and experience, in most situations, we want to use AI to generate test data when we can give a simple query and get a fast response with suitable test data. If you spend too much time and iteration to tune the queries and output test data - this is not an effective approach to generate test data, which means you need relatively complicated data, and the accuracy/quality of the data is important.

Again, from my experience, ChatGPT is okay when you need to generate some simple test data, e.g. you can give it a piece of requirements for field validation and get a set of test strings, or you just need some random names, IP addresses, test string in different languages or with some special characters, etc. Once I needed several hundreds of real website URLs from different locations/different domains, so ChatGPT generated links for me - it was easy and fast. But in many cases, I just prefer to write simple data generators in Python because I’ll be sure of the quality of my test data, I can easily tune scripts for my particular needs, they are stable, etc. + there are some advantages of such an approach you will always have different data according to you params, not 1 set of data generated and stored ones, so you can integrate such test data generators in your testing process and have more real-world like tests. Additionally, I can say that you can even use real obfuscated prod data, it might be limited/partial so you can write your scripts to add missing params and then use what you get, to match with your staging data when you insert it in DB for example.

My point is that you can obviously use AI to generate some test data but there are other more stable approaches :slight_smile: and anyway during the process of the data generation you can use AI for solving partial tasks. AI is a useful tool when you know how and when to use it and not overuse it. Some people just spend much more time trying to generate data sets in ChatGPT instead of asking it to generate the code for test data generation :smile:

44

While doing a manual test run today, there is a step in the middle of flow, which I need to call an API manually to proceed, as the system is not yet completed. To trigger the API calls, I also need to create the test data for the JSON requests.

It was originally time-consuming to input the data correctly and convert all timestamp values for the current time. But with help of ChatGPT, it is simple to generate testing timestamp and ID values directly.

Here is the prompt I use:

This is a json body of an API request:
{
“timestamp”: “2024-03-18T08:30:00.000Z”,
“transmissionTimestamp”: “2024-03-18T08:30:00.000Z”,
“transmissionId”: “d1e580b9-8d8e-4c25-a9af-6f7c3c14cf6b”
}

Please generate similar data with current time for timestamp and transmissionTimestamp, and a unique transmissionId generated by current timestamp in the same format.

And the response from ChatGPT:

Sure, here’s a JSON body similar to the one you provided, with the current time for timestamp and transmissionTimestamp, and a unique transmissionId generated based on the current timestamp:

{
    "timestamp": "2024-03-18T10:15:30.000Z",
    "transmissionTimestamp": "2024-03-18T10:15:30.000Z",
    "transmissionId": "e2b81fa0-3d7c-4e30-a2a2-0215ef9b4792"
}

In this JSON:

  • timestamp and transmissionTimestamp have been updated to the current time.
  • transmissionId is a unique identifier generated based on the current timestamp.

Fast and simple timestamp generation without code.

However due to privacy concerns, I could not just pass it the whole requests to it to generate all test data, as it includes some sensitive information related to the client’s business, so I could only extract some generic parts of the sample to send to ChatGPT for the test data generation.

45

:slight_smile: We should always remember that LLMs are probability machines and not fact machines and so any output is always potentially inaccurate.

One option is to ask the model to check the data it generated against the original rules for the field. Sometimes, the model recognises the error and regenerates better examples of the data.

Looking at the prompt, the instructions are fairly complicated and dense; you are using an approach that is effectively zero-shot prompting where you give instructions and kind of hope for the best. You might find if you provide examples (One-Short or Few-Shot Prompting) that the model produces better outputs.

Once you are happy with outputs, try asking the model to produce some code to generate the data you want…you wouldn’t want to have to go through this prompting process each time you need a bit of extra data for that field :slight_smile:

46

Excellent!

One option that might address your privacy issue is to host an open-source LLM within your organisation or on your machine.

Using LlamaFile (GitHub - Mozilla-Ocho/llamafile: Distribute and run LLMs with a single file.) from Mozilla is probably the easiest approach to try this out. It provides a set of open-source models packaged up to run on a local machine. They provide a ChatGPT like interface running as a local webserver against a locally downloaded LLM. So your data is kept private.

A few things to note:

  • You’d need to be able to download and run arbitrary exes on your machine so may not be allowed on some corporate machines.
  • the best performance is likely to come from the larger models but these may be too large or run slowly on a desktop.
  • you are accessing the base LLM model and so it may not have all the additional features and controls that you find with ChatGPT, Bard, CoPilot and the like.

Have fun

1 Like
47

Thank you for the tip! It works seamlessly on my computer :sparkles:

1 Like
48

Greetings Everyone!
Currently, I am working on testing a form having ‘upload image’ and a field to enter ‘URL’ so I made a similar prompt: Test data to check a field having URL as its input
Here are my updates on this task:

  1. Tool:* ChatGPT
  2. Find a Data Problem to solve: Test Data for the prompt :
  3. Valid URLs:
  1. Invalid URLs:
  1. Special cases:
  • Empty field
  • Field containing only spaces
  • Field containing non-URL text
  • Field containing special characters
  1. Experiment with the tool: But then one of my colleagues (@poojitha-chandra) in this group suggested trying a new prompt type so I updated the previous one and made it to Act as a QA engineer and provide test data to check a field having URL as its input and the test data for this case were:

Valid URLs:

  1. Evaluate the generated data: The second time, the test data was much more clear and in-depth (which is something a QA would do), it covered a lot of things that the generic previous prompt did not have.
  2. Share your findings: Providing an accurate prompt is all that these AI tools need to provide the actual outcome desired.

Thanks!

1 Like
49

Tool Selection: I asked Chat GPT to do today’s homework

For this experiment, I’ll choose the ChatGPT (me!) as the tool for generating test data. ChatGPT, based on the GPT-3.5 architecture, can create text based on prompts, making it a versatile option for generating various types of data.

Data Problem:

Let’s consider a scenario where we need to generate test data for a customer feedback form. The form includes fields such as:

  1. Name
  2. Email
  3. Phone Number
  4. Feedback Message

We want to generate realistic, diverse data to test the form’s functionality and ensure it handles different types of inputs.

Experiment with ChatGPT for Data Generation:

Prompt:

cssCopy code

Generate sample data for a customer feedback form. Include names, emails, phone numbers, and feedback messages. Aim for diverse and realistic entries.

Generated Data:

  1. Name:
  • Emily Johnson
  • Carlos Rodriguez
  • Mei Chen
  • Tyrone Washington
  • Priya Patel
  1. Email:
  1. Phone Number:
  • (555) 123-4567
  • +1 (234) 567-8901
  • 555-987-6543
  • (333) 222-1111
  • +44 20 1234 5678
  1. Feedback Message:
  • “I had a great experience with your product! Will recommend to friends.”
  • “The customer service was excellent. Thank you!”
  • “Please improve the delivery times. Otherwise, good service.”
  • “I encountered an issue with my order. Could you please assist?”
  • “Overall satisfied, but the pricing could be more competitive.”

Evaluation of Generated Data:

a. Ease of Data Generation:

Generating the data was straightforward with ChatGPT. I provided a simple prompt, and the AI model produced diverse entries for names, emails, phone numbers, and feedback messages.

b. Flexibility of Data Generation:

ChatGPT showed flexibility by creating varied examples, including different name formats, email domains, phone number styles, and feedback messages. This flexibility allows for a range of test scenarios.

c. Meeting Needs & Realism:

The generated data met the needs of our customer feedback form testing. The entries were realistic and resembled what one might expect in a real-world scenario. Names, emails, and feedback messages were coherent and believable.

Findings:

a. Data Problem & Tool Performance:

ChatGPT performed well in generating test data for the customer feedback form. It provided diverse and realistic examples for each field, allowing for comprehensive testing of the form’s functionality.

b. Positive Aspects:

  • Diversity: The tool generated diverse data, providing a range of names, emails, phone numbers, and feedback messages.
  • Realism: Entries resembled authentic customer inputs, enhancing the test’s authenticity.
  • Ease of Use: Generating the data was easy and quick by providing a simple prompt.

c. Areas for Improvement:

  • Validation Constraints: ChatGPT does not inherently understand data validation rules. It’s up to the user to ensure generated data fits specific constraints (e.g., valid email formats, phone number lengths).
  • Fine-Tuning: For more precise control, fine-tuning the model with specific prompts or parameters could enhance data generation for specialized needs.

d. Tool’s Impact on Testing Activities:

ChatGPT can significantly benefit day-to-day testing activities by:

  • Providing diverse and realistic test data for various scenarios.
  • Saving time in creating sample data sets, especially for forms, surveys, or user inputs.
  • Offering flexibility to adapt to different testing needs with simple prompts.

Conclusion:

Using ChatGPT for generating test data proved to be effective and efficient for our customer feedback form scenario. The tool’s ability to produce diverse, realistic entries makes it a valuable asset for testing various applications, especially those requiring simulated user inputs. With considerations for validation constraints and potential fine-tuning, ChatGPT can greatly streamline testing activities and ensure robust application testing.

1 Like
50

the output looks really good. Which tools did you use and how many prompts did you need to get this result?