🤖 Day 14: Generate AI test code and share your experience

[nao (Nao Deng)]

1. About Choosing a Tool

This time I chose Postman AI Assistant because I am currently implementing API testing and API automation regression testing in the project. I hope to gain some practical experience in using AI to enhance API testing efficiency that can be applied from the trial process of the Postman AI Assistant tool.

About the use of the Postman tool: Since Postman announced in May 2023 that it would gradually phase out the Scratch Pad model with offline capabilities, most functions will move to the cloud, and you must log in to use all the features of Postman. Our company has been notified to stop using Postman and migrate to other tools. Since then, I have been researching and learning to use Bruno, an open-source tool that can replace Postman for API testing and API automation regression testing. Recently, I have also implemented Bruno+github in the project team for interface document management and interface automation testing, and worked with developers to manage and test APIs using Bruno+github.

About postman replacement tool bruno introduction blog post:Introducing Bruno for Replacement Postman | Nao's Blog

Postman AI Assistant’s official introduction:

Postbot, an AI assistant for API workflows, will be available in an early access program on May 22, 2023.

• Availability: Early access program starts on May 22, 2023.
• Features: AI-driven autocomplete, test case design, documentation writing, test suite building, data report summarization, API call debugging.
• Pricing: Available in Basic and Professional plans at $9/user/month starting October 15, 2023.

I downloaded Postman and tried Postbot with commonly used demo interfaces:

2. About Creating Some Test Code

After adding the demo interface request in the postman interface, click Postbot on the bottom menu bar to start the Postman AI Assistant. A suggestion command menu for the request appears in the Postbot window, currently with the following recommended commands:

• Add tests to this request
• Test for response
• Visualize response
• Save a field from response
• Add documentation

Next, I will try the functions suggested by Postbot one by one.

Trying Add tests to this request

Click Add tests to this request in the Postbot interface

If you have added a request but have not clicked send to run that request,
Postbot will prompt “I’ll need a response to perform this action,” and Postbot will also provide a menu to quickly run the request and output the response; after clicking the “Send request and continue” button, Postman will automatically run the request and write the test script, as shown below:

Postbot’s test script for the demo request is as follows:

pm.test("Response status code is 201", function () {
    pm.response.to.have.status(201);
});

pm.test("Response has the required fields - title, body, userId, and id", function () {
    const responseData = pm.response.json();
    pm.expect(responseData.title).to.exist;
    pm.expect(responseData.body).to.exist;
    pm.expect(responseData.userId).to.exist;
    pm.expect(responseData.id).to.exist;
});

pm.test("Title is a non-empty string", function () {
    const responseData = pm.response.json();
    pm.expect(responseData).to.be.an('object');
    pm.expect(responseData.title).to.be.a('string').and.to.have.lengthOf.at.least(1, "Title should not be empty");
});

pm.test("Body is a non-empty string", function () {
    const responseData = pm.response.json();
    pm.expect(responseData).to.be.an('object');
    pm.expect(responseData.body).to.be.a('string').and.to.have.lengthOf.at.least(1, "Body should not be empty");
});

pm.test("UserId is a positive integer", function () {
    const responseData = pm.response.json();
    pm.expect(responseData.userId).to.be.a('number');
    pm.expect(responseData.userId).to.be.above(0, "UserId should be a positive integer");
});

The written test covers the interface response’s status judgment and body field type judgment and can run through.

At this point, I noticed that two new recommended commands were added to Postbot’s suggestion menu

• Add more tests
• Fix test

I first tried running “Add more tests,” and then Postbot added a few more tests

Interestingly, one of the tests failed, so I clicked “Fix test” to try to let Postbot fix this wrong test

However, Postbot did not fix this wrong test case successfully

This wrong test case is as follows:

pm.test("UserId matches the ID of the user who created the post", function () {
    const requestUserId = pm.request.json().userId;
    const responseData = pm.response.json();
    pm.expect(responseData.userId).to.equal(requestUserId);
});

I can only manually fix it, and the corrected script is as follows

pm.test("UserId matches the ID of the user who created the post", function () {

    const requestUserId = JSON
    .parse(pm.request.body.raw).userId;
    const responseData = pm.response.json();
    pm.expect(responseData.userId).to.equal(requestUserId);
});

The script was wrong because the request body was in raw format and needed to be parsed into a JSON object before being read.

Trying Test for response

After clicking Test for response in the Postbot interface, Postbot will update the test cases generated by Add tests to this request as shown below:

By examining the results of the updated tests, I found that most of the updated cases could not run through.

Then I tried to fix the wrong cases through Postbot’s “Fix test”, most of the cases could run through, but there were still errors in the test cases generated by the Add tests to this request command.

In addition, clicking on Postbot’s “Fix test” to fix the cases generated by the Test for response command will update most of the cases to the test cases generated by the Add tests to this request command

I wonder where the difference between the Add tests to this request and Test for response commands is?

Trying Visualize response

After clicking Visualize response in the Postbot interface, you need to select the generated format, which can be a table/line chart/bar chart. Here I choose a table, and then Postbot will display the instantiated table style of the response on the result page after the request.

This table instantiation of the response is achieved by generating a script under tests, and the specific script is as follows:

var template = `
<style type="text/css">
    .tftable {font-size:14px;color:#333333;width:100%;border-width: 1px;border-color: #87ceeb;border-collapse: collapse;}
    .tftable th {font-size:18px;background-color:#87ceeb;border-width: 1px;padding: 8px;border-style: solid;border-color: #87ceeb;text-align:left;}
    .tftable tr {background-color:#ffffff;}
    .tftable td {font-size:14px;border-width: 1px;padding: 8px;border-style: solid;border-color: #87ceeb;}
    .tftable tr:hover {background-color:#e0ffff;}
</style>

<table class="tftable" border="1">
    <tr>
        <th>Title</th>
        <th>Body</th>
        <th>User ID</th>
        <th>ID</th>
    </tr>
    <tr>
        <td>{{response.title}}</td>
        <td>{{response.body}}</td>
        <td>{{response.userId}}</td>
        <td>{{response.id}}</td>
    </tr>
</table>
`;

function constructVisualizerPayload() {
    return {response: pm.response.json()}
}
pm.visualizer.set(template, constructVisualizerPayload());

I haven’t found where the Visualize response feature helps API testing yet.

Using Save a field from response

After clicking Save a field from response in the Postbot interface, Postbot will generate a test script to store the id from the response as an environment variable, as follows:

// Stores the postId in an environment or global variable
var postId = pm.response.json().id;
pm.globals.set("postId", postId);

Then I clicked Postbot’s Save a field from response command again and found that Postbot still generated a test script to store the id from the response as an environment variable, instead of generating a test script to store other fields from the response as environment variables.

Trying Add documentation

After clicking the Add documentation command in the Postbot interface, Postbot will generate a very detailed interface document on the right side of the postman interface, as shown below.

The interface document describes very detailed interface-related information, such as interface request information, request field definitions, response examples, etc.

3. About Sharing My Thoughts

After trying the AI Assistant Postbot tool provided by postman, the functions provided by Postbot for adding test cases for request and response are quite convenient, and can quickly generate mostly usable interface response verification test scripts with high coverage. Although there are errors in the generated test scripts that need to be manually fixed, Postbot can quickly generate test scripts to improve the efficiency of interface testing.

In addition, Postbot’s interface documentation generation is also quite useful. After developers add the request in postman, Postbot can quickly generate relatively detailed interface documentation, which can improve R&D efficiency and interface document quality to some extent.

However, Postbot currently does not seem to support custom commands. I want to try to output different types of test cases for the demo interface through Postbot, such as empty request body interface test cases, illegal request body interface test cases, etc., but Postbot cannot give the correct response.

My blog post link：30 Days of AI in Testing Challenge: Day 14: Generate AI test code and share your experience | Nao's Blog

[vishnuar]

Hi all

Thanks for putting this topic. I started with the tool https://healenium.io/ which can self heal the object properties of the selenium automation. I could not complete its setup due to some technical error.

Then tried 2 tools in my Wishlist. The first one is Postbot and amazed with its results . It is very user friendly and prompts are automatically get displayed on the sidebar of UI.

The second one is CodiumAI, It is (Trial) integrated with VS code and ran against my github repo. It is super easy and felt that a must tool for developers. CodiumAI has a cool feature /improve, which can suggest the improvements for the selected code block

[anders62 (Anders Wallin)]

Sorry for late answer. I asked GPT to explain the things I did not understand,but mostly chatgpt produced pretty straightforward, readable code. NB what I did was not super advanced.

[blanchec (Blanche Carstens)]

Prompt: You are tasked to write functional automation code using Playwright - your test code is for JPetStore Demo

Output:
const { chromium } = require(‘playwright’);

(async () => {
// Launch browser
const browser = await chromium.launch({ headless: false });
const context = await browser.newContext();
const page = await context.newPage();

// Navigate to the website
await page.goto(‘JPetStore Demo’);

// Perform search
await page.fill(‘input[name=“keyword”]’, ‘dog’); // Replace ‘dog’ with your desired search term
await page.click(‘input[value=“search”]’);

// Select a pet
await page.click(‘a[href=“/actions/Catalog.action?viewItem=&itemId=EST-6”]’); // Replace with the actual link to the desired pet

// Add the pet to the cart
await page.click(‘input[value=“Add to Cart”]’);

// Verify if the pet is added successfully
const successMessage = await page.innerText(‘.messages’);
if (successMessage.includes(‘You have added’)) {
console.log(‘Pet added to cart successfully!’);
} else {
console.error(‘Failed to add pet to cart!’);
}

// Close the browser
await browser.close();
})();

further prompt: Can you generate more test scenarios

1. Sign In and Sign Out Scenario:

• Navigate to the website.
• Click on the “Sign In” button.
• Fill in the username and password fields.
• Click on the “Login” button.
• Verify that the user is successfully logged in.
• Click on the “Sign Out” button.
• Verify that the user is successfully logged out.

2. Adding and Removing Items from Cart:

• Navigate to the website.
• Search for a pet.
• Select a pet.
• Add the pet to the cart.
• Navigate to the cart page.
• Verify that the pet is in the cart.
• Remove the pet from the cart.
• Verify that the pet is removed from the cart.

3. Updating User Profile:

• Navigate to the website.
• Sign in with user credentials.
• Navigate to the user profile page.
• Update the user’s information (e.g., name, email, address).
• Save the changes.
• Verify that the changes are saved successfully.

4. Checking Order History:

• Navigate to the website.
• Sign in with user credentials.
• Navigate to the order history page.
• Verify that the user can view their previous orders.

5. Search Functionality:

• Navigate to the website.
• Perform searches with various search terms (e.g., different pet types, products).
• Verify that search results are displayed correctly.

6. Navigation Testing:

• Navigate to different pages of the website (e.g., Home, Catalog, About Us).
• Verify that navigation links work correctly and lead to the expected pages.

7. Checkout Process:

• Navigate to the website.
• Search for a pet.
• Select a pet.
• Add the pet to the cart.
• Proceed to checkout.
• Fill in shipping and payment information.
• Complete the checkout process.
• Verify that the order is placed successfully.

**Please write the playwright, automation test code for these tests.**const { chromium } = require(‘playwright’);

(async () => {
const browser = await chromium.launch({ headless: false });
const context = await browser.newContext();
const page = await context.newPage();

try {
// Sign In and Sign Out Scenario
await signInAndSignOut(page);

// Adding and Removing Items from Cart
await addAndRemoveItemsFromCart(page);

// Updating User Profile
await updateUserProfile(page);

// Checking Order History
await checkOrderHistory(page);

// Search Functionality
await searchFunctionality(page);

// Navigation Testing
await navigationTesting(page);

// Checkout Process
await checkoutProcess(page);

console.log("All tests completed successfully!");

} catch (error) {
console.error(“An error occurred:”, error);
} finally {
await browser.close();
}
})();

async function signInAndSignOut(page) {
await page.goto(‘JPetStore Demo’);

// Click on Sign In
await page.click(‘a[href=“/actions/Account.action?signonForm=”]’);

// Fill in login details
await page.fill(‘input[name=“username”]’, ‘your_username’);
await page.fill(‘input[name=“password”]’, ‘your_password’);

// Click on Login
await page.click(‘input[value=“Login”]’);

// Verify login success
await page.waitForSelector(‘a[href=“/actions/Account.action?signoff=”]’);

// Click on Sign Out
await page.click(‘a[href=“/actions/Account.action?signoff=”]’);

// Verify sign out success
await page.waitForSelector(‘a[href=“/actions/Account.action?signonForm=”]’);
}

async function addAndRemoveItemsFromCart(page) {
// Navigate to the website
await page.goto(‘JPetStore Demo’);

// Search for a pet
await page.fill(‘input[name=“keyword”]’, ‘dog’);
await page.click(‘input[value=“search”]’);

// Select a pet
await page.click(‘a[href=“/actions/Catalog.action?viewItem=&itemId=EST-6”]’);

// Add the pet to the cart
await page.click(‘input[value=“Add to Cart”]’);

// Verify if the pet is added successfully
await page.waitForSelector(‘.messages’);

// Navigate to the cart page
await page.click(‘a[href=“/actions/Cart.action”]’);

// Remove the pet from the cart
await page.click(‘input[value=“Remove”]’);

// Verify if the pet is removed successfully
await page.waitForSelector(‘.messages’);
}

async function updateUserProfile(page) {
// Implement update user profile scenario
}

async function checkOrderHistory(page) {
// Implement check order history scenario
}

async function searchFunctionality(page) {
// Implement search functionality scenario
}

async function navigationTesting(page) {
// Implement navigation testing scenario
}

async function checkoutProcess(page) {
// Implement checkout process scenario
}

• Share your thoughts:
  • Which tool you used and the automation you were trying to create. - chat gpt based, prompt was based off playwright and js
  • How you found working with the tool to build and update your automation. - Pretty easy and a good start but you still need to provide context like how to install playwright, installing node etc.
  • Did the code work the first time, or did you need further refinement? - Some of them actually worked first time, pretty rad
  • Did you find any limitations or frustrations with the tool?
    Yes:

Specific Website Interactions: Without direct access to the website’s structure and behavior, AI can only provide generalized automation code and scenarios.

Real-world scenarios can be more complex. Generating automation code for intricate scenarios requires a deep understanding of the website’s functionality which is where context is paramount.

Generating code that covers all possible edge cases and error scenarios requires thorough understanding and analysis of potential failure points.

Optimization and Best Practices: While the provided code demonstrates basic functionality, optimal automation code should adhere to best practices Generating highly optimized code requires expertise in software engineering principles and familiarity with the specific automation framework being used.

Suggestions:

Access to the Real Website and Documentation:

A feedback mechanism where users can provide real-time feedback on the generated code would help improve its accuracy and relevance.

Integration with testing tools or frameworks commonly used in the industry could allow for the generation of code that aligns with established testing practices and conventions.

Improved contextual understanding capabilities would enable AI to better grasp the nuances of specific testing scenarios and generate more contextually relevant automation code

[sargam (Sargam Panwar)]

Hi Testers!

This seems to be a very interesting challenge.
Tool Used: ‘Testim’
Site Used: https://automationintesting.online/
Automation Module: Form on the HomePage.
I tried automating the form present on the link above using the tool ‘testim’, it was very quick and helpful but I wasn’t sure how or even if I am able to get the actual automated scripts in any language. So still learning how to use this tool in a better way.

[futurelady25 (Amruta Bhokare)]

Tool Used : Testim
Site Used : Ministry of Testing
Automated Login

This tool has a completely different approach of automation. It automates screens, unless I am missing anything.

[mirekdlugosz (Mirek Długosz)]

I’m a bit late to the party

I decided to try Bing Copilot, and I gave it a task that I felt is relatively self-contained and not too complex, but at the same time, unlikely to be encountered before. I maintain an RSS reader app where users can change focused entry with a keyboard; once entry is focused, you can press another key to open it or mark as read. This is similar to email inbox. I asked Copilot to generate some test code for the feature when using Playwright with Python. The entire transcript is below.

In first response, it ignored anything specific to my problem, instead focusing on Python, Playwright and simulating keyboard pressing.

When I asked for corrections, it generated some code, but there were no assertions and it confused what I meant by focus with website form focus event.

Overall, I can’t give it a passing grade. I guess the generic pointers could be useful if I didn’t know much about Python or Playwright - I could use these instead of trying to find the same information in documentation. But I still would need to assemble these parts myself.

To be fair, Bing Copilot is generic chat, not specialized test or code generation tool.

---

Q: I have a web application with a number of elements in a list, like that:

<ul class="entries-list">
<li class="entries-list__item" data-vue-index="0"><div class="entry"><div class="entry__meta">...</div></div></li>
<li class="entries-list__item" data-vue-index="0"><div class="entry"><div class="entry__meta">...</div></div></li>
</ul>

User can use the keyboard to change which list element is currently focused. When element is focused, it will look like that:

<li class="entries-list__item" data-vue-index="0"><div class="entry entry--focused"><div class="entry__meta">...</div></div></li>

Pressing B or K on keyboard will focus previous element, pressing N or J will focus next element. There is no cycling - when you try to go from last to next, you stay on last.

Can you generate me some Python Playwright code to test this feature?

A: Certainly! To test keyboard navigation in your web application list using Python and Playwright, you can follow the steps below. I’ll provide a sample code snippet for each relevant action:

1. Initializing Playwright and Launching a Browser:
   First, make sure you have Playwright for Python installed (pip install playwright). Then, create a structured project directory layout. Here’s a common structure:

   project_name/
   ├── tests/
   │   ├── __init__.py
   │   ├── test_my_feature.py
   │   └── test_another_feature.py
   

   In your test script (test_my_feature.py or similar), you can initialize Playwright and launch a browser:

   from playwright.sync_api import sync_playwright
   
   with sync_playwright() as p:
       browser = p.chromium.launch(headless=False)
       page = browser.new_page()
   
       # Your automation code here
   
       browser.close()
   

2. Basic Navigation:
   Navigate to a URL, reload the page, and move back and forward in history:

   # Navigate to a URL
   page.goto('https://example.com')
   
   # Reload the page
   page.reload()
   
   # Go back and forward in history
   page.goBack()
   page.goForward()
   

3. Interacting with Elements:
   Perform actions like clicking an element, typing into an input field, and selecting an option in a dropdown:

   # Click an element
   page.click('button')
   
   # Type text into an input field
   page.type('input[name="username"]', 'your_username')
   
   # Press Enter key
   page.press('input[name="password"]', 'Enter')
   
   # Select an option in a dropdown
   page.select_option('select', label='Option 1')
   
   # Wait for elements to appear or become visible
   page.wait_for_selector('div#my-element', state='visible')
   
   # Check if an element exists
   assert page.locator("button").is_visible()
   
   # Check the page title
   assert page.title() == "Expected Title"
   

4. Keyboard Input:
   Simulate keyboard input (e.g., pressing keys):

   # Type text
   page.type('input', 'Hello, Playwright!')
   
   # Press and release keyboard keys
   page.keyboard.press('Enter')
   

Remember to adapt the above code snippets to your specific web application and test scenarios. Happy testing!

Q: This is some generic code that has barely anything to do with the problem I have provided. Can you give me the actual solution? Feel free to skip the parts where you do not have enough information to provide code, but mark them so I can fill them up myself later.

A: I apologize for the oversight in my initial response. Let’s focus on the specific problem you’ve described: testing keyboard navigation within a list of elements in your web application. Below, I’ll provide a more relevant Python Playwright code snippet to address this issue. You can fill in the missing details specific to your application:

from playwright.sync_api import sync_playwright

def test_keyboard_navigation():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        page = browser.new_page()

        # Navigate to your web application URL
        page.goto('https://your-web-app.com')

        # Assuming you have a list of elements with class "entries-list__item"
        list_elements = page.query_selector_all('.entries-list__item')

        # Simulate keyboard navigation (e.g., pressing 'N' for next, 'B' for previous)
        for i in range(len(list_elements)):
            # Focus the current element
            list_elements[i].focus()

            # Your application-specific logic here:
            # - Verify that the focused element has the expected class (e.g., "entry--focused")
            # - Perform any other checks or interactions you need

            # Example: Press 'N' to move to the next element
            page.keyboard.press('N')

        browser.close()

# Run the test
test_keyboard_navigation()

Please replace the following placeholders with your actual application details:

1. https://your-web-app.com: Replace with the URL of your web application.
2. .entries-list__item: Replace with the correct CSS selector for your list elements.
3. Customize the logic inside the loop to verify the focused element and perform any other necessary checks.

Feel free to adapt this code snippet to match your specific use case. If you have additional requirements or need further assistance, feel free to ask!

[ipedevi (Ivan del Viso)]

Hi,
I know I’m out of time but I think it could be interesting if you read it. I was using the ChatGPT and Gemini, not only to help me in the script(QA framework) creation, also to improve it.
Also as example, during the last 2 weeks I was using Gemini to help me in the language translation, from ruby to kotlin (I never work with kotlin) and the final kotlin framework had more quality than the old one (created by us in ruby).
The best tip I can give you is to retry… every time I had an error in the code obtained with AI, I asked again explaining the AI that the previous code has errors and listing them… then, the AI, try to fix them and explain the why. It help me a lot to understand what happen in a languaje that I don’t have knowledge.

Best Regards,
Iván