šŸ¤– Day 18: Share your greatest frustration with AI in Testing

30 Days of Testing
, ,
21

My Day 18 Task

My Concerns and Challenges Using AI Tools for Testing Activities

Data Privacy and Security Concerns with AI Tools

In the challenges of the past several days, Iā€™ve mentioned my concerns about data privacy and security regarding AI tools. Due to these concerns, Iā€™ve been cautious about using AI tools for testing activities, carefully filtering out any context related to the project. This cautious approach makes the process more difficult and results in some discrepancies between the outcomes provided by the AI tools and the expected results. Consequently, itā€™s challenging to directly apply these results to current project testing work, which hinders direct and real improvements in testing efficiency.

Functional Limitations of AI Tools

During the recent days of the AI testing challenge, Iā€™ve experimented with various AI testing tools, including Applitools Eyes, Katalon, Testim, and Postmanā€™s API testing AI assistant, Postbot. While most toolsā€™ AI features can indeed enhance testing efficiency, the improvement is still limited. There is a significant discrepancy between the AI testing functionality and the descriptions in official promotional materials. It feels like the hype is greater than the actual performance.

Learning Curve Challenges with AI Tools

Here, Iā€™d like to discuss the comprehension capabilities of different large AI models, such as ChatGPT-3.5, ChatGPT-4, Gemini Pro, and Claude 3. The results produced by these different AI models for the same prompts can vary, requiring time to adapt when applying these AI models to daily testing activities. It involves comparing and learning about different AI model tools to determine which testing activities are better suited for which AI models.

Difficulty in Accessing AI Tools

For many IT professionals outside of China, accessing the latest AI testing tools and large AI model tools is relatively straightforward. However, for IT personnel in mainland China, it is exceptionally difficult to access these tools. The first hurdle often encountered is in applying for an account and the subsequent challenge of paying for the service.

My blog post link:30 Days of AI in Testing Challenge: Day 18: Share your greatest frustration with AI in Testing | Nao's Blog

2 Likes
22

My challenges with AI testing

  1. Trust: These AI tools spit some good test cases and provide adequate test coverage. But me personally, I havenā€™t built up enough trust to believe in the results blindly. I still do some due diligence before using these test cases.

  2. Ethical and Regulatory Concerns: Testing AI systems involves ensuring compliance with ethical guidelines and regulatory frameworks. Ensuring fairness, transparency, and accountability in AI systems is a challenge.

3 Likes
23

Hi there :wave:

Data Privacy Worries: Not sure how these dataā€™s are processed. If youā€™re working with Sensitive dataā€™s then should be very vigilant.

Thanks
Vishnu

1 Like
24

Greetings Everyone!
Following are my greatest frustrations with AI in testing.

  1. Lack of tutorials/blogs: When Working on my previous task, it was quite difficult for me to get practical knowledge of the tool ā€˜mablā€™ since YouTube didnā€™t provide much content about it.
  2. The AI tools are paid: As soon as I get hands-on experience with these tools, their trial version expires, and then after learning I am back to zero.
  3. The auto-generated automation skills: These AI tools provide us the test scripts that make the ā€˜Manual testersā€™ not work towards actually learning the ā€˜automation testingā€™ and making them more dependent on these AI tools for even writing basic test cases and other programming test scripts.
  4. In the earlier blogs, I asked ā€˜ChatGPTā€™ to provide me with some blogs for the AI test cases but it gave me a couple of links with the attached URLs, but the URLs didnā€™t work, thatā€™s also because I have the free trial version of ChatGPT.
  5. Lastly, yes it gives me a sense of worriedness that AI might automate much of the ā€˜Manual Testingā€™ tasks thus making them easily replaceable.
2 Likes