đŸ€– Day 4: Watch the AMA on Artificial Intelligence in Testing and share your key takeaway

📆 30 Days of Testing
, , ,
21

Thanks @elsnoman for participating in that AMA!

As someone who evaluates the quality of AI solutions, two particular topics in the AMA got my attention:

  1. How can you assess confidence your users have in your AI powered software?

I interpret this as “how do I know that users agree with the AI’s predictions?” or “how do I know that my AI is giving the right answers?”

Yes, the key is observability – but that is a bit of a broad statement. The detail that’s missing is that you need to find a way to associate consequential user activity to the AI prediction that spurred it.

“Like”, “dislike”, or “pick the best answer” interactions like Carlos described are very obvious ways to get feedback, but I think their value is shallow and frankly suspect. It’s akin to asking the user for their opinion, but as the old saying goes, “actions speak louder than words.”

What’s better is to find a way to tie AI predictions to user actions that have consequence. If a user makes “important” decisions based on predictions, then I would feel a bit more confident about the quality of the AI predictions. Same if the AI predictions align with user actions (where perhaps there is no direct tie between prediction and action).

It reminds me a little of playing poker with friends where no money is involved. Risky betting has no consequences – you can just grab more chips. Players have no skin in the game.

  1. Train your models using high-quality data (try to avoid bias)

Reminder of the old computer science saying: GIGO - garbage in, garbage out

Your model is only as good as your training data. If your data has mistakes, has inherent bias, or doesn’t really match the data that your users work with, then it will not give very good/relevant predictions.

Consider the concept of verification vs. validation: building the thing right vs. building the right thing

A model trained on poor/biased/inappropriate data may get very good statistical results for accuracy, F1, etc. but it may not actually satisfactorily “built for purpose”.

8 Likes
22

Amazing Session about AI in Testing by Carlos

A couple of things that we need to consider are:

  1. Before using any tool for AI, just go through their terms because confidentiality and security can be at risk.

  2. AI = ML is all about training models, and your testing will depend on the training.

  3. AI can help you for accessibility testing but can’t fully replace the human interaction :thinking: . this is questionable need to think

If you’re newcomer just Play around with Chat GPT and find out how other people are using this to improve the testing.

Has anyone of build a modal for testing your APP/Web APP? :robot:

4 Likes
23

It’s Xmind!

Here is my video explaining it:

Inspiration Matters | Software Engineering Management with Xmind | Webinar with Rahul Parwal (youtube.com)

3 Likes
24

Great AMA. Hadn’t heard of Carlos before but will definitely be following him.

For me, I really liked his response to the question on using AI in our day to day testing work. His candid response was really refreshing, highlighting that without having it integrated into your tool or having deep knowledge of your context / data is usefulness will be limited. This is a stark difference to many in AI who sell it as a silver bullet.

5 Likes
25

Watched the whole thing, and picked up my highlights there:

  • <3 <3 <3 the pronouns and visually impaired intro Carlos did, for that alone the video was worth watching.
  • When asked about AI learning in production, there was a nice twist to the story shared: the AI in the story was not learning in production, and there were many hints that this is not so common that it would be learning from what you input (it needs retraining so there may be data collection) but about it NOT LEARNING about change of market and thus doing poorly at the job it was assigned.
  • Liked the correct - valuable - safe trio of assessing things that was almost like a theme through answers.
  • Unsure of the idea of AI as human replacement for junior testers when feeling ashamed of asking of all the terminology the industry is riddled with. Sure, we have limited time and any conversations can drive your learning forward. But you might also get stuck on a loop a human would pop you out of.
  • Shivers on the idea that our future may be analyzing results from AI trying to make sense what we should react on.
5 Likes
26

Hey! Just finished the video, and wow, it’s pretty mind-blowing. Here’s what stuck with me:

Making sure AI gets the right context is super crucial. The real estate company example was crazy – shows how things can go wrong if the info you give to AI isn’t spot-on. So, accuracy in giving details is a big deal.

Learning how AI can boost our knowledge is cool. I never thought about it like that before. It’s not just a tool; it’s like a helper that makes us smarter.

Keeping an eye on things all the time is super important with AI. Any little change can mess things up, so we gotta stay alert. It’s like AI needs a babysitter to make sure everything’s on track.

Talking about laws for data is a big deal too. You need to be a data law expert or have one on your team to avoid problems. But hey, what if AI could help out with that too? Interesting idea, right? Using AI not just for predictions but also to keep things legal.

So, the video got me thinking about how crucial it is to feed AI the right info, how it can be a knowledge booster, why we need to watch over it all the time, and even if it could help with legal stuff.

5 Likes
27

After watching the “Ask Me Anything on Artificial Intelligence in Testing” with Carlos, I have learned several key takeaways that have had a significant impact on my understanding of AI in testing. Here are the main points that stood out to me:

  1. The Importance of Context: The session emphasized the importance of context when it comes to using AI tools for testing. Generic AI tools like ChatGPT may not be very useful in testing as they lack the specific context of the situation. Instead, specialized AI tools that have more context about the testing environment are more effective.
  2. The Role of AI in Testing: AI can help with structured work, creating scenarios, and writing code. However, it is not as good at analytical techniques that require human judgment. AI is better at automation and can assist with tasks like Test-Driven Development (TDD) and generating code.
  3. Security and Confidentiality: It is essential to be mindful of the data we share with AI tools and ensure that we only share what is necessary to get the answer we want. This includes being aware of the legal and software laws related to the data we share.
  4. The Future of Testing: The session suggests that the future will see the development of specialized AI tools for testing, which will be more context-aware and better suited for specific testing tasks.
  5. Testing AI: Testing AI models involves ensuring data quality and relevance, as the AI model is only as good as the data it’s trained on. It also involves splitting data into training, validation, and testing sets to evaluate the model’s performance.
  6. Ethical Considerations: The session highlights the need to guard the quality of AI that changes how it behaves in production, known as data drift. This requires monitoring the AI model’s behavior and ensuring that it does not deviate from its intended purpose.

In conclusion, the session has provided valuable insights into the role of AI in testing, the importance of context, and the ethical considerations that need to be addressed when working with AI tools.

5 Likes
28

I was testing AI detection tools with safe samples and zeroGPT verdicts “:robot: Your content appears to be crafted by GPT technology.” claiming 94.6% of your post is written by AI. I am curious: is it? I also tested my own response and my verdict is “:writing_hand: Your test is authentically human-written” 100% and I am wondering how well these patterns really are recognizable.

You ended up as test case since you were last post when I looked, and since I did not want to upload data I really wanted to check - cover letters for a job posting I should work on.

3 Likes
29

The real life example was very compelling and showcases the current limitations.

I also identified with the ideas around context. This is big whenever using an LLM. The most success I have had is when I give it a role and a specific task to complete as that role. This is really helpful for me to put things into a QA context by having it explained as an “expert”.

4 Likes
30

I’m limited with how much I can reply, so I have to keep it shorter.

Great takeaways, everyone! Context is the name of the game right now in the AI world. You see new models that allow over one million tokens so you can provide more context (see Google’s latest Gemini models)!

This goes hand-in-hand with security and biases, because now that you can send a very large code repo, you want to make sure you’re not sending anything that can pose risks to you or your company.

I’m excited to see new tools specifically for testing AI systems and for AI in Testing!

Ethics is such a massive part when talking about testing AI systems and using AI systems! I only touched on it briefly here, but there are some great resources out there for anyone wanting to dive into more!

Check out this free course from Kaggle University: Learn Intro to AI Ethics Tutorials | Kaggle

8 Likes
31

Hi everyone !

My key take away from the video will be the importance of providing context to the “AI”. This is not something I have been trying to do yet, but I definitely should.

I have also been interested in the open source/free tools & sites pointers such as huggingface.co, reportportal.io and starcoder. This gives me plenty of stuff to explore.

Last but not least, thank you Maaret for the pointer on ZeroGPT. Did not know that tool and I’m pretty sure I’m gonna make a lot of use of it :slight_smile:

4 Likes
32

I’ve worked with many junior testers now that have been using AI tools like Gemini and ChatGPT. It’s easy to trust what the AI responds back because it usually does it very confidently, so that’s where the “risk” can come in, but after a few pairing sessions the tester starts to understand the nuances of how to ask certain questions and even bring it to others (ie senior testers) for further discussion and clarity!

Hopefully, knowing the weaknesses and risks of the AI tools people use will prevent a lot of the issues that I brought up like the real estate scenario and help testers approach AI with more confidence :smile:

3 Likes
33

Additional Resources and Examples

3 Likes
34

I watched the video by scanning the questions. Here are my takeaways:

  • Numerous AI tools exist to aid in testing, automation, coding, reporting, and documentation. However, human oversight remains crucial to guarantee accurate results from AI tools.
  • Data is paramount for AI/ML. Biases and ethical concerns are present, and data-centric testing is essential.
  • Prompting is crucial for optimizing AI tools’ performance and accuracy.
  • AI/ML tools cater to testers, automation engineers, software engineers, and various roles at all levels.

Questions:

  • How can we identify or select the most suitable AI/ML tools to address our specific needs?
  • How can we leverage AI/ML tools to test for bias within AI/ML systems?
  • Considering our primary focus on Generative AI (GenAI) and Large Language Models (LLMs), are there any reputable open-source AI tools available for end-to-end testing of GenAI/LLMs (excluding model testing)?
4 Likes
35

A brilliant, incredibly thought provoking video.

My key take away was from the question about what a tester’s role will look like in ten year’s time. Carlos predicted that the automation skills that we currently value so highly will become irrelevant and will be replaced by analysis and critical thinking skills. As a quality team leader I’m going to seriously re-consider my own personal development goals in this light as well as those of all of my team members. I’ve already encouraged them to take part in 30 Days of AI in Testing.

4 Likes
36

Excellent AMA, loved watching it and listen to Carlos, who’s clearly knowledgeable and passionate about both ML/AI and testing.

My main take aways:

  • Context is all. Either start training your own models, or get really good at prompt engineering if you want to take full advantage of GPTs to assist your work.
  • When dealing with a stochastic parrot it’s good to keep in mind that there is no creativity. This is what humans bring to the table.
  • Using AI tools to help create automated test cases will remove barriers for those who want to do the work of learning how to use the tool. The danger here is that the output still needs validating, so we’ll still need experts who understand code well enough to do that.
  • Pattern matching is something humans do extremely well, presumably because it’s an evolutionary advantage to be able to quickly tell when something’s “off”. Unsurprisingly that’s also what the machines we built excel at.

Random anecdote: It’s been less than 30 years since a computer was able to beat the best human chess player in the game. We’re in for one hell of a ride.

4 Likes
37

I reviewed the whole video with Carlos Kidman about AI in Testing. Many raised question are definitely relevant to today and are now widely discussed in public.

It was interesting to hear opinion about future role of testers and I agree, that same testing part will be delegated to AI and testers will have to work more with assisting to AI, organizing and analyzing AI generated data.

Also it was useful to listen about AI role in usability, accessibility and UX testing, security, ethical, confidential issues, related with AI, AI integration to testing tools, like Postman and etc.

4 Likes
38

My Key take aways from AMA on Artificial Intelligence in Testing

context, context, context and the importance of context to get proper answers and avoid hallucinating.
Vendors are moving in this direction to train their AI within context and not using the whole internet

To get a deeper understanding how AI/LLM works you can use a free and open source modell and train with your own datasets
“And at that point, you can either fine tune it further to give more examples that are specific to your context
or if you’re dealing with one of these bigger models like GPT.”

Carlos thoughts on how testing will envolve and which testing job will vanish

:grinning:

5 Likes
39

Hello everyone,

Just watched the Q&A session Ask Me Anything: Artificial Intelligence in | Ministry of Testing

And found the following insights

  • :bulb: Power of AI Tools: The speaker emphasized the importance and power of AI tools in testing, suggesting that people might underestimate their capabilities. Encouraged experimenting and exploring tools like prompt engineering to understand their potential.
  • :lock: Security and Confidentiality in AI Models: In response to a question about ensuring security and confidentiality of data processed by AI models in the Cloud, the speaker highlighted the importance of trusting vendors and their terms of service. Suggested that if trust is an issue, organizations may need to build and host their own models.
  • :robot: Difference between Machine Learning and AI: Explained the distinction between machine learning and AI. Machine learning involves algorithms and pattern matching, while AI involves intelligence and reasoning, finding meaning in data.
  • :crystal_ball: Future of Testing with AI: Predicted that in the next 10 years, software testing will see automation using AI becoming more prevalent. However, emphasized the continued need for human testers for analysis, risk assessment, and critical thinking.
  • :shield: Ethical Use of AI in Testing: Discussed ethical considerations in using AI for testing, including data privacy laws and potential biases in AI models. Emphasized the importance of diverse perspectives and testing for biases in AI models.
  • :robot: Testing Tasks Suited for AI: Highlighted that AI is well-suited for tasks like generating test cases, automation, and managing test artifacts. However, cautioned that context and understanding are crucial, and human testers still play a vital role in analysis and decision-making.
  • :eye_in_speech_bubble: Guarding Quality of AI: Emphasized the importance of observability and monitoring in ensuring the quality of AI models over time. Gave examples of AI models performing well initially but failing to adapt to changing circumstances, leading to significant consequences.
  • :globe_with_meridians: AI and Diversity in Data: Addressed the risk of bias in AI models due to the lack of diversity in training data. Highlighted the need for diverse perspectives and data to mitigate bias and ensure fairness in AI systems.

Overall, the Q&A session covered a range of topics related to AI in testing, including security, ethics, diversity in data, and the role of human testers alongside AI tools.

Thank you

5 Likes
40

Great thanks for posting this AMA. Many insightful points were discussed, particularly the question about the future of testers and testing in the age of AI, looking ahead 10 years.

5 Likes