ChatGPT has recently gotten a lot of attention.
It’s fascinating, even after a few rounds of exploring its capabilities.
It’s impressive and also kinda scary! As this type of technology becomes ubiquitous and perhaps pervasive, how might technology like ChatGPT impact your testing efforts, test data creation, test reporting and more?
Let’s discuss the good side, the bad side and everywhere in between.
I see @bethtesterleeds recently posted some thoughts on her blog. OpenAI : The-Tool-That-Must-Not-Be-Named ⚡ – Beth The Tester's Tales
I believe the underlying AI models (GPT3, or Bloom an opensource alternative) which are used to build tools like ChatGPT would have the larger impact, that comes in waves, and become integrated across various products.
Test data is the most obvious use case, followed by test script writing and editing.
For example, it is able to translate selenium code from java to javascript
It may not be 100% accurate, and it may need. some human help to fix and iterate… but it did the heavy lifting.
Getting the AI to accurately generate test scripts, and edit them is something I am working on within my company. And from my point of view, is my end goal, where you basically chat back and forth with a bot on how to write and improve the test scripts. (Like the above code conversion, but on your entire test code)
Its role will be more of an assistant alongside us human testers, as from my current experiments, it is very flawed in various ways. And does not understand some wierd things, especially when certain bugs are features. Basically the more subjective feedback parts.
To quote a co-worker, is like hiring your own intern. They will make mistakes, but they are still useful.
Also i do agree with @bethtesterleeds - this technology does have serious ethical implementation, outside the testing space, when mixed into the “privacy” or “right of an artist” areas. Though that might be another topic all together (calling it a problem, might be an understatement).
PS: it is very difficult to predict new frontier tech changes, so the above is just my guess, I may end up being completely wrong - im also a startup founder who works on test automation tools, and AI, so I do have a conflict of interest in this topic.
<cracks knuckles, grins> I cannot wait to see what horribly twisted, broken code it comes up with.
Nice one for asking an excellent question. 
I asked ChatGPT to write some selenium code to test Google’s search page. It came back with a pretty simple example (I am actually not sure it would have worked).
I then asked it to write a script to test SAP’s VA01 transaction (likely the single most frequently used transaction). It replied (basically) “what’s a VA01?”
This type of AI is great at culling down large volumes of information, but if there aren’t many Selenium VA01 examples floating around the net, it is not going to be able to help. It can not “figure it out”.
oh it will be very broken all right 50% of the time.
but i do like it that way still, as I always find it easier to edit test then to write it from scratch.
I would consider it more a muse than a teacher or librarian.
Don’t consider the information you get to be correct, but use it to get new ideas by playing around.
I’d second that! It’s impressive how I can ask pretty obscure questions where ChatGPT must synthesise information from one or two topics and give an overall response. For example, I asked it which of Star Trek Captain Jean Luc Picard or Captain Kirk’s styles would be appropriate to various situations. Each time I got a pretty generic answer about their styles and how each has strengths and weaknesses. I was impressed with its understanding of the question and its knowledge of the characters, but realistically it didn’t give any depth of information.
It’ll be interesting to see how it develops over time, but for now it seems to be a slightly more advanced search engine than a truly useful tool.
IMHO YMMV as always 
I recently tried ChatGPT and wrote some ways it might assist in software testing. USING CHATGPT TO ASSIST TESTING | VIV RICHARDS
I just posted a video about openAI and chatGPT on my channel.
I think that AI will be another “tool” that we as testers should know and use. We should also know that the results will not always be correct and that we should question them. There is a great example in the video I created. Also kudos to Joe Colantonio for his video on chatGPT. Find the link in the description of the video.
Its great to see what others are doing with ChatGPT. Ive tried many of the same things recently with slightly different results.
I think though the scary thing that this video and ChatGPT highlights from things ive tried is that the output generated in the wrong hands from ChatGPT can show ‘easy ways’ to introduce the wrong thing’s or introduce bad habits.
I think its a great tool to explore and help generate ideas but those newer maybe to automation for example might start applying the output directly without any critical thinking and start testing things at the wrong levels for the wrong reasons…
For example from experience you’d perhaps know that the cucumber scenarios shown in Daniels video were imperative, not declarative but would still be valuable to help maybe generate scenario ideas and not implement as a direct copy and paste. In the wrong hands without a proper understanding I worry many will just use the suggested code… in the video it also generated many scenarios which in my opinion would be absolute overkill at the UI level. Many of which could be better tested at a lower level for example i.e. testing Views perhaps for logic checks for a login.
Still interesting though to keep an eye on how people are using this and seeing all the ideas.
Interesting times ahead.
Hi @vivrichards,
agree with you. I think chatGPT is a great start into a new AI era. One should take the results with a grain of salt, especially new people in the industry might get the wrong idea. However, as I mentioned in the video, it’s great for generating some ideas, to get inspirations and many other things.
I am really looking forward to the AI topic and what it will bring us in the testing community.
It’s definitely a tool to be careful with. I’ve been using it recently to help me with copywriting. Where I’m asking it for a bunch of options, versus a scenario where I’m telling it to give me the answer.
It has huge potential in keeping people moving IMO, it’s an on-demand rubber duck
I was trying to use it to learn a test frameworks
(K6)
Its almost a bit too positive and has a habit of making things up.
Whats weird is that if you tell it that was wrong and that module you imported doesnt actually exist it’ll confess to its mistake and make something else up entirely.
Its got a lot of backing now Microsoft have pumped an obscene amount of money into it so they’ll hopefully tidy up the problems with it.
@dilkushi recently joined this community. They’ve run some experiments with ChatGPT and shared their thoughts in a recent blog post.
I used Notion’s AI bot to help me prepare for a presentation about continuous testing with regards to shift left and shift right. I didn’t use the answer verbatim, but it helped give me some ideas and snippets to use during the presentation.
Like others have said, it will become an ever more useful tool and shape our roles in different ways.
