PinkLion at TestBash San Francisco_answers to submitted questions

PinkLion answers the TestBash questions asked after Jennifer Bonine’s session in San Francisco!

  1. How do I get started?
    ANSWER: The first step should be to look for education that focuses on AI’s applicability to software testing. Start small. Learn the basic machine learning concepts.

  2. What open source tools are available to begin using AI?
    ANSWER: Tensor Flow, Google Cloud Auto ML, Google Tools, Keras, Scikit-Learn, Theano, Torch

  3. What are the AI enterprise solutions available?
    ANSWER:, Retest, MoQuality, Mabl, Appvance, Applitools, Functionize, Testim, bitbar

  4. What languages do you use?
    ANSWER: Python, Java, JavaScript

  5. How do you know what AI algorithms to use for testing different types of software applications: web, real-time, mobile, database, etc?
    ANSWER: Algorithms are not really dependent upon what you’re testing – how you’re testing and what you’re trying to extract are the questions to ask yourself in determining which AI algorithms to apply. ML is classified into four types of groups: supervised, unsupervised, semi-supervised, and reinforcement – each of these groups have strengths and weaknesses.

  6. What are the risks of training data falling out of sync with the product and is there a mitigation plan?
    ANSWER: The risk is that the AI will be less able to recognize elements in applications and web pages and not understand new flows – how to get to data that you want to test. In most cases (notwithstanding a complete application rewrite), a simple crawl and small manual effort should be enough to get your bots back up to speed.

  7. How do you specify the rules - does it generate the rules by itself too? I upload a spreadsheet - how does it know what questions to answer?
    ANSWER: AI isn’t sentient; you need to train the bots on what the rules and business logic are. It cannot read and understand a rule without some form of training.

  8. How can AI be used to test the data models created by AI?
    ANSWER: One potential way to do this would be to train a second instance of the bots to test the data models produced by the first.

  9. Would a company without existing automation be recommended to combine AI with a click/record-type of automation tool instead of something like implementing Selenium Webdriver?
    ANSWER: We would recommend NOT using a record and replay type automation tool in conjunction with AI. AI excels at user interface, which is the only thing that record and replay can do – and it is brittle at doing so. Using Selenium Webdriver is a precision tool that has a distinct place in complementing the use of AI.

  10. Is AI testing available for desktop C# applications?
    ANSWER: Yes – AI can be used to test anything with a user interface.

  11. I focus on API testing in my role, how can AI enhance my processes?
    ANSWER: AI for API workflows is in pilot and will be more advanced in the next 6-9 months

  12. What is a context for your AI driven test automation? Would it work for ERP?
    ANSWER: AI excels at performing repetitive UI validations in a manner that is not impacted by changes to the application. This is the optimal context in which to apply AI based application testing.

  13. How can we use AI in a voice platform?
    ANSWER: Tools are coming, but we’re currently not aware of a specific and reliable way to apply AI today to test voice platforms.

  14. Are there good AI testing forums to get help?

  15. More information on how you use this for performance testing…
    ANSWER: The solution captures performance related metrics for all of the web pages or applications that it crawls. We do not recommend using the tool solely for performance testing (especially stress or load tests), but you “automatically” get a significant amount of page load performance metrics by virtue of using the solution to test or even simply using it in explore mode.

  16. How have you found AI to work for testing web apps with lots of fields and workflows, like legacy insurance claims and underwriting?
    ANSWER: This is a perfect use-case for AI.

  17. Given that we are training this AI with machine learning, why should we think that this won’t provide building blocks to get rid of manual testers altogether?
    ANSWER: People are needed to train the bots, decompose requirements, etc. The role will shift from executing the tests to designing them.

  18. Would you recommend exploring something like’s Appium plugin to start “rolling your own” AI for UI automation, or are we in a place where it’s better to pay a vendor?

ANSWER: The plugin’s “roll your own” capability is limited. Its intended purpose is to give someone a chance to dabble in AI, not necessarily build out an AI infrastructure.

  1. Are there any products with good support for tablets?
    ANSWER: We support tablets

  2. Is there potential for generating test data?
    ANSWER: We are not currently doing this today but the founder of is working on creating a platform for solving this kind of problem –

  3. How do I implement/how do I start an infrastructure/how do I get started?
    ANSWER: The easiest way is to call PinkLion for a Solution Design Review.

  4. How does AI help replace UI tests traditionally performed with Selenium? Functional tests with asserts?
    ANSWER: It is more of an enhancement than a replacement. AI helps to create non-brittle tests that are not impacted by application or UI changes.

  5. What about non web programs?
    ANSWER: Anything with a UI!

  6. Are there legal rules around using AI to test your competitor’s site?
    ANSWER: Not if it’s publicly accessible. Where you run into trouble is if you’re trying to log into authenticated areas of the website or if there is obvious obfuscation or rights management in place designed to prevent crawling.

  7. What is the cost of getting AI testing up and running?
    ANSWER: It can vary widely based on what you want the AI to do and how much time is spent training the bots.