30 Days of Automation in Testing Day 4: What Types of Testing Can Automation Support You With?

(Heather) #1

Today’s challenge in the 30 days of automation in testing challenge is:

What types of testing can automation support you with? Share an example.

It’s been a while since I have used automation but here goes. For me, I see automation as a helper. I can use it at the API layer because let’s face it, I can’t be checking all of that manually :sweat_smile: So I can use tools like Postman there.

At the UI layer, I use Selenium to automate the boring/mundane/repetitive tasks. This then frees me up to explore the application more and use my time better elsewhere.

I know this isn’t the strongest answer, as I said, it’s been a while since I was automating.

What do you think? What types of testing can automation support you with?

30 Days of Automation in Testing Day 4: What types of testing can automation support you with? Share an example
(Kris) #2

For me, I sometimes create small Python scripts for those tasks that could be a huge time sink. Like comparing the data from three differently ordered documents of around 2000 rows. Or finding all of the valid possible test paths and therefore test cases of a phone IVR. Or even creating a Powershell script to run on my computer and shutdown orphaned processes from Selenium.
A large amount of my day to day work is now also taken up with creating and assisting the rest of the team with learning and using test automation; UI using Selenium and Node JS, Mocha, Chai, WebdriverIO; REST API automated testing also using Node; and SOAP API automated testing using Python.

(Magda) #3

Apart from the point raised by the posters above, the following come to mind:

  • data migration testing (comparing large sets of data, driving the application against the old and new DB along the same path)
  • data-driven and property-based testing (including input into exploratory testing by pinpointing “interesting” results)
  • repeatable flows at any level of tests in general
  • load testing
  • sanity checks (e.g. after installation)
  • cross-browser testing
  • chaos monkey testing

Additionally, automation can support testing via:

  • test data generators
  • drivers, mocks and simulators
  • environment setup and teardown
  • report generators (including e.g. screenshots to be evaluated by a human later)

When thinking of shifting right, I guess logging and monitoring with its alerts and notifications could also be understood as part of automation.

(Eman) #4

I agree to the last comment as I have read it only, but I didn’t understand “When thinking of shifting right, I guess logging and monitoring with its alerts and notifications could also be understood as part of automation.”
Kindly Maos explain.

(Paul) #5

Automation is very useful, and underrated by most, in creating test data for functional and fuzz testing. You can also integrate it into other tools so that they can be loaded in en-masse or non-technical people can use it.

I have a project that uses the Faker.NET framework to generate reams of test data in Excel if anyone is interested.

(Magda) #6

Hi @eman, shifting left or right means going beyond the traditional “testing phase” in the work cycle to contribute as tester. Imagine a waterfall diagram:

analysis — design — development — testing — deployment into production — operations/support

On the left, it can include pairing with developers and participating in events like three amigos, where the PO/analyst, the developer and the tester discuss a user story or requirement to catch questions and problems early and get a common understanding of what needs to be done.

On the right, it can include working together with support personnel or using monitoring of the production deployment to guide risk analysis and testing decisions.

Especially in a continuous delivery context but also in more traditional settings it’s helpful to look beyond the edge of the own plate to look for a solid foundation for these decisions - and it doesn’t get much more solid than looking what actually happens in live usage of your software.

I hope this helps :slight_smile:

(Trung) #7

30 Days of Automation in Testing Day 4: What Types of Testing Can Automation Support You With?

Below are some types of testing I applied automation during 5 years working on automation testing:

Regression Testing: For re-testing preexisting application functions that
are being carried forward to new versions (usually the majority, unless
app is brand new)
• Smoke Testing: For getting a quick high-level assessment on the
quality of a build and making go / no-go decision on deeper testing
• Static & Repetitive Tests: For automating testing tasks that are
repetitive and relatively unchanging from one test cycle to the next
• Data Driven Testing: For testing application functions where the same
functions needs to be validated with lots of different inputs & large data
sets (i.e. login, search)
• Load & Performance Testing: No viable manual alternative exists

(Ramakrishna) #8

Day 4 - #30daytestingchallenge

Challenge - What types of testing can automation support you with? Share an example

Automation is very useful in below areas:

  1. User Interface / GUI or Front-End: Automation tests written to mimic the different flows , functions and interactions on a Web /Client application. This can be done by using appropriate tools like Selenium/Protractor / QTP. The core approach is to first identify , locate and store the Elements on these pages/screens and then perform operations on them using programming. Validations are implemented using Native tool assertions or by using custom programming depending on the tool support for these. Frameworks like TestNG , Junit , Jasmine make it easy to manage large number of tests and plan batch execution of tests to achieve automated regression testing. Being able to run these tests on a nightly basis using CI/CD tools like Jenkins/Octopus allows us to achieve high level of reliability and quality in the application or system under test.
  2. Database: In projects that involve lot of back end validations i.e. projects that use ETL / SSIS / SSRS / SSAS etc. Automated tools can be used to run OLTP validation queries on the database and validate the results using the tool assertions or custom assertions. Tools like Selenium used in conjunction with programming languages like Java , C# can leverage ODBC/JDBC connectivity to achieve this.
  3. API or Web Services: Tools like SOAP UI , Rest Assured enable automating the REST / JSON service calls using programming languages like Groovy / Java. Being able to automate API testing means significant reduction in effort and improvement in quality since each API end point can have multiple methods and each method can have multiple possible tests. It’s humanly not possible to reliably test all these manually every time a change is made in the system. If we rely on manual testing for API validation then we will be forced to take a risk based approach which can cause regression defect leakage.
  4. Mobile The leading mobile platforms i.e. iOS and Android support three types of applications i.e. Native apps , Hybrid Apps and Mobile Web based apps. In addition to this AR/VR features , AI features are recently added to the mix. Mobile automation using tools like Appium , Robotium etc. enable us to execute automated tests on virtual mobile devices or device emulators or on a farm of real devices. In addition to mobile devices automation us widely used for Smart TV software based devise like Roku etc.
  5. Performance : Performance testing scripts using tools like Load Runner , J-meter can generate virtual user load on Information Systems to mimic actual user loads to measure and optimize performance. These can be added to a CI/CD pipeline to run more frequently and identify the performance issues early rather than at the end of a project lifecycle.
  6. Security : I’m not sure about writing automated security tests , but off the shelf tools like HP Web Inspect etc. run attacks and report the vulnerabilities which are then later assessed for possible resolutions.
(Celine) #9

Day 4: What type of Testing can automation support you with?

As a manual tester, there are certain tests that cannot be performed by simply executing a combination of steps and requires sophisticated tools to simulate load from the number of users accessing the same applications simultaneously and system vulnerability to attacks/intrusion (XSS scripting).

Non-functional Testing

Non-functional testing involves testing the application against the non-functional requirements, which typically involve measuring/testing the application against defined technical qualities (also known as the ‘-ilities’ because they all end in ‘-ility), For example, Vulnerability, Scalability, Usability.

  1. Performance, Load, Stress Testing- measuring how a system behaves under an increasing load (both numbers of users and data volumes), load testing is verifying that the system can operate at the required response times when subjected to its expected load, and stress testing is finding the failure point(s) in the system when the tested load exceeds that which it can support.

  2. Security, Vulnerability Testing (Penetration tests) -Security testing tests the software for confidentiality, integrity, authentication, availability, and non-repudiation. Individual tests are conducted to prevent any unauthorized access to the software code.

  3. Usability Testing-Usability testing looks at five aspects of testing, - learnability, efficiency, satisfaction, memorability, and errors.

  4. Compatibility Testing- tests that the product or application is compatible with all the specified operating systems, hardware platforms, web browsers, mobile devices, and other designed third-party programs (e.g. browser plugins).

Source: https://www.inflectra.com/ideas/topic/testing-methodologies.aspx

#30daytestingchallenge
#day4of30daytestingchallenge
#ministryoftesting

(Eman) #10

Hi @maos, thanks for the information.

It helped :slight_smile:

(Darya ) #11

Challenge - What types of testing can automation support you with? We can automate many types of testing. For example, regression testing will save us from routine. Security testing, Load, Stress testing are very difficult to perform manually. However, it is necessary to monitor the results of the autotests. I believe that this is one of the important parts - a clear presentation of the autotests results, it will help us to react quickly to changes and make the right decisions.

(Kumar) #12

I believe this varies depending on the context. But generally, automation can possibly support heaps in below areas.

Repetitive Tests: Test cases that have a scope to get repeated over a period of time are a good candidate for test automation. This is one of the main reason for teams to usually automate regression/ smoke/ sanity tests. They usually save heaps of time, cost and man power in the long run.
Unit Tests: Automation could also help run those system level unit tests that a developer would want to verify before deploying to staging or test or prod environment.
API Tests: In a situation where the application interacts with multiple services, it makes sense to test those interactions in an automated fashion.
Non Functional Tests: While non-functional sounds too broad, automation is usually in testing the performance aspect more than any other non-functional aspects (UX, UI, Usability) of the application.

You can find the references here too: https://qakumar.wordpress.com/2018/07/05/day-4-what-types-of-testing-can-automation-support-you-with/

(David) #13

Based on this thread and some conversations we’ve been having in slack, it looks like we share a lot of the same experience.

On one hand there seems to be a very conventional idea of “Test Automation.” That is, I have an application, perhaps a web-based one, and I use some software, perhaps Selenium to run tests on it.

On the other hand there is “task automation to assist in testing” and it looks like a lot of us are doing this as well.

I recently learned Selenium, and haven’t been able to use it yet (though I have been practicing!). I may not even have the opportunity to do conventional, automated testing where I work, but I can see a LOT of use for it to speed things up, reflecting what @heather_reid said above.

Similarly I’ve started using a program called AutoIT to greatly speed up some repetitive tasks in a GUI. It might be possible to develop some sort of test framework with AutoIT but I haven’t explored this, and I do not test GUIs at my job.

And then there’s Python to automate…everything. (You know, the boring stuff.) I have written programs to speed up checking of hard to read data files, to pull logs from servers and validate data, and to automate generally boring tasks in the office.

There are a ton of tools out there and I look forward to learning Cucumber, POSTman, and a few others next.

-Dave K

(Heather) #14

From the twitterverse


(Karishma) #15

#Day 4 : What type of Testing can Automation support with ?
Smoke Testing
Unit Testing
Integration Testing
Functional Testing
Keyword Testing
Regression Testing
Data Driven Testing
Load \Performance Testing
Security Testing
Black Box Testing
Web Service\API Testing
Mobile Testing
AOT Testing

(PRACHI) #16

I work on automation for automating the regression tests- of a web app.

(Margot) #17

Day4: Ok, here’s where I am sad to be late on my challenge since I’m a little late to the discussion.

So, for this one of the things I wanted to say is Regression. Because it’s obvious right? Repetitive tests, big test suits you have to run every 2 months or LESS which sometimes take your entire QA team to be testing for over 2 or 3 days (sometimes more). Count how much money that is. If you are going to be repeating this process every release automating regression sounds like a good idea.

However I was reading this piece which suggest automating Regression tests is bad advise. What the author says

“unfortunately according to many testers and devs, regression testing means run tests over and over – automatically or repeating testing procedures, even worse – repeating them blindly without considering context. Problem is a regression bug is not necessarily a bug that we already know about and have a test about, regression bugs are not bugs who “rose from their graves”. Therefore, repeating tests to find regression bugs is like throwing bunches in the dark to hit a mosquito.”

I’m very interested in your opinions on this.

I read this here:
https://mrslavchev.com/2018/06/21/hindsight-lessons-about-automation-what-tests-to-automate/

(Jesper) #18

Automate all the things :muscle:
… that can be automated.

Especially in the context of automation as tool supported testing/ confirmation. So everything from small scripts, to coding frameworks to robot automation.

(AMIT) #19

I have implemented Automated Security Testing using OWASP-ZAP where everything is automated…Enabling the zap and then running the scan through API’s. Also, automated functional testing was integrated and run as part of it to have better coverage.
Comparing two documents e.g. pdf’s. This was bit tricky as there are tools to compare web page url’s like launching the url and then taking screenshots. But comparing documents was bit tricky which I have implemnted by converting pdf’s to image and then comparing image pixel-by-pixel. This was achieved by using 2 tools and through 2 commands. Also, section of pdf’s can be ignored from comparing.

(rosiecorn) #20

If you visit the site:
Source: https://www.atlassian.com/continuous-delivery/different-types-of-software-testing

It list of all of type of testing and what types of testing automation could support you with

Below are types of testing that we could apply automation.

Unit tests : Very low level, close to the source of your application.

Integration tests: Verify that different modules or services used by your application work well together, These types of tests are more expensive to run as they require multiple parts of the application to be up and running.

Functional tests : Focus on the business requirements of an application.

Performance testing: Check the behaviors of the system when it is under significant load.

Smoke testing: Basic tests that check basic functionality of the application.

Exploratory testing: Automation is key to make this possible and writing tests sooner or later will become part of your development workflow.