How do you choose a programming language for software testing automation?

(Heather) #1

This is related to a question that came up in the Test Automation AMA with Richard Bradshaw (at 3 minutes 12 seconds). It’s something I see asked a LOT. Questions vary in wording but are along the lines of:

What advice would you give to someone who asked you this type of question?

I often hear Python recommended as a good language to get started with and get to grips with faster. I’ve personally used Java and JavaScript at different points in my career. I used Java the first time because it was a language I was familiar with so I saw it as a smaller ramp up time. I used JavaScript the next time because the devs wanted to help contribute to the automation suite and that’s what they programmed in.

(Onur) #2

Hello all, in my opinion, you should consider below items:

  • Language community of your automation framework. For example, if you use selenium webdriver, Language support of JAVA is %68, JS %15, Python %11, %5 Ruby, %1 C# Reference: [Dave Haeffner SauceLabs Event Presentation]

  • You should check which version is stable and get the updates first. For selenium, as I saw JAVA gets the first updates.

  • You should check StackOverflow which language has more support.

  • You should consider your competency of Programming Language which will be used in automation project.

  • You should consider Test Runner frameworks and their capabilities such as JUnit, TestNG, NUnit, MBUnit, etc.

  • You should consider how you can run your tests in parallel without any pain.

  • You should consider your software development house’s programming language. In this way, you will get help from developers but in general, developers focus on development rather than test automation.(not always but in general). If they write unit tests, that will be great! :slight_smile:

  • API Testing libraries of that programming language.

  • BDD support of that language.

  • I am in a .NET house but I changed the test automation language from C# to JAVA. Because of above reasons. In addition, when you want to search test automation experts, you can find JAVA experienced test automation experts much easily in our region. If you are a manager or decision maker, you should also consider this too.

(Brian) #3

For my current project, I chose Python for my UI test automation via Selenium simply because I’m more comfortable with it. When the team changed (new phase of the product), I changed the Selenium tests from Python to C# with NUnit because that is what the other in house programmers use.

For test data creation, I still use Python because I haven’t gotten around to changing it for the new circumstances. (And my Python scripts work very well for our needs)

For Web API tests, I use javascript and Postman. My department has long used SoapUI and Groovy, but I ran into enough problems with that, so I recommended a switch, and (much to my surprise) my teammates agreed.

(Augusto) #4

I have a different perspective when choosing the language to use. I choose to use a language that enables collaboration and team ownership of test automation.

Different languages create silos and silos are not good for collaboration.
Different languages do not encourage ownership of the tests codebase.

Using the same language as the developers encourages developers to dip their toes into automated tests and understand what automated tests are and what their goal is
This in turn will trigger interest in test automation and the whole team will collaborate to improve it
Using the same language as the developers the team will own the tests as much as the production code and feel responsible for them
Using the same language as the developers will bring the tester and the developer closer as they have something in common and as we know looking at things we have in common builds bridges while focusing on differences builds walls

(James Jeffers) #5

All other things being equal, it doesn’t matter which language you choose. Learning just 1 of the languages out there (Java, C#, Ruby, LISP) is enough to give you the experience to develop the skill of programming.

That being said, if they are feeling overwhelmed by trying to pick from the many different languages, I’d make this easy by deciding for them. As Augusto mentioned, pick the same language the developers are using on the system they have to test.

If they are still stuck, just tell them a language.

I will suggest Python more often than not. Short list of reasons and some resources at my blog.

(Onur) #6

For theory, this is an awesome approach but in general and in reality developers don’t have time for API or GUI level automation. If they write unit tests with a high coverage, that will be fantastic. But again I want to emphasize that your approach is more collaborative, more agile, more suitable for “Whole Team Approach” but I don’t know how many companies can accomplish this in reality.

(Augusto) #7

I, like most human beings, operate in my reality.
It is possible that in your reality, shared ownership of test automation doesn’t happen.
This certainly doesn’t imply that it doesn’t happen in other people’s reality.

When I coach teams, collective ownership of tests is a prerequisite. Many teams I worked with had testers that collaborated with the developers in creating the automation and in other teams we didn’t have individuals with the title tester but testing was a very prominent activity within the team.

Developers have x amount of time to do their job. Saying they don’t have time for tests to me doesn’t make much sense. They might consciously decide not to write them, but time is not the issue, would you not agree?

[quote=“obaskirt, post:6, topic:13213”] But again I want to emphasize that your approach is more collaborative, more agile, more suitable for “Whole Team Approach” but I don’t know how many companies can accomplish this in reality.

In my reality, many companies do this. I can tell you because I have coached many companies (in reality) and I can’t think of one that was not able to reach this level.

(Onur) #8

I am so happy to read that your reality is the ideal and agile reality. I agree with what you wrote. You coached the teams in a great way, you changed the mindset of the teams and you are just awesome!

My reality, unfortunately not the perfect one and it is not easy to change this reality with limited power and decision making authority. I am just so happy for you and your teams which you coached.

(Kate) #9

The one thing I’d add here is that if you start with a language that doesn’t use pointers (like Java or C#) then find yourself having to switch to one that does, it can be a lot harder than going the other way.

That said, the key skills for software testing automation are being able to decide what should or shouldn’t be automated and being able to break down what should be automated into a set of instructions a computer can execute every time. Which language you use to do that isn’t relevant.

(Kim) #10

@gus completely agree with everything you said looking at the team as a whole and what tools/language etc is best for all the team works most effectively. I am very fortunate to be in bedded within a development team that has this approach. Coming on board is taking some adjustment, to learn new system/language/tools but all the team is supporting my learning phase making this a fairly easy transformation.

(Augusto) #11

Great to hear another good experience from the trenches @kimberley
Another good outcome of skilled workers collaborating tightly is that the collaboration helps build skills that are not maybe the core of the role but that are vital in helping the team function. People become t-shaped.
I wrote a semi-serious post on this a few years back, see if you see yourself in it

(Kim) #12

I am trying to be M shaped :wink:
Never scared of the challenge but it only works when the business & team are with you and I am a firm advocate of Agile methodologies which encourages a certain kind of mindset. Ego’s and silo behaviour have no place within a highly functioning team.
Example tonight my team was releasing a new website and this morning the business came through with some changes. The team dropped everything and focused on this work. Testing was done by all the team which included our CTO (PO)/Devs/BA/Testers. We were all on-shore located using ‘Discord’ to talk as a team and work together to get this done. It made release as per schedule with all the changes put into place and it was amazing to be part of something like that. I truly hope all testers get to part of a team like I am in now.

(Stefan) #13

I’m amazed how easy people recommend various programming languages without knowing what is meant with software test automation, without knowing the context.

Is test automation related to:

  • building hardware, or programming micro-controllers or embedded systems or web apps or others?
  • software hosted on ioT devices, mobile, cloud, desktop computers, industrial systems, etc…
  • what kind of software type, level, etc… are you trying to test/use scripts for? unit testing, integration, system, backend, middle ware, databases, server side/client side, drivers, connected devices, apis, etc…
  • what’s the scope of your script? create test-environments, test-data, manipulation, generation of data, reading log files, calling rest/soap apis, regression checks, manipulating configuration files through ssh on a Linux server, setting up a script to run multiple scripts for a particular server, creating interfaces for various services, etc…
  • who’s creating the script, who’s using it, how maintainable does it have to be, how extendable does it have to be, how many times you have to run it, how scalable it should be, who’s taking care of it - rewrite, repair, does it have to integrate with something?

I’d say it depends. Any tool or programming language should be picked based on the context and scope.

(Onur) #14

PERFECT! This is Brilliant. :slight_smile:

(Onur) #15

I liked your answer. It is very very very meaningful. Thank you Stefan. I agree with every word that you wrote. For example, when I was a design verification engineer, I used Perl, Python, PSL, TCL, and also bash scripting. Because the context of the product, build environment, OS, etc. are more compatible with these languages. When I worked on a Network GateWay Project, I have to use C++, because again the product, OS, build environment, team knowledge, and also developers were using this language and this is the most meaningful choice in those circumstances and conditions. When I worked on web and mobile projects, I chose JAVA, TestNG, Selenium, Appium stack. I tried with my team also Phyton, Ruby, and C# and we gave this decision all together and we are much more productive with these stack. Before, I used and I sticked with JUnit but my team members told me that TestNG was more QA friendly. Then I tried and I saw that it is more comfortable Test Runner framework for test automation. So, we follow “Inspect & Adapt” principle and we gave decisions all together. Thanks again for your wonderful answer.

(Matt) #16

The advice I generally give is that unless there is a good reason otherwise, stick to the language the rest of your team is using. This is assuming a cross functional team where the testers/devs/etc are co-located.

Most modern languages are pretty much at feature parity when it comes to web/api automation, so you shouldn’t be too hampered with going with what they’re using. Ruby, Java, Scala, Golang, Javascript, C#, Python - they’ve all got great testing communities and you’ll be generally fine.

What you get out of this is that your testers and devs now read and write the same language. So if you need help doing something tricky, if you happen to be on holiday, they can pitch in too. Or even just review pull requests. If you write it in a different language you’ll see people probably avoid your test suite. If they code Java all day long, they’re not going to want to review a Ruby test script.

Of course, this doesn’t work for all cases. An SDK you might really need might not exist in that language. Or there might be a really neat test tool (something like Serverspec) that only exists in one or two languages.

One excuse I’d advise against using is “but I don’t know {language x}!”. Learning your first programming language is hard, but your second, third, etc, are far easier. If you’re on the project for the long haul, you’ll have time to learn it. And unless you’re doing Haskell or Assembler or something weird, then chances are the language is pretty similar. And it’ll look good on your CV!

(Andy) #17

My company has started accomplishing this by getting devs to update tests they break. While the testing team is reasonable for adding new tests. It also helps when a handful of devs have had exposure to a testing framework at prior points in their career. In return for that time we’ve learned how to add our own QA related selectors into the app as well as a add our own testing specific endpoints into the API. So it’s very possible, but just like anything, it needs to be worked at.

(Gary) #18

Does anyone use anything other than programming languages?

I have noticed quite a few products that do a record and click type automation test suite. I have had a play about with for automated regression testing but still at an exploratory stage. It seems to have a fair few advantages over programmed automation tests.