Simple automation testing language interest?

I’m looking to create a simple language for use in web automation testing. This is a business idea and I wanted to be clear about that upfront, not that I’m looking to sell you anything right now (there’s nothing to sell).

With everyone here being enthusiastic testers (myself included!), I’m hoping to get some friendly feedback on what I’m looking to create. What I most what to know is whether this is something you could see yourself using.

I’ll give a very brief overview and will intentionally leave out technical details and technical depth. Please ask anything if you’d like to know more.

Would you benefit from a domain-specific language (DSL) that let you easily describe the interactions with a web site necessary to verify a given piece of functionality?

Code-up automation tests without needing to know how to code. Reads more like natural language than code.

Easier for less-technical people than coding traditional automation tests but suitable for testers, developers, business analysts and product owners. More maintainable than using a visual recorder.

Pros/cons of proposed DSL:

  • Pro: Relatively easy, anyone with an understanding of web page elements can do it
  • Pro: Factor out modules of testing code, integrate modules together to represent user scenarios
  • Pro: Easier to learn than fully coding your own tests
  • Pro: Easier to maintain than using only a visual recorder
  • Con: Never as in-depth as writing your own code directly
  • Con: Not as easy (to begin with) as a visual recorder

Clearly this is a very brief overview of a concept that could go into a lot of detail. But as an overview of an idea, what do you think?

Is this something that would benefit you?

@webignition great to see people thinking about ways to help the community. I would welcome any news tools that help more people access automation in their testing.

I have quite a lot of experience programming and therefore usually write automated tests in whichever language the development team I am part of are using to ensure that everyone in the team can maintain the test suites. As you have said already any tool will never be as flexible as writing you own code but there may be a gap in the market for a new DSL. The barrier of learning to code is quite a daunting and time consuming task for anyone that has little experience.

My questions are really about where you would plan to focus your language around first? web UI, APIs, mobile etc. I would pick one small area and use that as a proof of concept.

Are you imaging something like Gherkin that allows for high level test cases which will drive other test code or will your DSL itself be the one driving the application under test?

Happy to get more involved for any feedback or impartial opinions.

@rob212 Thanks for the positive response!

I’d be initially looking at browser-based automation testing, focusing on anything where you might currently write code that utilises an implementation of the WebDriver API to programmatically drive and inspect a browser.

My first very minimal proof of concept will be the compilation of the DSL down into code which is functionally equivalent to the code you would write.

A (simplified) coded browser test might perform the following steps:

  1. Open https://example.com/login
  2. Verify that the page title is “Login”
  3. Set “user” as the value for the element identified by “#login-user
  4. Set “password” as the value for the element identified by “#login-password
  5. Click the element identified by “#login-submit
  6. Click the element identified by ‘.nav .user-account’
  7. Verify that the page title is “Your Account”

These steps can be expressed in C#, Java, JavaScript, PHP, Python (take your pick) in a manner with which you’re already very familiar. As you mentioned, such code can be overwhelming for the inexperienced and can be hard to master.

The above steps written in the DSL (using CSS selectors for the time being) would be:

open "https://example.com/login"
assert "title" is "Login"
set "#login-user" to "user"
set "#login-password" to "password"
click "#login-submit"
click ".nav .user-account"
assert "title" is "Your Account"

The DSL is unambiguous, concise, maps well to how someone would interact with a website and is capable of being understood by a broader audience than just experienced programmers.

The DSL would be transpiled to C#, Java, JavaScript, PHP, Python (again, take your pick) which would then be executed as it might normally be. The language to which the DSL is transpiled is not particularly relevant at this stage. Assuming the resulting code is not exposed to the user, I’d initially go with PHP since this is the language with which I’m most familiar.

The commercial service arising from this would provide the ability to write test scripts in the DSL and to execute the resulting code without the user ever needing to set eyes on the code to which the DSL is transpiled.

The DSL offers simplicity to the user and to me offers the ability to limit what code can be executed on machines under my control (elimination of malicious actions).

That’s as far as the proof of concept/minimal viable product would go in my mind.

I’d certainly like to expand the resulting service further to cover features (in no particular order) such as:

  • modularise code written in the DSL and let modules by imported and used in other places
  • create a tool to record interactions and translate these into the DSL
  • allow feature files written in gherkin to be linked to/associated with one or more scripts written in the DSL to support the execution of test suites that directly relate to requirements

I hope that gives you a better understanding of the direction in which I’m looking to head without getting into too much technical nitty-gritty.

When actual work starts, I’ll be building this all in the open, with the spec for the language on github and open to suggestions via issues or changes via pull requests. The implementation of the transpiler will be similar open for anyone to use in whatever way they like.

More comments are more than welcome!

I’m really tired and I can’t do friendly feedback right now, but I can do honest feedback flavoured with this prologue: I am honest and direct because I wish to see you succeed by putting your energy in the right place, forearmed with information. I do not wish to depress you about the minefield you wish to cross, I wish only to point out where the mines are in case you didn’t see them already. If you want emotional support over pragmatic support please ignore this post.

Language Considerations
Does your DSL support OOP? Can I build re-usable functions? Do I get encapsulation? If you simplify the programming language you can abstract out these very powerful ideas. How do I run the same test with multiple sets of test data? How does mocking work in your system? Can I achieve these things quickly and effectively, the way I want to?

Human-Readablenessability
I derive a lot of discomfort from “human-readable” DSLs for communication purposes, because they are useful lies spread beyond their context. I think playing with one’s Gherkin should be a private affair. A good programmer understands that they are writing a high-level series of commands that are compiled and executed via a sea of obscure abstractions. My fear is that trying to make computer commands read like natural language confuses what the computer actually does with what we’d like it to do. Automation artefacts with names like “test login page” exist because we need the affordance of purpose and we are therefore permissive of how incorrect that name really is. We know that it’s really just performing very specific checks on a specific system at a specific time with specific data and giving back specific data for us to interpret instead of doing real testing. When we take our useful-but-incorrect terms, reabstract them into human-readable language and hand them to people without that domain knowledge we spread the idea that computers can test themselves and we advance misunderstandings about the limitations of expensive automation systems. We also devalue the humans at the heart of testing by pretending there’s fungible humanity in our machines, and/or misinform people about the human-centric nature of software exploration (testing).

I have no problem with human-readable DSLs as just a higher level of high-level languages, but they should be used by people who know their limits and can use them responsibly. By this point, though, many people with the domain knowledge to use a higher-level language can either switch back to the more flexible high-level language they already use or write their own DSL.

Buying a Tool
I don’t have any problem with it being harder than a visual recorder because recorders are mostly useless for serious commercial automation projects (where “commercial” also infers “has money to spend on your tool”). I think that should be a concern for your idea, also, because many people abandon projects and tools because they’re not fit for purpose - and they do this because they assume their tools are more capable than they are and don’t yet know what they want them to do until they trip over it. Selenium is partly popular because of its flexibility and power. If your system can’t read an external log file or pause for Ajax or deal sensibly with iFrames then it does not matter how easy it is to use. When I’m researching automation I want to know if it’ll plug into my stack - will it work with Jenkins? Can my deployment process trigger it and can it trigger my deployment process? Can I run it headless? Can I run it in a grid or in parallel? Is there a simple way to run the same suite across different browsers? A floor mop is easy to use and suitable for beginners and understood by all team members, but it can’t support my automation project.

I want my tool to be able to perform automatic checks and do something with the results and those checks include things that are not in the browser - they are external files I might upload or download or the server might change. I might want performance data on page loads. I might want to look into a message queue on a Rabbit server. I might want to change settings on my web server and run pseudoidentical checks for each state. If I can’t do those things then my tool is much less powerful, and likely simpler checks I’m likely to not want to run long, costly and fragile UI tests for.

Rules of Acquisition
I’d also look into this from a business perspective. If you’re going to try to sell this I should warn you that some DSLs like this sort of exist. The “page object” gem for Ruby is one example that already has advantages over a simple declarative DSL. I wrote my own version which was similar, as DSLs are pretty easy in Ruby. The DSL is neater and nicer and easier to use/read, but its main advantage is programmatic as it gives you the encapsulation and inheritance of a page object framework with less effort.

Also, you are intending to sell a web browser testing DSL when Selenium/Webdriver/Watir/Cucumber exist, are mature and are free. People who make web software, and want web automation at a scale that they are willing to pour money into (for it is a development project with all associated costs), often have a team of software engineers who already know one or more programming languages and how to use Selenium for that language and platform.

So who is your target audience? What’s your USP here? What does your tool do (or for whom) that a series of existing, free and very popular open source systems fail to achieve? Does that thing warrant payment? Will it scale? Go through your explanation of your tool and cross out everything that Selenium Webdriver + Cucumber does already, and see what you have left. If it’s enough to part the fat cats from their swag bags then you have a better shot. If your sell is, let’s say, that non-coding testers can use it for businesses without an existing automation solution then you’ll need to either say that using it easily translates to other tools or that it scales into a viable commercial automation project - otherwise why not just train them in Selenium or hire someone who knows it? Some technical so-and-so is going to have to set it up on a local server and so on, so unless you’re planning on one-click deploy cloud solutions that’s a factor. I presume it’ll need an installation of C# and Visual Studio (or conceptual equivalent), so who’s doing/paying for that? Now I need to learn Visual Studio as well as the DSL tool. How do I run it locally? How do I kick off a build? What do I do if there’s an error? There’s a mountain of hand-holding that those versed in the programming arts take for granted. The journey for a newbie is going to have to be extremely smooth; or, from your perspective, time-expensive and frustrating. I always tripped up on new systems not because of the code I’d have to learn but the systems I’d have to install and the crappy setup tutorials I’d have to follow. If I’m paying for simplicity then I want this done for me.

This is one example for one particular audience, but it’s worth thinking on how the tool will practically solve a business problem.

Remember that business wants automation these days because they want some vague idea of DevOps and instant deployment and zero cycle time covered in money-flavoured ice cream. You need to support that need if you want to suckle at the plump teat of paid tooling OR you need to find and serve another need, otherwise you have a neat toy instead of a business idea. I love neat toys, but they rarely make stable money.

I point these things out because I’m testing your idea and I hope to give you useful information to inform decisions. I’m always in support of powerful tools used thoughtfully and responsibly, and I wish you luck.

3 Likes

Well said, Chris!

I preface my comments in the same manner. I have opinions and thoughts that may assist in deciding direction but not all that supportive of an abstraction layer around an already great tool.

When I saw the initial post, my thoughts turned to the two tools built in-house around WebDriver. In my opinion, that was two too many tools. The people that built them, fine test engineers all, have their days filled with either tool maintenance or adding tests. The tool maintenance sucks the greater energy where, in my opinion, it would be better directed at improving application testability, unit test construction or consulting, or environmental improvements.
A test engineer’s primary purpose is the support and improvement of testing. While tools and utilities serve that purpose, the efforts must be tempered by testing value and benefit. A layer of abstraction tends to reduce the benefit because maintenance distracts from testing.

In my opinion, Jon, new test engineers have a greater barrier of entry when attempting to learn, use, and become proficient with an abstraction of a well known tool. I’d rather hire test engineers who can make WebDriver sing tests; it’s much more time consuming to hire a test engineer and train them on an abstraction. I’d rather have that time spent in aiding testers.

Joe

I understand the intent, but two main challenges I see whenever someone talks about a DSL for browser automation/testing/etc:

  • The abstraction is going to be much more limiting than someone who understands coding - very similar to Selenium IDE or other record-and-playback frameworks - you make things easier, but there’s going to be limits. Is the benefit of making the tooling available for more folks worth losing the power/flexibility at the other end?
  • The abstraction is going to have a steeper learning curve - e.g. if the test was written in C#, then any C# developer can quickly get up to speed, but if it’s a special tool, there’s going to be one more thing to learn

Arguably, the tooling around Cucumber is exactly what you’re describing. I’ve never actually seen Cucumber work. In the context of the above:

  • the abstraction layer - someone has to write the code for what “User has requested a password reset” means. Dissonance here as there may not be definitions for cases the analysts want to handle, or …
  • the learning curve - the definitions exist, but the analysts don’t know what the proper string to use is. Even with regex and what not, not knowing the definition names leads to challenges, as someone writes a scenario, then finds one of their steps lacks a definition (or even worse, the definition has subtle side effects the user isn’t aware of), the scenario gets handed off to whoever is responsible for Cucumber implementation, and then they have to spend a lot of time figuring out what the analyst really wanted, and matching it up to the definitions that exist.

That being said, I have talked to folks who say they’ve gotten this working, that they invested the resources to train their analysts, maintain a list of definitions that everyone uses, and have the intermediate help layer to sort out scenarios when they’re not doing what the end user hoped, but it sounded like they invested a ton of effort to build that culture. It wasn’t clear to me if in the long run it was any better than if they’d used the resources to get more developers/testers to write automation using a lower-level framework/language.

@kinofrost Thanks for your colourful and detailed dive into the pitfalls that I could encounter. I certainly hadn’t considered all of these matters. Your thoughts and opinions are of great help to me.

@devtotest, @ernie Thanks also both of you for furthering the discussion. I’m glad to see people with experience-based knowledge chime in here.

There are some common themes surrounding the comments made so far. Let me first say that I agree wholeheartedly the sentiments brought up.

I realise that I have failed in highlighting the audience I believe will benefit from what I am looking to create.

I see two common points raised against this DSL: it can never provide what hand-coded tests created by experienced test engineers can provide, and that an intentionally-limited DSL is going to be less powerful and less flexible that hand-coded tests.

For both points, I agree entirely. I wasn’t envisaging this DSL (and the related commercial service) as being something that would, or even should, appeal to experienced test engineers. I was hoping for something that wouldn’t be totally off-putting to experienced test engineers but not necessarily something that would be a direct alternative to how test engineers currently work.

@ernie You make an interesting point about how an abstraction presents a steeper learning curve. What interests me more is not the point you make but the context from which you make it.

What I propose absolutely would present a steeper learning curve for an experienced test engineer who would otherwise spend their time writing WebDriver-based tests in pure code. I fully agree.

What I’m ultimately aiming at is a DSL that can produce really good tests with minimal coding knowledge.

This is for businesses who cannot afford a £60,000 test engineer but can afford 2 x £25,000 junior testers plus a tool for what could be considered business-equivalent results.

This is for organisations who already have business analysts with detailed knowledge of their products and where the BAs, by the nature of needing to translate business needs into development needs, have a suitably-technical understanding of matters such that they can translate the interactions they perform on a website into a few lines of simple code.

The learning curve consideration changes for an audience of this type.

The learning curve involved in transforming a manual tester into an automation test engineer is really, really, really huge.

To be an excellent automation engineer you need to be an excellent tester and an excellent coder. Myself excluded (I’m only an excellent coder!), everyone so far who has chimed in on this discussion fits what is needed of an excellent automation engineer.

My partner runs in-person group training courses specifically for turning manual testers into automation testers and from hearing back about how the course attendees handle the learning involved I have made the following observations (science!):

  • Some people just aren’t cut out for coding (no surprises there). For all of us here who can code, the process of developing your skills into being able to code effectively and in a way that has business value is a progression that involves constant effort and constant learning but is ultimately achievable. Some others can just never get their brains around this.

  • Some people expect a 2-day manual-to-automation tester primer course to turn them into market-ready valuable people capable of demanding a strong salary. I’m sure we can all agree that this can never be the case.

  • Some people want to be able to tell their browser what to do (I’ve automated it!) without realising that they have to be a coder who can write tests (a fair amount overlap here with the first point)

  • There are many more people like this than there are who are like us

  • In our places of work, we never encounter those who can just never code no matter how much they try to learn. Those people are filtered out early in the recruitment process and never set foot in the same places we do

With that all said, I’ve hopefully given some better understanding of where I’m coming from.

To sum it up:

Does what I propose fit as a perfect (or even sub-optimal) alternative for the coding carried out by experienced automation engineers? Nope, not even close.

Does what I propose allow people with suitable testing knowledge and/or suitable product knowledge create browser-automation tests in a manner that might otherwise not be achievable? Maybe, possibly, but probably not a hard “no”.

What this all boils down to is an alternative to visual recorders that kills off the downsides to using a visual recorder without introducing the real downsides that exist to pure coding when considering the effort required to become a competent coder for people who cannot code.

From my experience, a good tester can write simple tests in Cypress.io with zero previous JavaScript knowledge - just basic knowledge of Chrome dev tools and Cypress documentation for reference. Of course, these tests would be pretty shallow and based around interactions with UI.

For more complex stuff you need to introduce stubs/mocks and more advanced code. It will be challenging to present such code in a simple way so that it’s easy for non-coders to work with it.

This all makes sense, but who’s writing the translation layer that goes from a statement like assert "title" is "Login" to the appropriate code? Short of a solution like http://test.ai/, the creation and maintenance of that translation layer between the DSL and Webdriver seems like a deal breaker for most organizations, i.e. if you have someone who can maintain that abstraction layer, unless you drive the culture like you would in shifting to Cucumber/Gherkin, it seems really expensive.

And I still don’t quite get how this really differs from Cucumber/Gherkin.

@ernie I’d be creating a commercial service to handle the transpilation of the DSL to actual code and the execution of such code.

At the absolute most basic, imagine a web page with a single large text area. Pop your DSL code into there, click a button, get results. That is an intentionally oversimplified description, devoid of almost anything worth paying for, just to cover your question without getting sidetracked.

The DSL is just a small part of the matter that I wanted focus on for the time being. I didn’t want to get into the commercial side of things in this discussion.

@milos Cool, thanks for bringing that to my attention! Challenging is fine by me :slight_smile: