I’m really tired and I can’t do friendly feedback right now, but I can do honest feedback flavoured with this prologue: I am honest and direct because I wish to see you succeed by putting your energy in the right place, forearmed with information. I do not wish to depress you about the minefield you wish to cross, I wish only to point out where the mines are in case you didn’t see them already. If you want emotional support over pragmatic support please ignore this post.
Language Considerations
Does your DSL support OOP? Can I build re-usable functions? Do I get encapsulation? If you simplify the programming language you can abstract out these very powerful ideas. How do I run the same test with multiple sets of test data? How does mocking work in your system? Can I achieve these things quickly and effectively, the way I want to?
Human-Readablenessability
I derive a lot of discomfort from “human-readable” DSLs for communication purposes, because they are useful lies spread beyond their context. I think playing with one’s Gherkin should be a private affair. A good programmer understands that they are writing a high-level series of commands that are compiled and executed via a sea of obscure abstractions. My fear is that trying to make computer commands read like natural language confuses what the computer actually does with what we’d like it to do. Automation artefacts with names like “test login page” exist because we need the affordance of purpose and we are therefore permissive of how incorrect that name really is. We know that it’s really just performing very specific checks on a specific system at a specific time with specific data and giving back specific data for us to interpret instead of doing real testing. When we take our useful-but-incorrect terms, reabstract them into human-readable language and hand them to people without that domain knowledge we spread the idea that computers can test themselves and we advance misunderstandings about the limitations of expensive automation systems. We also devalue the humans at the heart of testing by pretending there’s fungible humanity in our machines, and/or misinform people about the human-centric nature of software exploration (testing).
I have no problem with human-readable DSLs as just a higher level of high-level languages, but they should be used by people who know their limits and can use them responsibly. By this point, though, many people with the domain knowledge to use a higher-level language can either switch back to the more flexible high-level language they already use or write their own DSL.
Buying a Tool
I don’t have any problem with it being harder than a visual recorder because recorders are mostly useless for serious commercial automation projects (where “commercial” also infers “has money to spend on your tool”). I think that should be a concern for your idea, also, because many people abandon projects and tools because they’re not fit for purpose - and they do this because they assume their tools are more capable than they are and don’t yet know what they want them to do until they trip over it. Selenium is partly popular because of its flexibility and power. If your system can’t read an external log file or pause for Ajax or deal sensibly with iFrames then it does not matter how easy it is to use. When I’m researching automation I want to know if it’ll plug into my stack - will it work with Jenkins? Can my deployment process trigger it and can it trigger my deployment process? Can I run it headless? Can I run it in a grid or in parallel? Is there a simple way to run the same suite across different browsers? A floor mop is easy to use and suitable for beginners and understood by all team members, but it can’t support my automation project.
I want my tool to be able to perform automatic checks and do something with the results and those checks include things that are not in the browser - they are external files I might upload or download or the server might change. I might want performance data on page loads. I might want to look into a message queue on a Rabbit server. I might want to change settings on my web server and run pseudoidentical checks for each state. If I can’t do those things then my tool is much less powerful, and likely simpler checks I’m likely to not want to run long, costly and fragile UI tests for.
Rules of Acquisition
I’d also look into this from a business perspective. If you’re going to try to sell this I should warn you that some DSLs like this sort of exist. The “page object” gem for Ruby is one example that already has advantages over a simple declarative DSL. I wrote my own version which was similar, as DSLs are pretty easy in Ruby. The DSL is neater and nicer and easier to use/read, but its main advantage is programmatic as it gives you the encapsulation and inheritance of a page object framework with less effort.
Also, you are intending to sell a web browser testing DSL when Selenium/Webdriver/Watir/Cucumber exist, are mature and are free. People who make web software, and want web automation at a scale that they are willing to pour money into (for it is a development project with all associated costs), often have a team of software engineers who already know one or more programming languages and how to use Selenium for that language and platform.
So who is your target audience? What’s your USP here? What does your tool do (or for whom) that a series of existing, free and very popular open source systems fail to achieve? Does that thing warrant payment? Will it scale? Go through your explanation of your tool and cross out everything that Selenium Webdriver + Cucumber does already, and see what you have left. If it’s enough to part the fat cats from their swag bags then you have a better shot. If your sell is, let’s say, that non-coding testers can use it for businesses without an existing automation solution then you’ll need to either say that using it easily translates to other tools or that it scales into a viable commercial automation project - otherwise why not just train them in Selenium or hire someone who knows it? Some technical so-and-so is going to have to set it up on a local server and so on, so unless you’re planning on one-click deploy cloud solutions that’s a factor. I presume it’ll need an installation of C# and Visual Studio (or conceptual equivalent), so who’s doing/paying for that? Now I need to learn Visual Studio as well as the DSL tool. How do I run it locally? How do I kick off a build? What do I do if there’s an error? There’s a mountain of hand-holding that those versed in the programming arts take for granted. The journey for a newbie is going to have to be extremely smooth; or, from your perspective, time-expensive and frustrating. I always tripped up on new systems not because of the code I’d have to learn but the systems I’d have to install and the crappy setup tutorials I’d have to follow. If I’m paying for simplicity then I want this done for me.
This is one example for one particular audience, but it’s worth thinking on how the tool will practically solve a business problem.
Remember that business wants automation these days because they want some vague idea of DevOps and instant deployment and zero cycle time covered in money-flavoured ice cream. You need to support that need if you want to suckle at the plump teat of paid tooling OR you need to find and serve another need, otherwise you have a neat toy instead of a business idea. I love neat toys, but they rarely make stable money.
I point these things out because I’m testing your idea and I hope to give you useful information to inform decisions. I’m always in support of powerful tools used thoughtfully and responsibly, and I wish you luck.
