New codeless automation tool

Hi everyone!

I’m the co-founder of Reflect and I wanted to share our product with this community. (To the mods: I see the suggestion is to post new products to the ‘Cool Projects’ category but that link does not work so hopefully I’m posting to the right place)

Reflect is a no-code test automation tool for testing web applications. We know that record and playback tools have been around for a long time and have had some major drawbacks, specifically around test brittleness and lack of expressibility. We’re taking a different approach vs. other tools with the goal of making tests easy to create and maintain. While we’ll never let you achieve the level of expressibility of code, we think there’s a lot of value in a tool that hits the 80/20 rule; letting you test the 80% most common scenarios with 20% of the time you’d need spend with other approaches.

We can accurately record a ton of actions - I don’t want to list them all here for fear of this looking too much like an advertisement. But I’d love to get your feedback on what you want to see in a no-code tool and what kind of scenarios you’d need us to support in order for you to get value out of a product like ours.

Here are some questions.

1 - Does it allow us to code in case we need custom functionality also?
If yes, then which programming languages does it support. If no, then is it in your roadmap?

2 - Does it allow us to debug tests, i.e. run them step by step?
Are there any limitations in your debugger like needs too much memory, can be used only for certain parts of the test cases and not the entire test case etc.

Can we inspect any variables if we need to during debugging?

3 - Code or no code, does it have refactoring features?
E.g. Does it allow you to extract a test or its steps and convert them into some kind of reusable utility or test?

4 - Does it allow certain tests/“functions” to run before a group of tests just like TestNg?
E.g. Do login before every purchase related test. But, don’t login for product searching tests.

5 - What is the format in which you store test cases behind the scenes?
Eg. Java tests would be stored as Java files and class files. So, does it store tests in some proprietary format or common one like json, xml etc. ?

6 - How does it handle versioning and test case reviews?
Can we use Git & Github with it?

7 - Is every or at least major feature well documented?
Does every feature have a link to the documentation for that feature?

8 - How do I migrate away from your tool when I need a lot of customization?
Is there any tool which automatically converts the tests into documents or code?

9 - Which learning resources do you have for beginners to your tool?
What is the cost of these resources? Do any of these resources cover realistic use cases or just toy examples?

10 - Do you have any publicly accessible forums where people can ask questions about the tool?
Personally, I’d prefer forums such as stack overflow which are not owned or run by the tool company.

11 - Do you have any publicly accessible forum where people can suggest features and bugs, and also see their status?

12 - How is this tool better than its competitors?

13 - This one is for a user of any tool and not for the tool company.
Do some frequently used features of this tool require too movement with mouse/trackpad?

Too much movement of mouse/trackpad can potentially cause discomfort or even Repetitive Stress Injury/RSI. In contrast, RSI is alleviated, but not prevented or cured, by auto complete in coding IDEs which low code tools don’t have. So, you hope that the UX design of the low code tool does not take too many steps to do things, especially simple things.

PS -
I guess these could be used as a template for any low code, no code or codeless automation tools.
I think I could ask more questions, but I’ll stop for now.

2 Likes

At my company, we are already using Reflect and my colleague wrote her review of the tool. We think you guys are doing great thing! https://abstracta.us/blog/tools/reflect-review-the-new-automated-web-testing-tool/

Great questions raghu - see my responses below:

1 - Does it allow us to code in case we need custom functionality also?
If yes, then which programming languages does it support. If no, then is it in your roadmap?

No, and adding code support is not on the roadmap. Our goal is to enable users to automate as many scenarios as possible without writing code. The closest thing you’ll have to writing code in our system is Variables, which is akin to Excel macros. Variables allow you to set up more complex assertions, generate random values, things like that. More info on that feature here: Variables | Reflect

2 - Does it allow us to debug tests, i.e. run them step by step?
Are there any limitations in your debugger like needs too much memory, can be used only for certain parts of the test cases and not the entire test case etc.

We have support for re-running tests in our recorder up to an arbitrary point. Adding step-by-step debugging is on the roadmap!

Can we inspect any variables if we need to during debugging?

Yes we provide diagnostic information for variables but it’s more simplified vs. what you’d have if you were debugging a Selenium test, for instance.

3 - Code or no code, does it have refactoring features?
E.g. Does it allow you to extract a test or its steps and convert them into some kind of reusable utility or test?

Yes, we support test re-use via our Test Composition feature (Test Composition | Reflect) and refactoring tests via our “re-record” feature (Editing / Re-Recording | Reflect)

4 - Does it allow certain tests/“functions” to run before a group of tests just like TestNg?
E.g. Do login before every purchase related test. But, don’t login for product searching tests.

Yes you can do this via our Test Composition feature (e.g. every test that relies on login shares the same login test, but the actual execution of the login test runs in a separate browser context for each run so no state is shared across tests). More complex test orchestration is on our roadmap!

5 - What is the format in which you store test cases behind the scenes?
Eg. Java tests would be stored as Java files and class files. So, does it store tests in some proprietary format or common one like json, xml etc. ?

We store it in a proprietary format and expose test data via our API. We’re planning on adding integrations to test case management tools and bug trackers in the near future so tests data can be synced with these systems.

6 - How does it handle versioning and test case reviews?
Can we use Git & Github with it?

Tests are versioned within our system, but we don’t sync with Git. To be honest I’ve never seen syncing from a third party system to Git done well. It either is a one-way sync which doesn’t seem terribly valuable or it’s a two-way sync that runs the risk of overriding data if you’re making changes both via Git and via the web interface. Have you seen a product that does Git sync well?

7 - Is every or at least major feature well documented?
Does every feature have a link to the documentation for that feature?

Our documentation is pretty comprehensive! https://reflect.run/docs

8 - How do I migrate away from your tool when I need a lot of customization?
Is there any tool which automatically converts the tests into documents or code?

No, although I’d argue that any tool you use has an inherent lock-in cost. For example there’s no easy way to migrate code between open source frameworks. Certainly some vendors provide an export to Selenium though.

9 - Which learning resources do you have for beginners to your tool?
What is the cost of these resources? Do any of these resources cover realistic use cases or just toy examples?

Our docs cover all of the features of our tool, are completely free, and we also have videos embedded in the docs that cover the most important features. The videos show tests against both demo sites and real sites, but all of the sites are publicly accessible so that you can follow along step by step. Reflect also has a free tier so you can sign up and try it out.

10 - Do you have any publicly accessible forums where people can ask questions about the tool?
Personally, I’d prefer forums such as stack overflow which are not owned or run by the tool company.

We get questions about our tool periodically on public forums. Here’s an example thread on Reddit: https://www.reddit.com/r/QualityAssurance/comments/l287cu/which_noncoding_automation_tool_would_you/

11 - Do you have any publicly accessible forum where people can suggest features and bugs, and also see their status?

No but we’re very open to feedback - you can always email us at support@reflect.run!

12 - How is this tool better than its competitors?

We’re very focused on test accuracy and maintainability, and a lot of the design decisions about the product and built around those two concepts.