Good Ways to Write Automated Regression Tests for Ugly Apps

The Background:

The app was first created (spawned?) in 2002 as a classic asp web application that included faux object orientation (the only way classic asp could do anything of the sort) and database interaction. It has since… grown. I estimate over 1 million lines of code including the logic buried in stored procedures, logic in the “objects”, logic in the main asp files including a ton of Javascript, embedded <% %> tags and nothing that could possibly be unit tested.

It currently takes me around an hour for a manual smoke test, so obviously I want to automate some kind of smoke test.

Which is where the fun comes in…

The Problem:

This monstrosity is also loaded with dependencies. It’s a multi-tenanted application, which means if I wish to edit record A for company 1, I need to start by logging in as a user who is allowed to edit records in company 1 (the easiest condition, since nothing happens in this app unless someone is logged in). Then I pick the company I’m working in. Again, basic.

Now the “fun” begins. I need to be able to check or ensure that company 1 is in fact in an editable state. Since I can’t start with a clean system (1/2 TB database and growing), or even be sure that nobody else has changed my test company before my automation starts, I have to check the company state, and if it’s not in an editable state, put it into one.

Similar conditions apply to any action I would take in the system.

Now, I’ve worked with ugly apps like this in the past. The method I usually found myself working with (Method 1) was a monster case statement handling the test running, where each test in a CSV file specifies the type of test it is. The end result is a csv “database” that comes to somewhat echo the actual database and ultimately gets overblown and difficult to change.

It also causes problems when one of the early tests doesn’t end correctly because that can put the whole thing into an unstable state, and it can be challenging to work out what actually failed when unless it’s architected very carefully.

Method 2 is slightly less ugly (code-wise) – namely using reflection (I’m working with C#, so there’s no “eval” type statement I can use) to invoke the method named in the input CSV file. As far as I can tell, the only advantage of this method of managing the dependencies within the app is that instead of a humongous case statement I end up with a modestly-sized invoke method. The disadvantages include the code becoming even more opaque and a speed penalty.

Method 3 is to create custom attributes and use these to define what is required for a specific test method. That way instead of the edit record test method having massive chunks of setup code (that’s repeated for the delete record test method and the add record test method), it would have one or more prerequisite attributes (such as “CompanyStatus” == “Editable”) and a call to check attributes then if prerequisites aren’t met, calls to make the changes needed to meet said attributes. This keeps the test methods self-contained, but will slow the overall run since each test handles its own dependencies.

My problem is that I don’t know which way would work better to deal with this monster. I’m the only tester in the team, and there’s no plans to add another for at least another year (which will probably be another 2, then 3, then… ). If there’s another, better way, I’d love to hear it.

Ultimately my goal with this is to have a regression suite that’s checking known critical functionality so deployments don’t break it.

Technology Requirements:

The company is a Microsoft shop, so I’m using Visual Studio to write my tests. I’m currently running with MS unit test framework and Selenium because I’m pretty sure we will be upgrading our Visual Studio versions soon and CodedUI is being deprecated (more’s the pity – if you worked with it on a code first basis, it was/is a nice way to do things and allows an easy way to find elements with any arbitrary attribute. The UI Map on the other hand was/is a nasty thing. All my opinion of course). I’m coding with C# since that’s what the exceedingly slow migration to more modern code is using.

Other Requirements:

I need to be able to data-drive this beast. I can’t see any other way to handle tests of multi-page action flows (you have no idea how much I wish I could do this via API), or tests where I’d add a record to three or four different organizations, each with a different company configuration and therefore a different flow), or data entry tests that handle the different configurations.

It would be wonderful if I could wait for the new work to reach the app – the problem being that I’ve been here over 6 years now, and the team has been trying to get app modernization happening the whole time. Anything that does happen will happen at a glacial pace barring a massive change in how things are done.

Help?

Hello @katepaulk!

Given the provenance of this application, a description of monster seems kind. Determining a method or methods to evaluate it pales in comparison to my perceived fragility of it. While you have identified many testing risks and testability challenges, a larger concern is the growing technical debt.
In my opinion, any maintenance on the application appears to risk impacting dependent components. A regression test created at this point in time may help exercise primary functions and business behaviors; I would make it clear that testing alone will not protect the company from production issues.

Aside from my concerns, I thought that some automation of verifying the readiness of Company 1 might be a start (it seemed that method 3 touched on this but I was not sure). Using some automation to establish a good starting point and the same starting point should help the tests produce valid results.
I agree that checking critical functions is a great start. Perhaps they might be prioritized by business value. If there is a profile of which functions are used the most, it could help you determine where and how much time to spend on those functions.
Your suggestion of using Reflection is sound. I would use it with caution and to an extent that it assists in reviewing or verifying the critical functions. I agree with you that it could become its own technical debt.
Lastly, stored procedures are a testability challenge. I’m experimenting with DbFit (a library for Fitnesse) to explore stored procedures. Perhaps that is an alternative. One advantage is that, once available, someone else may be able to set up tests in Fitnesse allowing you to spend more time in other automation.

Joe

@devtotest - I was trying to be polite. Fragility is a massive issue, as is the continual increase of technical debt.

Just for reference, this monster is maintained and extended by all of 4 developers and me as the tester. We also have a non-coding team lead whose main focus is filtering requests to our team and subject matter expert. There are also three program managers who are mostly trying to keep the new requests low enough that we can make headway, but when C-level wants, C-level gets.

Verifying the readiness in some way is definitely a plan. I’m trying to come up with a way to keep the tests independent, without turning the automation into a mass of spaghetti and even more technical debt.

You’re right about stored procedures. Considering the thousands of them that we have, testing them is going to be… interesting. We’re trying to reduce reliance on them, but it’s slow going.

To be fair, everything involved with this application is slow going.

Have you got database access on this environment? It may be worth using the database to check whether it’s in the correct state rather than doing any kind of page interaction to discover it. Every page interaction is another chance for something to go horribly wrong. You’ve been there 6 years, so you probably know the database as well as the developers do, use that to your advantage if possible.

If the database shows the data is in the wrong state for running your test, go ahead and fire off some page interactions to get it into the right state.

In terms of how to make the setup fairly lightweight, yet maintain flexibility, I would be tempted to use builder pattern in order to make the intent clear, but abstract away all the complex page interactions.

Something like this (my c# sucks, so treat this like pseudo code:

The Ensure method would then handle all the setup based on the details provided in the builder. No messy setup code everywhere, just in one place.

You could use interfaces to handle alternate page flows because of configuration options as well. Different flow? Just pass the builder a different implementation of the page (which is why in the example the WithComponentXDetails methods have a page object as an argument.

I think this would still be relatively easy to create in a data driven way, just with far less or perhaps no reflection going on.

I do have database access and could feasibly use database calls to check that the company is in the state I need before I run the test.

The builder pattern looks like a good way to structure the setup - I’ll definitely have to try that as a way of handling complex prerequisites.

I’ll have to experiment with this - although I’ve just come to a screaming halt with the discovery that Dot Net Core doesn’t have the simple support for test data that standard dot net has. Honestly, being able to specify the data source then call TestContext.DataRow to pull my data is essential. I may have to play games with resetting my proof of concept code to standard Dot Net.