Good Ways to Write Automated Regression Tests for Ugly Apps

🗄️ Archive
,
1

The Background:

The app was first created (spawned?) in 2002 as a classic asp web application that included faux object orientation (the only way classic asp could do anything of the sort) and database interaction. It has since… grown. I estimate over 1 million lines of code including the logic buried in stored procedures, logic in the “objects”, logic in the main asp files including a ton of Javascript, embedded <% %> tags and nothing that could possibly be unit tested.

It currently takes me around an hour for a manual smoke test, so obviously I want to automate some kind of smoke test.

Which is where the fun comes in…

The Problem:

This monstrosity is also loaded with dependencies. It’s a multi-tenanted application, which means if I wish to edit record A for company 1, I need to start by logging in as a user who is allowed to edit records in company 1 (the easiest condition, since nothing happens in this app unless someone is logged in). Then I pick the company I’m working in. Again, basic.

Now the “fun” begins. I need to be able to check or ensure that company 1 is in fact in an editable state. Since I can’t start with a clean system (1/2 TB database and growing), or even be sure that nobody else has changed my test company before my automation starts, I have to check the company state, and if it’s not in an editable state, put it into one.

Similar conditions apply to any action I would take in the system.

Now, I’ve worked with ugly apps like this in the past. The method I usually found myself working with (Method 1) was a monster case statement handling the test running, where each test in a CSV file specifies the type of test it is. The end result is a csv “database” that comes to somewhat echo the actual database and ultimately gets overblown and difficult to change.

It also causes problems when one of the early tests doesn’t end correctly because that can put the whole thing into an unstable state, and it can be challenging to work out what actually failed when unless it’s architected very carefully.

Method 2 is slightly less ugly (code-wise) – namely using reflection (I’m working with C#, so there’s no “eval” type statement I can use) to invoke the method named in the input CSV file. As far as I can tell, the only advantage of this method of managing the dependencies within the app is that instead of a humongous case statement I end up with a modestly-sized invoke method. The disadvantages include the code becoming even more opaque and a speed penalty.

Method 3 is to create custom attributes and use these to define what is required for a specific test method. That way instead of the edit record test method having massive chunks of setup code (that’s repeated for the delete record test method and the add record test method), it would have one or more prerequisite attributes (such as “CompanyStatus” == “Editable”) and a call to check attributes then if prerequisites aren’t met, calls to make the changes needed to meet said attributes. This keeps the test methods self-contained, but will slow the overall run since each test handles its own dependencies.

My problem is that I don’t know which way would work better to deal with this monster. I’m the only tester in the team, and there’s no plans to add another for at least another year (which will probably be another 2, then 3, then… ). If there’s another, better way, I’d love to hear it.

Ultimately my goal with this is to have a regression suite that’s checking known critical functionality so deployments don’t break it.

Technology Requirements:

The company is a Microsoft shop, so I’m using Visual Studio to write my tests. I’m currently running with MS unit test framework and Selenium because I’m pretty sure we will be upgrading our Visual Studio versions soon and CodedUI is being deprecated (more’s the pity – if you worked with it on a code first basis, it was/is a nice way to do things and allows an easy way to find elements with any arbitrary attribute. The UI Map on the other hand was/is a nasty thing. All my opinion of course). I’m coding with C# since that’s what the exceedingly slow migration to more modern code is using.

Other Requirements:

I need to be able to data-drive this beast. I can’t see any other way to handle tests of multi-page action flows (you have no idea how much I wish I could do this via API), or tests where I’d add a record to three or four different organizations, each with a different company configuration and therefore a different flow), or data entry tests that handle the different configurations.

It would be wonderful if I could wait for the new work to reach the app – the problem being that I’ve been here over 6 years now, and the team has been trying to get app modernization happening the whole time. Anything that does happen will happen at a glacial pace barring a massive change in how things are done.

Help?

2

Hello @katepaulk!

Given the provenance of this application, a description of monster seems kind. Determining a method or methods to evaluate it pales in comparison to my perceived fragility of it. While you have identified many testing risks and testability challenges, a larger concern is the growing technical debt.
In my opinion, any maintenance on the application appears to risk impacting dependent components. A regression test created at this point in time may help exercise primary functions and business behaviors; I would make it clear that testing alone will not protect the company from production issues.

Aside from my concerns, I thought that some automation of verifying the readiness of Company 1 might be a start (it seemed that method 3 touched on this but I was not sure). Using some automation to establish a good starting point and the same starting point should help the tests produce valid results.
I agree that checking critical functions is a great start. Perhaps they might be prioritized by business value. If there is a profile of which functions are used the most, it could help you determine where and how much time to spend on those functions.
Your suggestion of using Reflection is sound. I would use it with caution and to an extent that it assists in reviewing or verifying the critical functions. I agree with you that it could become its own technical debt.
Lastly, stored procedures are a testability challenge. I’m experimenting with DbFit (a library for Fitnesse) to explore stored procedures. Perhaps that is an alternative. One advantage is that, once available, someone else may be able to set up tests in Fitnesse allowing you to spend more time in other automation.

Joe

3

@devtotest - I was trying to be polite. Fragility is a massive issue, as is the continual increase of technical debt.

Just for reference, this monster is maintained and extended by all of 4 developers and me as the tester. We also have a non-coding team lead whose main focus is filtering requests to our team and subject matter expert. There are also three program managers who are mostly trying to keep the new requests low enough that we can make headway, but when C-level wants, C-level gets.

Verifying the readiness in some way is definitely a plan. I’m trying to come up with a way to keep the tests independent, without turning the automation into a mass of spaghetti and even more technical debt.

You’re right about stored procedures. Considering the thousands of them that we have, testing them is going to be… interesting. We’re trying to reduce reliance on them, but it’s slow going.

To be fair, everything involved with this application is slow going.

4

Have you got database access on this environment? It may be worth using the database to check whether it’s in the correct state rather than doing any kind of page interaction to discover it. Every page interaction is another chance for something to go horribly wrong. You’ve been there 6 years, so you probably know the database as well as the developers do, use that to your advantage if possible.

If the database shows the data is in the wrong state for running your test, go ahead and fire off some page interactions to get it into the right state.

In terms of how to make the setup fairly lightweight, yet maintain flexibility, I would be tempted to use builder pattern in order to make the intent clear, but abstract away all the complex page interactions.

Something like this (my c# sucks, so treat this like pseudo code:

The Ensure method would then handle all the setup based on the details provided in the builder. No messy setup code everywhere, just in one place.

You could use interfaces to handle alternate page flows because of configuration options as well. Different flow? Just pass the builder a different implementation of the page (which is why in the example the WithComponentXDetails methods have a page object as an argument.

I think this would still be relatively easy to create in a data driven way, just with far less or perhaps no reflection going on.

5

I do have database access and could feasibly use database calls to check that the company is in the state I need before I run the test.

The builder pattern looks like a good way to structure the setup - I’ll definitely have to try that as a way of handling complex prerequisites.

I’ll have to experiment with this - although I’ve just come to a screaming halt with the discovery that Dot Net Core doesn’t have the simple support for test data that standard dot net has. Honestly, being able to specify the data source then call TestContext.DataRow to pull my data is essential. I may have to play games with resetting my proof of concept code to standard Dot Net.

6

Hello @katepaulk!

I have been in your situation once already, so I know how it feels. Besides all the things that the others have mentioned, there is a very important thing I would keep in mind: the situation is bad enough. Don’t make it even worse by complementing the existing technical debt with a growing testing debt. So whatever solution you find, make sure it is scalable and sustainable. So for example, having the 1/2 TB of database makes the testability hard, but I would still rather make a duplicate of this database for every single test execution and have a strategy how I can gradually shrink it (by learning more about the domain model and building more automation functions to generate test data) than establishing a testing practice that is built on the single, shared test database (where every change is either super costly or causing unwanted side-effects).

The technique I was using was the following: I have made the entry points for setting up the different preconditions (like: Create_an_order or Create_a_user). Initially I have implemented them by finding good concrete records in the existing database that satisfied my needs (let’s say order Nr 2345 looked good, user “Marvin” sounded like a good test user). With those I was able to make my first tests. As I moved on by covering more parts of the application, I learned how to create a new user and replaced the hard-coded ID with the creation code and deleted all user records from my test database. Later I learned and implemented the logic that is needed to create and order, so I could again replaced the hard-coded ID with the creation logic and I could also delete all existing order-related records from the test database. And so on. At the end, I was able to shrink the test database to a few megabytes and so all my existing tests got faster (without needing to touch them).

This is just one example, but as I said, the general idea is that long-term strategy and sustainability is more important than having “quick success” in the beginning but no way to improve later.

My other advice: you should not solve all the technical problems alone (like which method of this three would be the best). Ask the devs for help. They probably have a good overview of many useful libraries and framework that work in your context, but what is even more important, by asking help from them, they feel better responsible for and have more respect to the testing work that you do.

7

Good points, all.

I would love to be able to restore a known-good database as a starting point… What is happening is that in between all the other work we’re trying to do (things like move everything on a Windows Server 2003 system to something a little more modern before we start on the SQL server 2008 instances and then the Windows Server 2008 instances - which has to be a joint exercise between network services and us because we’re dealing with equally creaky old software that might need modification to run on something newer), we’re putting together a dedicated test automation instance that will be set up to sync code and database (without data) from either the test or the staging server. With an automatic sync, the code won’t get out of date, and I’ll have something that’s half-way clean for regression purposes.

What I’m doing at the moment is spawning a variety of proof-of-concept projects to play with and see what works best for what I’m trying to do. Your method of gradually reducing the database to a workable minimum is a good one - I’ll definitely be looking at that long term.