What are your go-to methods for achieving seamless data migration?

MoT Content Discussions
1

Here’s our latest article, " When your data moves house: Practical testing tips for clean data migrations" from @shad0wpuppet. This piece provides essential insights for testers working on data migration projects.

What You’ll Learn:

  • The key phases of data migration testing to ensure no data loss or corruption.
  • Hhow to use real-world data effectively in your testing process.
  • The importance of backward compatibility and how to test it.

After reading the article, we’d love to hear your thoughts:

How will you implement these strategies? Or what additional advice do you have for others?

3 Likes
2

A few things I feel could be better:

  • What role is the person that should be reading the statements mentioned in this article in? Which of the responsibilities are of the tester, which are of the team, which are of the lead of migration, of the product manager?
  • I’m usually the kind of tester that thinks negatively; Like: think of these risks, or these potential failures, or consider that thing X can go wrong in Y way, how about experimenting with this and trying to find this kind of problems?..
    So I’d like to see as well others look for problems instead of: ‘Check that the data looks fine’, ‘everything works’, ‘there are no crashes’, ‘application was properly handled’, ‘data will be handled correctly’, ’ data sets are correctly represented’, ‘migration commands and logic work correctly’, ‘application works with it as expected’, ‘features and systems work with migrated data as expected’, ‘Confirm that all data is migrated’, ‘Ensure compatibility’, ‘Ensure that data…’, ‘Check that everything works’, ‘ensure seamless software transitions’ …and many more.
  • I’d like to see a distinction between data migration, data conversion, and data integration. Without knowing much about the context of the examples I feel like they are learning toward data integration and conversion.
3

What I would add, is to look at the actual data migration activities and other supporting activities. For two data migrations I made a deployment plan.

Some of the things I looked at:

  • what and how are customers and suppliers inforped?
  • how long does it take to migrate the data?
  • are the settings of the environment right before, durin, and after migration?

More info about setting up a deployment plan:

1 Like
4

Answering your questions:

What role is the person that should be reading the statements mentioned in this article in?

It is mostly for QA Engineers/Testers or anyone else who planning similar activities or is responsible for the results and testing.

Which of the responsibilities are of the tester, which are of the team, which are of the lead of migration, of the product manager?

I’m not talking here about responsibilities, I believe that each team in its unique situation will deal with this on its own, it’s not so important. In my example, I work in collaboration with devs, with their help and support because as a team we were responsible for the result and quality.

I’m usually the kind of tester that thinks negatively; Like: think of these risks, or these potential failures, or consider that thing X can go wrong in Y way, how about experimenting with this and trying to find this kind of problems? . . . . . . .

Nice approach, generally good questions but I can advise you to use common sense, logic, etc. This is an article, more like an overview, general approach, this is not a course or book, even with such detailed and larger formats I won’t be able to cover all the details and nuances of each particular case, risk, situation, etc

‘Check that the data looks fine’, ‘everything works’, ‘there are no crashes’, ‘application was properly handled’, ‘data will be handled correctly’, etc…
– for me, it looks pretty straightforward in the generalized approach and I’m surprised that you expect here more details.

I’d like to see a distinction between data migration, data conversion, and data integration. Without knowing much about the context of the examples I feel like they are learning toward data integration and conversion.

Technically, I can agree here with you, but if I get data from one table/DB and put it in another one it sounds to me like migration. I used my real experience, and real tasks I’ve had and there weren’t so obvious distinctions between data migration, data conversion, and data integration. In real life, you in most situations also won’t have such a clear distinction but this is a valid point and I think it should be properly addressed in more comprehensive guides on data migration, data conversion, and data integration.

In my opinion, for the article format, this piece has enough details and makes it pretty useful as a starting point, and as a general checklist for similar situations.

2 Likes
5

A few things that have helped me in the past aren’t usually under a tester’s control. (A tester could suggest them, if they’re relevant.)

The first applies if you can migrate a subset of the data without things breaking. If so, then migrate the lowest risk data first. This could be data for e.g. accounts that have been dormant for ages, or are closed, or data that changes very rarely if at all.

Once that’s been migrated, you can test it before you migrate the rest. It reduces the time taken to migrate the higher-risk data, because you’re not also trying to migrate the low risk stuff.

You might need to do this migration of higher risk / more rapidly changing data in a few goes. If new data arriving during the migration will get missed by the migration, you might have to run the migration again to catch just the data you’ve missed. Even though this second migration will finish more quickly because it’s moving less data, it might still miss data that arrives while it’s running. So you might end up doing it several times, migrating less and less data each time until you finally catch up.

It also might be possible and/or necessary or desirable to change the relevant production code to temporarily maintain the old and new versions of the data in parallel. To start with, the old version is the source of truth and the new version is checked against it. Once you have confidence in the new version, you can switch so that the new version is the source of truth and the old version is kept up to date for a while until you’re confident with the switched-over version of everything.

1 Like