Supercharge Your Data Migration Plan with Process Automation
Steve Winkler | October 26, 2016
The term data migration can refer to data migration per se — a one-time movement of data from a legacy system to a new platform — as well as more general, ongoing processes for moving information back and forth between the new and old systems.
In either scenario, automating processes is absolutely essential for successful project execution. Whether it’s a one-time data migration project or any large-scale repetitive transfer of data in the enterprise, there are four key reasons why process automation is critical:
Most data migration processes will be complex. Data is going to come from lots of different sources, and map to many potential targets. Target data structures need to be created, indexes created, dependencies managed, in addition to the extraction and load activities.
In addition, the number of distinct steps can be extensive. For government agencies that are migrating significant amounts of legacy data into a modernized system, the migration process may comprise thousands of individual activities and steps, each of which needs to be sequenced and checked for errors. But by automating some or all of the process, with activities running in parallel, it can be implemented without delays as one job, with one command — and then it’s done.
Data migration is always a time-sensitive operation, and migration windows usually involve some type of outage. No one on the mission side likes to hear that. So it’s impossible to overstate the importance of optimizing the speed of the entire migration process.
When you have terabytes of information to migrate from a legacy system, it’s essential to be able to assess the accuracy of existing information and fix it wherever possible before you migrate it. You simply can’t take that amount of data and manually review it quickly to identify and resolve anomalies, possible conversion issues, and information that does not comply with target requirements. A data migration plan that applies automated scanning tools can look for anomalies and correct them, while minimizing or even eliminating any impact on the migration window.
Process automation is also a key quality factor. For very complex migrations involving lots of activities and steps, process automation takes human/operator error out of the equation. No one wants to find out in the 20th hour of a 24-hour migration window that someone missed a critical step in the 2nd hour that negatively affects the rest of the migration process.
Let’s face it: unless you are a very brave individual, you are not going to try to migrate your data for the first time during the production cutover window. No matter how completely you evaluate and scan legacy data, you’re going to come across data you did not expect. And don’t forget, just because you looked at the data yesterday, does not mean that you’ve seen it in its final state prior to the migration. After all, your users are still using the system, and new data could be added at any time. Unexpected obstacles and data anomalies are going to occur, especially in complex, older, or proprietary systems, so it helps to test your process thoroughly before you go live.
Also, before you go live, you should have repeatedly migrated test databases as well as production data. Having the process automated allows you to perform this necessary repetition with minimal resource overhead.
4) Stakeholder buy-in
Outage windows are not the only things that will keep key stakeholders up at night during modernization. They also want assurances that when the migration is complete and users begin to use the new system, the information that was there when they logged off of the old system is correctly reflected in the new system. Again, when data volumes are large, neither the integration team, nor the entire collected user base for that matter, could reasonably be expected to check all the data in the new system. Fortunately, a data migration plan that leverages process automation can overcome this challenge by utilizing tools and techniques to verify post-migration data against its pre-migration state. Use the results of the verification to provide stakeholders with detailed reports confirming that the post-migration data meets expectations.
The choice is clear
It may seem like a lot of additional work to automate processes in your data migration plan, but failing to do so would be “penny wise but pound foolish.” By automating the process of data migration, you can get to your modernization goals far more quickly and efficiently — and at the same time, end up with better, more reliable data.