Top 5 Mistakes in Data Migration
One of the biggest trends in the data management market over the past 2 years has been the migration of legacy systems to cloud-native solutions. This is not to be confused with the early migrations to the cloud (2012-2016) which were often just trading a data center lease for a AWS “rental” agreement. These early migrations were mostly just moving the location of the existing systems. So instead of owning the data center, they would keep the same operating systems and simply “lift and shift” the datacenter to be managed by AWS, then later Azure and Google. However, this model was more of a method of offsetting the physical security of the servers to another entity and the cost savings for doing that was often in question. While organizations were doing this, the cloud vendors were sharpening their base offerings. As time went on, the native innovations in the cloud were outpacing the on-premise, or firewall-driven, solutions. While the difference in features and functions were slight in the beginning, they began to be compelling enough to entice organizations to convert. However, doing such a code conversion was not simply just a no-brainer migration, rather it required a translation of the existing code base to a new compliant language to the target technology. In the case of databases, this could mean converting all the SQL to be native to the target platform of choice; in the case of an ETL tool, it could mean converting all the legacy metadata (often in XML) to the target ETL technology’s metadata language. This is the state of the market today with over a billion dollars worth of code conversions in the balance.
Many organizations have been conducting projects to convert code within the last year or so. The need for automation was painfully obvious. There are many lessons learned which Telligens takes from client to client and here are the Top 5.
​​#5 CONDUCTING THE CODE CONVERSION BY HAND
With labor costs abroad so low in certain countries, a bruit-force code conversion is often a consideration many organization evaluate. The sizing appraisal for these efforts is almost impossible to measure accurately mostly because the body of code now has a body of people. The performance of the code conversion leans heavily on process controls to manage that body of people and what suffers is the consistency of the converted code. To state the obvious, one person converting a Stored Procedure may take a different approach as another person converting a stored procedure.
When conversion of code happens manually, the typical approach is to order the conversion based on what is easiest for the organization to test. Naturally in a Data Warehouse, organizations will want to convert code by subject matter. Thus when hand converting, the System Integrator will often reconvert the same code multiple times as they traverse each subject matter. However, this code will likely get reconverted by different people each time. So the patterns that simplify your deployment are completely lost in the end.
Conducting code conversions by hand is not recommended for anything over a 20 job pilot. This is one of those situations where cheaper doesn’t mean you get the same thing for less.
#4 NOT USING YOUR AUTOMATION FOR REPETITIVE ERRORS
Perhaps it goes without saying, but automation exists for these kinds of code conversion efforts. However these tools are not magic. They are pattern-based engines that see coding scenarios and provide analogous code in the target platform. Every migration is going to have oddities in the code that make it unique. Often during a conversion project, it can be tempting to meet a deadline by making manual adjustments to the code rather than making updates to the pattern engine. Organizations that do this often pay the price later when this pattern reemerges. Thus one of the steps during a conversion that is critical is to have a full analysis of the code patterns so the repetition of those patterns is fully understood. Organizations that do follow Sprints for running projects may need to be a little more practical to ensure they aren’t shooting themselves in the foot with a “quick fix” to meet a self-imposed short-term deadline.
#3 NOT FULLY DEFINING THE FUTURE STATE ARCHITECTURE AHEAD OF THE CODE CONVERSION PROJECT
As mentioned earlier, the features and functions of these new cloud solutions is surpassing what the on-premise solutions can accomplish. However implementing these new platforms requires a plan in place for both how data will land in the cloud and how it will be consumed from the cloud. So before the code migration gets started in earnest, a Solution Architecture needs to be defined. Additionally, many of these new database solutions require a well-defined Access Control plan to use some of the new features for data sharing and mixed Data Lake/Data Warehousing use cases. This will also impact where integration code execution occurs after the code is converted.
#2 NOT DEFINING THE PRODUCTION PROMOTION PROCESS
Organizations often have managed services for maintaining their production environments. Along with this, there are often gate-keeping measures for how to promote something to production. Organizations need to dust off their requirements for production promotion so they understand what all goes into that process. For example, it could be that there is an enterprise scheduler that is responsible for executing jobs which injects code into the SQL, and thus needs to be accounted for in the end converted code. The code conversion project team needs to open a dialog with the production promotion teams to ensure the two are aware of the processes which will be eventually intersecting.
This, by the way, is part of the reason code conversions require adaptable automation tooling. There will be a surprise in every code conversion project. There’s simply no way of encapsulating 20 years worth of code into an end-to-end requirements document.
#1 INJECTING TOO MANY ENHANCEMENTS DURING THE CODE CONVERSION PROCESS
There is no universal consensus that says you can’t capture technology advancements in the target solution during a conversion. After all, the code patterns can be defined in any way. However, don’t forget that the organization will need to ultimately conduct a data test of all the converted code. Should the data not match, where is the problem? If the code is highly modified from the original code, it could inject complexity at a moment where it’s least needed. If there are too many upgraded dependencies, it can inject a testing nightmare into the conversion. So non-critical enhancements are best left after the conversion project is complete. In Telligens’ past experience, most performance enhancements happen natively simply through the adoption of the more capable indexing and query engine in platforms. If there are stored procedures or other aspects that can be optimized, we recommend conducting them after all the data testing has completed. This ensures that the milestone of a successful conversion is not in doubt, and the organization can make edits moving forward.
WHAT MAKES TELLIGENS DIFFERENT?
While Telligens conducts highly intricate and complex data management projects, Telligens is first and foremost a Business User Centric consulting company. Our internal slogan is to Simplify Complexity. This means that we take complex data management challenges and not only make them understandable to the business but also make them easier to operate. Intricity does this through using tools and techniques that are familiar to business people but adapted for IT content.
TALK WITH A SPECIALIST
If you would like to talk with a Telligens Specialist about your particular scenario, don’t hesitate to reach out to us. You can write us an email: letstalk@telligens.com
