We talk often, here, about Big Data and Data Transformation and how it's such an important part of a seamless, and streamlined, business operation strategy. An important and exploding use case is Data Migration.
All true – it's not new of course.
Data Migration has been around since before the first Apple Mac Plus. The difference is that today, everyone is affected: startups, small business, mom and pop shops all the way up to enterprise. Whether we're updating or upgrading a point of sale, migrating a website, changing an upstart's ecommerce platform, or setting up newsletters; somewhere along the way, those 500, 5 000, or 50 000 products, or users/subscribers, or blog articles must be kept reasonably intact for your community and client base to continue to move and grow with you.
As with all things business, Data Migration must be valued by all the standard benchmarks of productivity, business analysis, reporting, trends, and opportunity.
Data migration is a process to move data from one place to another. Data migration may be part of a broader series of steps, integration, and automation. For the purposes of this discussion we'll focus on one-time, or infrequent Data migrations.
There will always be preparation steps in planning what is going to move, and where it is going to move to; ensuring that when your data gets to its destination, it will be appropriately usable. Some of these steps will involve Data Transformation (which we have discussed elsewhere in the site and other articles), validation, and in some cases, corrective sequestering or staging.
Legacy investments are too often thrown away, especially where data is involved or simiarly esoteric abstraction concepts where C-level managers may not see the connection to keeping and maintaining data-as-a-business-value as part of their ongoing strategy. "Legacy", while its definition could be reinterpreted a thousand ways, could broadly be argued to mean any system, platform, or format, that is currently operational, but is likely to become less required, unsupported, or otherwise irrelevant in the foreseeable future.
As with all things business, Data Migration must be valued by all the standard benchmarks of productivity, business analysis, financial reporting and opportunity:
If this looks familiar, it should; resembling a Software Maintenance Programme plan. Starting, we have cost drivers, operational levers, and efficiency. As we move down, we gain agility, speed, opportunity, growth drivers, competitive advantages: the kind of advantages businesses realize when they unlock all the data at their disposal.
Looking at the four key benchmarks, businesses should look at achieving Data Migration via manual methods. "What would it take to get X information re-data-entered in our new system?" If the answer exceeds 40 hours, there may already be a good case for investigating a Data Migration automation project.
Your team's training, documentation, and ongoing core competance must keep in line with the value of your systems. How difficult is it for your team to use, re-use, train new staff, and perform well with the various disparate platforms you have? There is a value to keeping productive teams actively aware of legacy platforms - are you at a point where that value is exceeding the advantages of a migration to a newer platform where your team is already well trained and up to speed?
Envisioning and unlocking new value opportunities can be one of the longer term processes in understanding the value of Data Migration and Transformation. Quite often the reason rests with the very model that the legacy platform allowed painting your business deliverables into a box. That box was likely sufficient for its time and customer expectations. That box may be limiting your vision moving forward, however, it's limitations you want to unpack and throw out in order to see beyond and what more you can do. If monthly compiled reporting was sufficient for customers before, would real time KPI access prove more powerful? If batch-compiled data required 10 person-hours of back and forther to validate an application, could fully interactive Data Migration and Transformation achieve this in 1/10th the time? Saving not only internal labour, but errors, and closing time for new accounts.
Your very value proposition can be transformed by understanding and embracing how Data Migration and Transformation affects not only internal workflow and customer service processes, but additional services you could offer, new instances where customers may find reasons to spend more money for higher and higher level value you can now deliver that you couldn't before.
Where the data is coming from, what it contains, how it's structured, and how it must transformed to fit the new platform is key when Assessing your original data.
Design, in this case, refers to data mapping and transformation design of the solution. Is this one-time, infrequent, or real time? This is another aspect of solution design.
The Build will reflect not just the migration and transformation required and mapped, but also the robustness, performance, and timeliness of the solution requirements, by its accuracy, validation, and sheer performance and power depending on the volume involved.
Staging, or Testing, is central to gaining stakeholder alignment. By providing a staging area to monitor, validate, and re-test, you have a proving ground to understand and envision how the model will operate; and if necessary, make adjustments before committing.
Process is the carefully planned real execution of your import and transformation.
Audit, is essentially the ongoing testing of the data migration and transformation. The mission critical nature of your data or platform will dictate how important it is to ensure that you are prepared to review, assess, evaluate, and correct the quality and performance of your data migration and transformation strategy.
Be extremely clear on the how of executing your project.
Be extremely clear on who is responsible for what skillset of tasks. Are they properly equipped to be successful?
Be extremely clear on validating the data source. What amount of automated vetting is required to ensure that the source data is reliably and predictably accurate.
Be extremely clear on how errors are managed. Expect that your source data is going to cause a failure of some kind. Due to manual handling, legacy characters, missing fields, one way or the other is likely to cause some form of discontinuity, therefore you need to have a plan for how tests (see "tests" above) and ongoing validation will manage such gaps gracefully.
Understand how changes to the process, model, or adding new or changing data sources may affect your process, stakeholders, and budget. How important is security to this plan? If speed, volume, scale, or any other aspect of performance is central to success, has your team evaluated the server, platform, and technology proposed to run your process application? Is the proposed solution adequately prepared to managed the volume and flow you need to be successful?
Engaging business leaders in this process will improve the chances of smooth and predictable success. The reason is that understanding the value proposition of Data Migration and Transformation planning, systems, and scale for the business is extremely important to value what level of resources are significant for broad business objectives.
Having business leaders involved, helps to flatten decision-making not because we want to bring executive decision-making down to minutae planning, but precisely because we don't. This is what will help them scale up and away from operational data gaps, which helps to demonstrate, business operations that will be improved, best practices leaders can take away, demonstrate value propositions of more or less automation and speed. Seeing how value can be driven may spur more engagement and a richer data planning culture. Similarly, if a Data Migration and Transformation plan is not as critical, it can be scaled back.
Business value and time to realize ROI is key for leaders and all stakeholders to embrace the value proposition of a Data Migration and Transformation process, project, or solution. By understanding the scale of opportunity, whether smaller and simpler, or larger and complex, a clear perspective on value proposition is empowering for leadership to understand how much they can drive value.