Database Migration – Lucrative or Lethal!

As the pace of change continues to increase companies strive to improve their business and gain a competitive edge. This will often require the business to advance its information systems and in many cases they will need to migrate data from a core database to a new system. However, a lack of understanding or poor previous experience has fostered a popular misconception that database migration is expensive, time consuming and disruptive so many organisations either migrate only a limited amount of data or avoid moving to a new system until there is such a pressing business need that they have no choice. Nevertheless, a successful migration of valuable historic data can bring substantial benefits. If executed correctly, it can offer real business advantages and rewards but get it wrong and the benefits which the new system should provide may be severely diminished.

Historic data enables organisations to understand the present and helps them to forecast the future so data built up over a prolonged period is a valuable asset which most companies will want to retain. However, migrating this data to a new system presents many business challenges which must be well managed in order to minimise disruption and enable the business to continue functioning:

• Internal staff will need to work closely with external parties in order to complete the data migration project within pre-defined time lines
• Workloads are likely to increase, which may lead to an increase in staff stress levels and
• Specialist skills will be required to access the data held in antiquated systems, these are often rare and can therefore be expensive.

There are also many technical challenges, such as:

• Data structure inadequacies - meaning that some source data will have no obvious location in the target database requiring time to work out how, and even if, the orphan data can be dealt with.
• Poor legacy data requiring extensive data cleansing.
• Legacy data having to be merged into a live data set.
• Limited or out of date documentation making it difficult to understand the structure of legacy data.

Data migration projects commence with gaining a thorough understanding of the source data. This is commonly done as a two stage process: data profiling, which involves gaining an understanding of the content, structure, quality and integrity of the source data and mapping - whereby a detailed map of the data is drawn up based on the profile.

Profiling legacy data is generally complex, time consuming and prone to errors, as such it is crucial to the success of the project. A lack of understanding of the source data will inevitably lead to an inaccurate data model which will translate into problems with the new database. Additional work will then be required to revise the data map resulting in time and cost overruns or in extreme cases the project being cancelled and the company living with a partially functional database.

For some considerable time statistics have shown that eight out of 10 database migrations have suffered budget overruns or were not delivered on time. One of the primary reasons for this is that companies underestimate the amount of effort needed to gather, cleanse and standardise legacy data that is to be migrated into a new database. This is largely due to the fact that many business leaders misunderstand the data migration process and view it as a purely technical issue. They fail to realise that the IT department does not have a sufficient understanding of the applications or the data to deliver what is required by the business. Since it is the business leaders who define system requirements based on business goals, it follows that they should take responsibility for driving the project. Basically they should have a good understanding of why the data is being migrated, what should be migrated and when it should be migrated.

Data migrations can also be prolonged by overestimating or not understanding the quality of the source data. Whilst an organisation will naturally strive to achieve perfect data quality the quest for data perfection can have a negative impact on the time and resources required to complete the project. As such a realistic level of data quality should be decided upon at the outset and communicated to the project team so that sufficient time and budget can be allocated in order to achieve the required standard.

Reporting is an area which is often overlooked in a database migration project. Legacy system reports will invariably have embedded value to an organisation so it is important to understand how the business wants to report on the data in the new system. In some cases it may not be possible to reproduce some of the reports generated in the legacy system due to differences in the two data structures. Businesses reporting requirements should therefore be carefully documented so that sufficient resource can be allocated to this area of the project.

Migrating data can be a fairly time-consuming process; however, the benefits can far outweigh the cost. The main benefit of a complete data migration is that the organisation retains use of all of its valuable historic data, however, the migration also provides an opportunity to check for accuracy and inconsistencies, remove duplications and generally cleanse the data being migrated into the target database. This will lead to performance benefits in the new system and provide improvements in efficiency translating into lower staff costs and providing the business with a competitive advantage.

By adopting a structured approach organisations can migrate all relevant data in line with agreed project timelines, without substantial expense and with minimal disruption to the business thereby disproving previous misconceptions.