Don’t forget your data

Companies involved in major IT projects or mergers and acquisitions should ignore the ABC of data delivery at their peril.

All the dominant business models of recent years have called for increasing integration between computer systems.

For example, ERP or CRM solutions require far greater contact between front and back office functions and for e-business systems to be a success, they demand a high degree of integration in an organisation’s application and information infra-structure. In parallel with these developments, the last decade has seen a massive rise in take-over and merger activity that continues to place great challenges on IT support.

But whether it is a data warehouse, ERP implementation or systems integration, the single common denominator is data. Without accurate, reliable, high-quality data all of these major IT initiatives will be compromised and not deliver on their full potential.

But the problem is that data is not very ‘sexy’ and in most cases it is a headache for both IT directors and the application vendors. Maybe that is why the subject often gets sidelined or brushed under the carpet by eager sales teams pushing the benefits of CRM or the compelling business case for latest version of SAP.

Before you know it, the deal is done without any mention of who is going to deliver the data and how – despite the fact that it can account for up to 80% of the effort and 50% of the cost of a major IT project. Eventually, either the customer gets lumbered with the task or the vendor – who probably knows a lot more about applications than data – takes it on at additional cost. So, it is not surprising that data delivery is often the reason why projects overrun on time and budget.

All application programs are fed by data, from a variety of sources and in an increasing variety of formats. Disciplined IT users have always tried to build consistent data models, but their efforts have been undermined by a number of factors. For example, the traditional distinction between front and back office systems and between operational data processing and business intelligence as well as the idiosyncrasies of generations of programmers.

The ability to deliver data to an application, from whatever the source, in a format in which the application can use, is absolutely fundamental to the operation of information systems. As the saying goes, ‘rubbish in, rubbish out?’ But it is seldom straightforward. Imagine, at a railway junction where tracks of different gauges terminate, trying to disembark a carriage-full of people and their bags, marching them down a platform and settling them into a carriage of different dimensions in exactly the same arrangement as previously.

This challenge in IT terms has been one of the last great bastions of manual effort. The solution for most IT managers is to throw people at the problem. As new and old applications need to exchange data, the tendency has been to hand-code the interface. But eventually, organisations succumbing to this approach find the interfaces proliferating. Their data models are forced into contortions and the number of exceptions begins to exceed the instances of standard conformity.

Alchemy Migration Systems refers to the ABC of Data Migration – one of the most complex, least glamorous and potentially most critical areas of enterprise computing. ABC stands for Analysis, Building and Cleaning – the processes applied to data to make it fit the purposes of the target application. The abbreviation involves a little oversimplification but there are three distinct stages involved. Any single data migration tools vendor that says it has the end-to-end solution is being economical with the truth.

However, Alchemy lives and breathes data having developed an end-to-end solution by bringing together the best technologies available for each stage. Underpinning the solution is a framework and methodology within which products in each phase of the data delivery cycle will work together to produce an optimal result. In the ABC of Data Migration, A for Analysis simply means studying the size and scope of the problem, in discussions with business and IT professionals, as a first step. This has traditionally been a haphazard manual process. Instead of studying the documentation or the metadata, automated tools focus on the source data itself to build a detailed understanding of content, structure and quality and to produce a model featuring the business rules and relationships associated with types of data.

This makes it possible to ask very precise questions of IT and business people, to confirm the relationships and rules and to generate a full understanding of their interaction. Identifying data problems and issues early on has a dramatic impact on reducing the time of overall project path.

Understanding the data makes a huge practical difference to B, the Building procedures. This stage at which data is carried forward from one environment to another usually involves cycling through a number of iterations. However, good analysis cuts this down to two or three iterations, when otherwise 30 is not an uncommon total.

The third stage, C for Cleaning, is about ensuring the integrity of the migrated data. It is a series of quality control checks. Typically, five per cent of the data may not fit a business rule it is supposed to, often because of an undocumented spur of the moment programming fix to a one-off problem. Cleaning is the last stage of data re-engineering and again there are a number of optimised tools that do the best job possible.

We make no bones about the absence of glamour in data delivery but we are passionate about its importance. With up to 80 per cent of a project cost going towards it, if you get it wrong it can quickly take away the benefits of the final application. Data should always be a key business driver and on the critical path of any major IT project. Managing data is a bit like handling an oil tanker.

Get it on the right course from the start and you can enjoy the ride but if you get it wrong, it takes five miles for an oil tanker to change course. So, if you want to be very sure in advance that the course is the right one, make sure you follow the ABC.