Fast forward to the present, and while the work performed by outsourcing companies is still considered valuable, so is the data generated by that work. Our ornate supply chains mean that vendors of vendors to your immediate vendors hold important information about the business. This knot of relationships can make it very difficult to collect and make sense of that information. Even something as simple as finding out how many of your partners rely on a single source of raw material could take months. It can’t be easy to know that a failure of one company several relationships removed could impact several relied-upon partners, and you wouldn’t even know it.

E Pluribus Unum

Getting this data from suppliers can often be quite challenging since they don’t want it used against them. It is important to stress that the findings from analysis are likely to benefit their entire business, not just your account, and that partners that share data are far harder to part with than transactional vendors that keep customers in the dark.

Ultimately, they should be willing to report this data and more to keep their customer, but they are not likely to follow a standard format, and for good reason. It isn’t practical to keep their records in formats that work for every company upstream. That means that each company’s data tends to live in its own silo, and the manual process of getting a single view of what’s going on across the supply chain is challenging at best. It’s far more likely that much of the intelligence spread across these companies seems destined to rot away unused.

If this data is going to be useful, it needs to be harmonized, and quickly. Suppliers update their bills of materials and pricing constantly, making a long data preparation process an enemy in the battle for sound analysis. It may be comforting to know that this problem isn’t unique to companies that outsource. Even companies that are vertically integrated often have important information in a variety of formats. Companies have been working out strategies to pull heterogeneous data together for some time, and have beaten down paths for others to follow. One of the most effective strategies for this particular problem is data unification, a marriage of technology and human intelligence that erases the barriers between data sets so analysts can turn the din of hundreds of data sources into a chorus of information.

Don’t Move Data; Understand It

Specifically, data unification solves two problems with data source diversity. First, it applies technology guided by human intelligence to the challenge of pulling together data in diverse formats. Second, it retains the intelligence on how to treat each data source, so analysts can repeat their work as the data changes.

Traditionally, data integration was performed by hard labor, investigating static data sets by hand, pulling data from various sources into a static master data set used for the analysis. It isn’t difficult to imagine the work required to combine reports from every company in a typical supply chain. This effort to prepare data for analysis often took up to 80 percent of the analyst’s time, and the end result could only be used once – often being completed after the data had already become outdated.

Further, decisions about what the data referred to were left for the analyst to figure out. Conversions between currencies and measurements, combining close but not exactly matching information (Apex, Apex Co. and Apex Inc. may or may not be the same company, for example) were left for someone who didn’t live and breathe the information. Human errors had to be accepted as their best effort because tracking these conversions down added too much time to an already too long process.

In contrast, data unification looks at the data in its various formats and learns how to combine it for use as a whole – without taking the data out of its source file. Where the analyst can’t decide how data should be treated, experts are identified and called in to make the call in simple yes or no questions, and that information is added to the recipe for making the data work together.

How Data Unification Works

Data unification is a two-step process. First data sources are cataloged so analysts have a complete view of what sources are available for any analysis. This does not require that the data all reside in the same location. All that’s needed is a record of what is in the data set, where it is and who owns it, so that person can be among the experts tapped should questions about the data arise.

Second, the data is reviewed – typically by software – to identify fields with commonality. For example, it can find where company names go in each set, and scan each column for trends. If 20 percent of the mentions of Apex are “Apex Inc,” or variations like “Apex Inc.,” it’s easy for the analyst to spot that and ask an expert for clarity.

Once the data has been organized, the analyst saves the steps for organization – not the data itself. Any time a new data source needs to be added, the steps for adding that data to the finished set are added to the mix. Because the analyst doesn’t output a final data set, the same process can be done any time. Other analysts can even use relevant work done by the first analyst for a new project without re-inventing the wheel. As a result it’s not only faster the first time than manual data preparation, but even faster to repeat.

Outsourcing With Vision

Using data unification, it’s easy to regularly answer the kinds of questions your entire supply chain might ask if it was one department, and know if there are outsized risks buried under layers of supplier relationships. You can identify the best sources for common products and push upstream partners to improve their sourcing. You can avoid overdependence on single sources. You can establish benchmark pricing for not just your vendors, but their vendors. It’s the kind of vision that makes supply chain management a core competency itself.

Source: Tamr Inc.