Data Management Significance in IOT of Manufacturing Industry

In the previous blog, we discussed IOT (Internet of Things) impact to consumers and challenges in data management. This blog focus on the various industries impact of IOT and how it is revolutionizing them. Later we also discuss the importance of applying the data management principles for improving the outcomes.

Consumer IOT made the significant impact by providing valuable services to customers with many smart devices makes life easy for the users. But commercial usage of IOT adaptation might be slow but there is a potential to disrupt and revolutionize the industries and solve many of the business challenges they never had solutions in the past. As shown in the visual of ice burg the Industrial/commercial IOT   is very large with many solutions like smart sensors, smart buildings, smart factories, digital supply chains.

Industry 4.0

 The world of manufacturing is going thru major disruption and revolutions which is called as  Fourth Industry revolution.

 First industry revolution which happened till the eighteenth century where availability of steam and water. Availability of steam provided the first mechanization of the world. Hand production methods moved towards to machines. Textile production improved productivity, steam machines provided better transportation . Second industry revolution started in the early nineteenth century with innovation and adaptation of electricity and improvement of the manufacturing process of assembly lines. This revolution moved towards mass production of goods. Availability of transportation facilities to provide these goods  to sell across the country or even world.  The assembly lines process managed might be a paper-based process but still efficiently deliver product in mass scales like hundreds of cars in a month or thousands of soup cans. The mass production process even went in the agricultural process of invention of better seeds by GMO, drift irrigation, supply chain to reach product in the market intern able to reach all kinds of foods on everybody's kitchen tables.

After the invention of the computer and software started third industrial revolution improved the industrial engineering to next phase of improvements. Moving paper based process into computer applications, automation of process all improved overall performance which improved output. Automation of the production, which improved productions worldwide and made mass production more efficient. Now the world of manufacturing going thru next revolution. The fourth industrial revolution, also termed Industry 4.0, is being characterized as an increasing digitization and interconnection of value chains and business models. Industry 4.0 creates Smart Factories and is based upon cyber-physical systems allowing the manufacturer to control the entire production from one platform.   Industry 4.0 involved achieving total connectivity with IOT (Internet of things) which also called industry internet.  Industry 4.0 makes the existing manufacturing applications increase the complexities in very significant level which involving adapting new machinery or changes to existing physical systems. With the emergence of big data platforms where they can collect the massive amount of data and as well as intelligent devices which can collect and send a large amount of data paired together comes with the new generation of software applications which are a lot more intelligent with least amount of human interference.

In short, whether it is a jet engine, turbine, commercial conveyor belts when connected with smart sensors sends data in real time can help to track, optimize, predict what to repair, when to repair and even when to replace. This will change how manufacturing industries operate be supporting their large and expensive machines. For example, GE who is known to build large Jet Engines, Turbines are looking to digitize their operations to improve their productivity as well as their customers.  Also Electrical engines in cars with many smart sensors taking cars close to driver-less cars.

Smarter Devices /Machines

 Many existing Machines are integrated with external IOT devices which collect a lot of information and send to central IOT application. A smart Nest thermostat will help to operate the HVAC systems more efficiently and also collect information about Air conditioner performance. Next level of HVAC systems is coming with inbuilt IOT chip which can help to operate the system, collect health data, diagnosis and selfheal the systems.  New generation turbine has built in 200 plus sensors collect a large amount of data helps to operate more efficiently and manage its maintenance schedule.

 Whether it is Industry distribution network or Factor Floor there will be a lot of smart sensors gathering a large amount of information and alerts any changes significantly impact the outcome, many of these devices can be operated over the internet thru an IPAD, kiosk or Laptop. With all the real-time information and instructions efficiently processed could improve processing efficiency very significantly also reduce human interaction or manual work.

Many times in factory floor some of the machines are critical for the production process and if any breakdown will stop the production. When machines break down technicians want to fix the machine with very limited data and spend a lot of time troubleshooting to find the root cause. Due to the criticality of the machine sometimes these are replaced even they are operating in good condition because there is no way they know when it is time to replace. Machine replacement cycle is based on the usage duration not the status of the machine at the current state. But the smart machines will constantly send health diagnostics and performance data which could help to easily identify the problems with the machinery. Also with all the information from IOT machines can gather health information and could even heal very trivial problems without a technical get involved.

Smart Meters

Utility companies had the major technological revolution by introducing smart meters which can collect the usage data and send to the Local utility company in very real time.This information can help to optimize the distribution of utilities and also evolve incentives to change the usage patterns. The real-time energy usage data can help to identify fallouts, energy loss etc.. This gives the tremendous ability for the utility company to have more accurate energy forecast, efficient distribution. The next level of revolution could be capturing data of various circuit breakers to identify inefficient appliances and suggest customer on replacing them.  The smart utilities, smart sensors, smart devices could lead to smart buildings which have higher energy efficiency and comfortable environment for the customers. Also, management of facilities is a lot more efficient with smarter devices around over the Internet/Intranet.

Smart Factories

Automated process control, real-time production monitoring, real-time integrated manufacturing and environmental conditioning/monitoring are made possible because of IOT devices adoption, the platform for able to consume a large amount of data and analyze ( Big data platform).  Part of industry 3.0 moved many manual processes into computer applications but still managed by Factory operators and quality control also manual process in some cases. All of these are slowly moving to automation of business process with many health and quality information from the machines collected, analyzed and decisions are made in control centers.  Smarter logistics, smarter Machines/Robots, smart grids, smart buildings, smart sensors and smarter controls (Kiosks, control center, mobile applications) are making factories optimizing and improving productivity least amount of labor costs. In short smart factories, revolution is fueled by IOT devices and large data platforms.

Similarly, supply chains are optimizing with total connectivity between suppliers, warehouse, Factory (Production), distributors, consumers with real-time information. This helps to move forecasting model to close to real time demand model, real-time inventory management, accurate logistics and overall revolutionize the manufacturing and retail industries.

Now let’s discuss about “Data Management”

With the industrial revolutions brings the new set of software applications and a large amount of data/information.  These applications are very different than conventional enterprise applications we are building from last three decades. These applications connect with devices in real time and gather information in very real time at times many gigabytes of data for every hour. Most of these applications have high scalability, high availability, demanding and critical business needs. When hundreds of these devices communicate in very real-time required sub-second response times and at times in the range of thousands of Transactions of seconds. Also, devices could be set up in the very large geographical area so that forces to keep these IOT applications running in cloud-based infrastructure (sometimes it could be even on a private cloud).

The cloud-based IOT applications also need to still connect with existing enterprise applications (ERP, CRM, sales/marketing, customer service, etc...) and exchange data so the IOT application operates to fulfill the customer needs. In last thirty years, the enterprise application collected lots of data in application and current data management methodology segregates this data into two groups, Master data, and Transactional data.  Master data tells who is the customer, where is the location and what is the product. Transaction data will be all the operations using customer data like customer purchases, customer support requests, customer browsing history and much more.  In this environment, Transactional data is typically 100 times of master data but still, master data is critical to better understand and analyze Transactions for better business decisions.

If you look at below diagram wherein current data ecosystem without IOT devices where current enterprise applications focus on master and transactional data, but device usage data never captured by the application as all those are the manual and user-driven process.  The usage data if they collected could be the larger volume than transactional data. The usage data might have lot more critical information to help enterprise operate efficiently, improve the device to perform better. Before IOT  there is no easy way to collect the usage data and the industry 4.0 enabled to collect usage data and also automate the device usage.

                             Data organization Before IOT

                                         Data organization with IOT

As you could see in the diagram of "Data Management with IOT devices" where IOT applications are collecting the usage data will change the data management paradigm. The usage data still need to connect with Transactional data and master data to give the complete context of the information. The IOT device data still need to understand the customer, location, and device.

Even smart devices will be maturing on both hardware and software design and implementation. We all heard of multiple generations of the same device and sometimes even have subversions of the hardware of the same device in the usage. Similarly, there are multiple versions of firmware exist which is communicating to the central IOT Cloud application.  To understand the interactions between the device goods to, it is very important to know the right hardware and software versions the device running. Next Similarly we need to know where the device operating accurately, also who is the customer and users of the device. The device could change locations or change the users/customers.

All these lead to the importance of Master data and managing it with MDM methodology and then integrate the MDM data with Transactional and usage data. So it is important to understand the intersection between these different elements of data and plan enterprise echo system where all of these can co-exist.

When an enterprise already invested in Enterprise operational systems like SAP ERP, CRM and many other sales, marketing, customer service systems it is important to integrate with the cloud applications and also their corresponding analytical platforms. Every new sale, operational systems need to connect back to IOT applications. For example, when a device gets registered first times, the device customer/user get identified and connected to Enterprise customer master. Same time location information also tied with the Device and customer. Whenever customer/location changes they need to be updated in both IOT application and enterprise applications. Enterprise applications provide support and IOT Cloud applications should have consistent customer information related to the device.

 In manufacturing plants, the devices could be a machine in the product plant represented in ERP systems and the Information from the device need to reflected into ERP systems so an enterprise can take advantage of up to date latest information from the devices.

Data Management in IOT (Home Automation)

We all hear Internet of Thing and Big Data is big words we hear as new big trends which could change the future of computing and will bring next technical revolution. This Blog is focusing on discussing the IOT in home automation and its impacts. We also discuss how data management is changing with IOT.

IOT in Home Automation

IOT is maturing slowly despite the push from big fortune 500 companies. If you look at smartphone revolution of last 15 years, I would say we are in first the few years of that adaptation curve.  If you look back to years 2003 to 2005 smartphones are predominantly used to check emails or browse the internet at the slowest speed. For many people, it is the luxury item in those days where the smartphone is very necessary within 15 years’ time.  The change is so phenomenal that we could practically operate our bank accounts, purchase Tickets, interact on a social network with friends on another half of the earth or have fun by playing candy crush or watch your favorite football game live anywhere sitting in the world.

Now coming back to home automation we are in that early days of a smartphone revolution. There are home automation tools that can be operated from your smartphone. You could automate (in US market) your central Air conditioner/heat, garage doors, sprinkler system, security camera’s, security sensors, Fans/lights, front door locks and much more. Many of these have very practical uses for more comfortable life and building a smart home. But at the same time, people are cautious of the aspect of privacy (Some big guy is watching you) or these could lead to additional security threats. For example, tools like “Sesame” could be used to open your front door without using a key from your smartphone anywhere in the world. There could be practical uses at times where you are stuck in traffic and your kid (or well-known guests) is waiting at the front door.  But what happens when a thief is able to hack it? Take an example of Tile, where it helps you track things like wallets, car keys and pets. Tile could be a useful tool but could easily be used to track people and loose their privacy. All they need to do is to leave the tile in your car to track where you’re driving or leave it in your laptop bag to track where you currently are. So does privacy trump convenience?

Things like Amazon “ECHO” or Google “Home” are more like “toys” for people but could evolve central consoles in the future to manage all IOT devices. At the same time, it is very hard to replace the smartphone in people’s hand with these devices. In short, there is a lot of opportunities for a company like Apple to revolutionize IOT.

A few of the IOT devices that are commonly used are:

    1. A Smart Thermostat (NEST) can be controlled by smartphones and self-learning features that can reduce the energy cost.  The new versions of Nest can be controlled thru Amazon “Echo”

    2. Smart Sprinklers controller (Rachio) which is the replacement of a manual sprinkler controller and gives an additional functionality of managing all the functionality thru iPhone and with the additional intelligence of integrating to weather reports.They promise the intelligent management based on soil type and weather conditions. The latest versions can be controlled thru devices like Amazon”Echo” or google “Home”.

3.what If your car can tell what’s wrong with it?. If there is a smart device can communicate with the Car owner about the condition of Car?. There are many devices in the market “FIXD”  is one of those used the diagnostic tool connect to redirect the information to your smartphone. 

4.Smart Garage Door Opener (MYQ) which helps to control your garage doors thru smartphone and thru internet..

5.    Front Door Opener (SESAME)  is the smart lock is an add-on to existing locks/deadbolts of your home controlled thru smartphone and can open the door without keys and also lock and unlock the home over the internet.

6.    Remind the important things (Tile) again an application can warn you important things like Wallet and also helps to locate them where you left them lost time and also find the lost things.

7.    Home security Camera (Ring):  there are many applications where you could see if somebody knock your door before you open the door. You can see them on your smartphone and communicate with them too.

There are lot more of similar devices and they keep evolving to specific needs Probably adaptability and price is barriers as well as security and Privacy. But in our distance future our dishwasher, refrigerator can alert us on our smartphone or Alexa (Amazon Echo) when there is a problem.  Same time are we ready to pay additional $300 for an appliance for this feature.  One of the major concern for consumers will be about somebody could take advantage of the information. For example, if somebody can hack “sesame” and open the front door without a key could risk the family. Many of these hiccups will be fixed in future and confidence on these devices to grow.

This is just illustration of how Data Management happens in IOT

Now look at Data management aspects of IOT:

Most of the IOT devices interact with the central cloud application and sends a lot of data as well as user interactions thru the smartphone application. Most of this data is very helpful for the consumer to measure the usefulness of the device as well as Device manufacturer to improve the device. But the data gathered thru the process could be enormous value for communities if used for positive causes. The data could be enormous as over the course of a year a device could be sending many gigabytes and collectively could lead to trillions of records.

Is this unstructured big data?

It all depends on how you look at it. Most of the data going thru interactions are very structured with specific information and instructions like” switch on my Air conditioner”.  But same time, if you use a relational database as backend for operational management, could be possible depend on volume but could get into scalability challenges pretty quickly.    ACID (Atomicity, Consistency, Isolation and Durability) principles force you to either accept higher response time or look for alternative solutions in big data stack. But same time high availability, reliability and quick response time are the challenges need to work thru as still need to be considered. Solutions like Mongo DB or Casandra could be alternatives.

But the data collected thru these interactions and organized properly could provide a lot of insights for consumers, device manufacturer as they could provide valuable information provider as the additional source of business or even helping the community.

Let say example like smart Thermostat can provide insights into his energy usage based on the number of hours the Air-conditioner operating/Heater operating which could help to manage costs and improve efficiency. If this data collected over the years can also provide impacts on seasonal changes and alert conditions like insulation wareoff or malfunctioning of appliance etc.  Similarly, if usage records of all devices within a city can give you greater insights like weather impacts, the condition of houses, energy demands and what not. Again if privacy is not a concern there are tons of air-conditioner companies would like to access this data to sell new one if they know the customer is looking to replace soon based on the information. So the data is very useful if we put statistical and predictive analytics on top of this data to get answers to a lot of unknown with significant confidence (not necessarily 100% but mmaybe70% ). Given the high volumes of data an EDW might not a right fit and push for data lake and big data as an option but the results of insights still need to be structured to articulate in a very meaningful way, that could results still having a conventional BI solutions.

How MDM fits in?

Master data management has a very critical role in the solution despite hypothetically the user has only one system of record (that could change the scenarios like using devices like Amazon “Echo” to control).

Taking same smart Thermostat example, when user register himself and his device(s) is the first origination of master data and should be simple as there is a single source of entry.  It should be static and should be decent quality, is not it?. Not so fast. Let's look at some of the scenarios

If the facility is a rental home or vacation home. You add devices for few weeks to few months and then the devices move to a new owner, so the device movement between users could be challenging thing to manage.

What about many life changing events like marriage, divorce, move out of the home all these could have an impact on the underlying data and need to be managed properly in master data?

As well as when a device is retired and replaced with a new one the continuity of the new device statistical data connected to the old device is also critical to have the connectivity of statistical information. Most of the electronic devices are outdated in few years or becomes faulty and need replacement in few years the device mastering is very critical too.

So Both Devices and Users need to be mastered with MDM principles and best practices for improved quality of service to customers as well as to keep accuracy of analytical data.

When and where to use PLM (Product Life Cycle Management) ,PMDM (Product Master Data Management ) and PIM (Product Information Management)

In retail and manufacturing companies product data management has interesting challenges. There are many off shelf products claiming to be having PLM (Product Lifecycle Management), PIM (Product information Management) and MDM (Master Data Management) features. With this different acronyms and features it is very confusing for many IT/Business executives to make a right choice to fulfill their business needs.  Let me try to put some clarity around these, to help for better decision making process.

Product Life Cycle Management (PLM) applications are primarily focus on design, development and refinement process of the products. In manufacturing vertical the features may include product design, specifications, development cycles, packaging, customer trials, approvals process and much more. Most of the product research development business units of the organization use this toolset to manage the products.   In retail vertical the PLM means vendor selection, product approval process, product configuration changes, product /packaging redesign as per their business needs and workflows between the departments are part of it. Both Retail and Manufacturing verticals the focus is to have a well-defined process to make products designed and made available for sale.

Product information management(PIM) applications are focus on providing information about products in all channels. Current retail world is selling products in many different channels (E-commerce, Mobile, catalogs, instore and more) and each channel might have different set of needs. This can be complicated further with usage of multiple languages and demographical/localization needs.  Spanish customer want to see Spanish content of the product than English and similarly product pricing or dimensions could vary by country (product pricing in their own country, or dimensions in Feet vs meters).Need of the hour is that all channels need to have same accurate information with centralized content management. There will be needs of work flow management where different marketing /sales department are involved in generating/managing the content or doing a quality assurance of information. The product information can be received from vendors or product information syndicators like GS1. For a manufacturing vertical these application involve in sending the product information to product information consolidators or their product distributors or sellers. The  PIM systems still used by business users in marketing and sales departments with very little involvement of the IT departments.

Product MDM (Master Data Management)  

These applications focus on providing  limited set of product data used with in the internal operational and analytical systems and eliminating duplicate product ID’s/SKU’s. The focus of MDM to have golden copy of product data by consolidating product records from multiple source systems (There could be different Product ID’s (Even SKU’s) for different channels, regions or subsidiaries).  Mostly these data is actionable for the analytical systems for decision making. Eg.: a clothing retailers could see pattern of  more red shirts or sold vs blue. Electronics retailer could see there is more demand for 40 inch T.V. than 65 inch.  The Product MDM will be able to link all source identifiers to consolidate to have single version of truth in operational systems.

See Below illustration of how these three application suits fit in an enterprise (Retail or Manufacturing)

Does Enterprise Need Three different set of Tools ?

Given how these three different tool sets focus on different business needs is an enterprise need to have three product suits  to manage the product data or can they leverage one product for all the needs is a frequent question asked by business and IT executives. The answer to this question varies by organization and their data quality.  PIM and MDM have some common features, so if there is well established product ID and SKU management with very little duplication then a PIM can work for both Information and mastering. But if there is concern of data duplication with different product keys then the organization will benefit having both products. Also with enterprise moving to multi domain MDM solutions then PIM will focus on information management and product mastering will be part of multi domain MDM.

Similar if the current product lifecycle management process is well defined with some custom build application then there might not be a reason to change it unless it improves the performance of the business units.

Why The Product Migration/Upgrade is very challenging exercise for Every Organization

With Latest Mobile technology you could upgrade your IPhone Operating system or any app almost instantly, But why an enterprise   application/product migration/upgrade projects takes years  to complete and lots  those projects fail before reaching finish line. 

What’s the difference?  The most obvious reason is typically the corporate applications are lot more complicated and carry lot more risk with incorrect upgrade or failure. The risks are even more if it is involves a business operations involving customer data.    With the SAS (software as service) model does this makes it easy for organizations to manage/Upgrade much more efficiently. Before we make the decision, let’s look the reasons and complexity of Product upgrades.

Enterprises typically built their own software or purchase it from software product development companies like IBM, Microsoft, oracle and many more. Some of the Enterprises will provide Software Product/Services to their business customers  it could be likes of Sales Force, Workday ,QuickBooks online and many more. In all these scenarios typical product implementation goes thru following patterns

 Every enterprise has somewhat unique business process and business needs that application fulfills. Typical commercial application has configuration features where either Product implementation team or System Integrators will do configuration work. On top of the enterprise go ahead and do more product customization specific to their needs which could even involve tinkering the code of specific components. As you could also see that a product might become integral part of the business operation where data/information is exchanged with many other application/products within the organization. That’s bring the complexity and fear of something going wrong. Even small change in a product might impact somewhere in downstream and could hurt the business outcome. On top of it goes across multiple business units/departments  so each of them might look  product from vary different  context.

Here is the some of the Important reasons for any Product Migration will get in to challenging:

1.   Product development ,Product Sales and Product Implementation teams are not always on same page and will pull  product road map in  different directions:

 Product development team looks the industry trends, what competition doing and try to build the features  (Many times irrespective of current customer demands).

Product Sales team goal will be selling the product and would push lots of Ad-hoc requests for customization of the product for it to go thru. Depends on the size of the customer they are pursuing , the noise they can make, the influence they have on product company  the product company goes with specific customizations to impress the client.  So Product upgrade will have challenges to these customizations where they might not incorporated back in to the product.

Product Implementation team will also similarly use their creativity in the product configuration to fit customer specific business needs, even they are not direct fit for the product they are using. Data /Feature over loading to take care of the customer current problem in hand only make it bigger unmanageable problem at upgrade time. Also this is just configuration so it is seldom documented with very little trace left behind until totally different problem presented to customer at the time of product upgrade. Many of these hacks done by Implementation teams never reach product developers as needed new features. so they very rarely show up in product development roadmap.

                   So every time a Product releases a new version, these configuration(s) and customization(s) will potentially become road blocks to upgrade. Discovering the problem, finding solutions (one more round of Band-Aid fix thru configurations and customizations) will take lot of time and resources. Take a look of your last ERP/CRM upgrade, how rarely you hear it went smoothly?.

2.   Product upgrade sometimes drastically change its Architecture 

With ever changing nature of current marketplace and technological innovations, Products want to stay above the curve and provide new and fancy features to customers. While implementing these new features leads to drastic changes to existing product architect and discontinuing old features and services.  With the change of architecture all the plumbing between applications need to be changed leads to lots of rework and changes to many other applications in echo system.  Again that’s add Time cost and resources.

3.   The Fear of the change and lack of understanding the impact

With vary matrix nature of the current corporate world but where common thread of applications/data ties them together, there is always fear of change scares every organization. It is very hard to evaluate the change and what could be impact in the organization.   Even a small change  in echo system could have big impact in lot of places, so that fear force organizations to spend lot of time to evaluate and test every possible way  to reduce the risk before getting on to new product release.  If we look around we see companies still using  Internet Explorer version 8 as enterprise standard and not allowing to upgrade to latest version for the fear of some applications are not going to work and ready to hold on to a version of more than three years old. There is a telecom company still using their green screens to provision the land lines, just because they are not able to get the applications to latest technology.  Lotus notes survived as email engine for more than 20 years even it became outdated for same reason and there are companies still hold on it.

4.   The Testing cycles takes for ever

In most organizations given the intricacies about the usage of product or its produced data, the organization really want to be sure that the  upgrade is not negatively impacting any of its current business usage. For that reasons there will be enormous effort is spent on thoroughly testing the upgrade with all the scenarios. Even after that still try to keep both versions ran in parallel for months until every corner of the enterprise build confidence of the upgrade. Sometimes even a small impact could also push back the upgrade to start over again.

          Here are some of the main reasons for product migrations/upgrades takes forever to complete. Now  if we take SAS(Software as Service) model does it  makes any Better?.

          Many Product companies selling their product as SAS model by just installing the product on a Private cloud. But product is still the same and going thru same versions/releases.  I don’t think there will be much change still it depends on how strongly it is connected and integrated with in the enterprise. There is also additional challenges comes when in SAS model with multi-tenant environment (Many customers/companies using the same application) as all of them need to be ready to migrate to new version of the application or  the application should able to run multiple version(s) in same instance.

Challenges in Mastering Customers in Retail industry

Retail companies want to understand their customers and their purchasing patterns, their interests, their behavior. Retail biggest dilemma is understanding who is their important customer and how to being him back to the store or on any channel for repeated purchases. Many Retail chains went on road to implement MDM Solutions and only able to get limited success in this goal. Let’s look some of the major challenges

Inconsistent Data Capture:

Retail Stores:

A customer is willing to provide lot of information about himself when he purchase Health insurance, or open a bank account. But in retail world the customer provides very limited information while shopping in a store. In a Retail store customer will provider very little information to complete the sale and similarly Sales rep on POS also keen on finishing it faster so there won’t be long lines. If the transaction is done with credit card part of the swipe customer name is captured, but if the transaction is cash then there is no useful information is available.  The Data capture may wary based on the products he buy, when a customer buy expensive product there is higher chance of gathering reliable information.

o   Grocery Chains, Department stores might not get meaning full customer data part of the sales transaction. The Customer data might limited to Name.

o   When Purchases are High end like Home appliances, electronics the customer might provide Name, address, phone number and may be Email.

o   By storing credit card or doing reverse look up might provide more information about the customer but there is regularity restrictions on what can captured.

o   Storing Credit card with transaction is also carrying lots of risk for retail companies based on recent data breaches with some of famous retailers.

Loyalty Program/Store Cards:

Lots of retail stores moved towards having a loyalty programs or Store cards to capture customer information and track the sales of customers. While registering for the membership customer provides information as minimum as needed for the membership and depended on motivation of customer.

Some stores  (Eg: Kroger, Safeway, CVS) expect a physical card and some goes with simple searches like Phone number ( Eg. : Toysarus, Gamestop ). But many of these still able to Track Household than an individual in most cases.  As the loyalty/store card is tied to each sales transaction which helps to understand the customer more than earlier but still probably hard to contact or market to customer given the information inaccuracy.

For many Warehousing retailers like  SamsClub, Costco might have better quality of data as they have infrastructure to capture and validate accurate customer data and also with membership cost involved to the process customer has enough motivation to provide accurate data.

Online Stores

Online stores of the same retailer might capture little more information as customer provide accurate address, phone number, email etc.. as part of the registration or ordering process. But there could be inaccuracy as the shipping address might not belongs to the customer as he might be sending as a gift to  friend or family member not living with him.

Overall the Data capture from various channels is very limited and less accurate.

Credit Card Reverse Lookup

Lots of Retailer companies in the past captured credit card numbers and tried to reverse lookup and get the information about customer (Name, address, phone number etc…), But this is proved very risk with recent data breaches with major resellers.

Here are major issues with this approach

Retails are lot more hesitant to keep Credit card information for longer period than required give the liability and regulatory restrictions.

Many Credit card processing providers are restricting on the information retailers can capture part of the transaction.

Many states are regulating retailers, credit card processing companies on data captures and purpose of the use.

There all limit the Customer data capture while the sales transaction going on.  

Inadequate Data to Match:

Overall the data capture from all the channels is very limited and less accurate and problem is to get more usable from customers directly very hard. Given the data itself is very sparse, it becomes hard to match and merge them across the channels reliably.

Matching Using Customer Data Providers:

Lots of Retailers use  Third party providers who can match with the limited information they have about the customer to get more details about the customer .The Third party providers are able to match  using limited information like name, email address, phone number  and provide  accurate details of customer and including customer profile information.

This seems to be a way to get better contactable information about the customer and able to understand customer well, this is also very expensive option too.

Constraints with third party providers are:

1.       Most of the Providers only rent the record for limited use. After the expiration of the rental period retailer is obligated to remove from their system

2.       Providers only provide information for specific use case , which limits usage for enterprise customer master data  management

3.       Also whenever information is received from provider is the movement of truth of that time. With time the data becomes stale, which force to validate periodically which also incur cost.

4.        As most providers get their data refreshed  only periodically (may be once a month), at times the information captured with client might be more accurate.

Given all the constraints around getting the data from Third Party providers, the cost and restrictions are enormous and need to be managed carefully.

Best Practices:

Given all the challenges here are few best practices for customer data management

1.       Build data capture business process to get more accurate information about the customer.

2.       Put together Loyalty programs and encourage customer to provide accurate information providing incentives

3.       Have same data points capture in all the channels

4.       If third party data provider is used, understand the cost and restrictions around the data usage.

5.      Try to  Identity the customer/household  with in your data accurately so  you will be able to understand customer better.

6.       Honor customer preferences likes Do not call, Do not mail  thru customer mastering to have better customer experience.

Why Data Management Matters

Flowers are static, we don’t consume them except with our eyes and perhaps our sense of smell (If they are decorative plastic flowers then there won’t even smell). In that sense they are unlike food, which we do consume and from which we derive sustenance, nutrition, and life itself. A life without flowers might be unfulfilling and perhaps even miserable, but  life without food would be certain death. Same goes about Data for  any Organization

And so it goes with data. Data is not static, sitting on a shelf. We keep receiving it and consume it. Data meant to be consumed by our business applications for everyday use. Provide better experience to its customers, understand the customer behavior, Identify fraud/risks, Provider deeper understanding for better business decision making.

Just as we need food from basic four food groups for sustainable health, data needs to Managed and Governed with following norms to make it fit for consumption

1.      Data need to be Complete: Many times incomplete data leads to wrong conclusions and bad decisions.
2.      Data need to be Consistent: Data does not have consistency many times it is impossible to analyze and make sense of it.
3.      Data need to be Conform: Data need to conform to standards
4.      Data need to be consolidated: If data is not consolidated to usable levels could lead wrong aggregations. Eliminating data duplication is very important for data management to have accurate reporting in analytic's   and right information in operational side.

When data follow all these norms is healthy and delicious for consumption. It builds trust of business to use the data which leads to healthy decisions for the organization. When data managed consistently will keep getting better, healthier and more dependable.

The way we take care our food for cleanliness, nutritious values, calories, taste the same constant focus need to be given to data management to keep following the norms. As we getting new data sources or building new data centric applications making sure they follow same norms is equally important.