The Chief Data Officer

Many organisations are experimenting with the concept of a Chief Data Officer. Although according to Raconteur most are struggling with the concept as well as making headway with the concept of managing data for value.

It seems that there are a number of problems:

• Business Understanding of the Role and Data Issues;
• Resourcing, Funding and Provisioning Data Initiatives;
• Current Starting Points for many Businesses with Poor Data Quality;
• Focusing too much on compliance and not enough on Value.

Personally, I would suggest that there is a value staircase for Data Management starting with Compliance and Protection on the Bottom Rung and then progressing through Quality, Integration, Sourcing (external data feeds), Insight & Intelligence to Adaptive Control (involving use of machine learning, experimentation and up to the minute data feeds to optimise control).

It is difficult howver to obtain value if a business tries to progressively work its way upwards through this staircase if it attempts to do everything in the first one before attempting the next step. The scope is too wide. Instead busnesses need to prioritise a business problem or opportunity and address a vertical slice down this value staircase to deliver value quickly. Progression can be achieved by addressing the next opportunities and reviewing what is needed in each step to improve overall capability as each increment is implemented.

However, probably the biggest issue is the role. What is the difference between a CIO, a Chief Digital Officer, a CTO, a CSO, a Principal Data Archiect and a Chief Data Officer? These all overlap and are often confused in scope. Many businesses are handing the Chief Digital Officer role back to the CIO, so it may be time to do the same with the Chief Digital Officer, because if they are separated the CIO's role is hollowed out to just look at the plumbing and not the value to the business.

It is also important to delegate data quality and integrity to business managers responsible for exploiting data in their day-to-day operations. IT may provide the policy framework, tools and measurement framework for managing data quality, but line management needs to take responsibility and include data quality in their personal objectives, just like they do with budgets. Otherwise value will never be achieved.

Can Machine Learning Save Big Data?

Business Intelligence (BI) and Big Data (BD) are supposed to help enterprises improve their overall business control by identifying trends, anomalies and insight into what is really happening, so that they can take appropriate action to correct problems and address opportunities, as well as predict what might happen with a degree of informed confidence. Yet most businesses fail to either exploit BI and BD solutions or even if the do just don't get the expected returns. 

One of the biggest problems facing anyone trying to do Business Intelligence or Big Data is quality of data. In some cases, someone has just chosen the wrong source of data (a Master Data Management issue). An example of this that I fell over many years ago was that an organisation tried to build a customer database for Sales and Marketing, using financial sales ledger data. This caused immense problems because Finance is not interested in the same sort of data as Sales and Marketing and don't care who the key contacts are or what the organisdation structure is, as long as someone pays the invoices.

However, an overwelming problem is that the data is dirty. Legacy systems usually were not designed with BI data in mind and external sources used for BD usually involve different terminology and definitions when discussing or dealing with the same thing. Alternatively, the same terminology may be used to use completely different things by different people or organisations. Furthr issues arise when users are asked to input data which is of no utility to them into a system. The act of inputting the data just complicates their day job without helping them do it. So they often provide default information. Additionaly, other forms of context may inform what was meant by an iitem of data or a vague description.

So this often reduces organisations to using a Subject Matter Expert (SME) to manually interpret data and stick everything onto a spreadsheet to conduct a particular analysis. This is costly, labourious and slow. It may even be innacuarate as SME fatigue (or conflicts in opinion between different SMEs) leads to inconsistency and errors in interpretting data. So many analyses just don't get carried out or produce information too late,  lack much detail beyond generalistic interpretations or may even be suspect in accuracy.

In some cases, if the analysis is likely to be needed frequently with new or updated data sets, it may be possible to write programs with interpretation rules to deal with many of the problems mentioned above. However this is painstaking and laborious and often requires continuous maintenance, constricting its viability for many analyses as new costs and delays are built in.

Machine Learning however may offer an answer. If you can get an SME to provide some examples and use them to train a neural network, you can quickly develop the facility to improve data quality. The machine learning tool should clean up data where there are clear rules and identify areas where there is uncerainty or conflict in interpretation. This requires a relatively small amount of effort to put right and then improves consistency. The more data and examples you feed at a machine learning system, the better it becomes, so you get ever improving interpretation of dirty data. This is quick too. 

I found this out from experience, when in a previous company I worked in, we used machine learning to interpret data readings and categorisation of test results for thousands of readings taken of ship hull  plate thickness around different parts of the vessels being inspected, so that they could be analysed by structural analysis experts. A manual process which used to take a highly skilled engineer several days to complete (without creating any value) was reduced to less than 20 minutes and improved accuracy. This meant improved job satisfaction, higher productivity, better quality of output and a more responsive (i.e. quicker) service for the client.

So there you have it. Machine learning can bring some very quick gains with quite simple applications. You don't need complicated Deep Learning solutions to deliver value and it can vastly improve your BI and BD efforts where you have to live with dirty data.