DevOps and AI means that Lean Data has another Principle

In an earlier post I expounded on the principles of Lean Data:

• Organisations know what data they hold and manage;
• Data is classified according to subject area and criticality;
• Only the minimum data necessary to Add Value to the business is held;
• Data replication is kept to the minimum level necessary to optimise business performance;
• Data Value is determined by its utility in Serving the Customer, Supporting Essential Capability, Protecting the Organisation, Providing Insight for Business Decision Making.

I also covered Data Portfolio Management. However, what I missed was the emerging issues around database management systems and Machine Learning. It appears that as people start to scale their Lean efforts (Strategic Alignment, Design Thinking, Agile & DevOps) they are beginning to realise that this includes data too.

What becomes increasingly important is having a handle on configuration management over data base design (i.e. schemas) and data sets themselves, ensuring that it is co-ordinated with code configuration management and integration across multiple deployments. This should have been implicit, but apparently it was not.

Therefore I suggest adding the principle that:

• Data configuration management of schemas, test data conditions and machine learning training data is co-ordinated with applications and code configuration management.

The Way of DAU - The best a business can be

Just to let you know, The Way of DAU is now available from Amazon as a paperback. (click on Amazon to see) and the e-version should be available soon. 

This is a deliberately simple book on the quite complex subject of how to adopt a sustainable digital business model. It was inspired by my personal frustration with incomplete models and advice available for Digital Business.

Digital operations have now replaced traditional Business As Usual business models. The Way of DAU promotes 10 basic principles, an iterative Framework (the DAF or Digital Adoption Framework), and positive cultural values to achieve the behaviours needed in successful digital businesses.

The book is based on a mixture of personal experience (in an organisation struggling to reinvent itself) as well as collected best practices. The current edition represents an MVP version. I hope to collect constructive feedback via a LinkedIn Group to drive future releases of the book. (see: The Way of DAU Group ).

GDPR, CIO issues, Lean Data & Data Portfolio Management

Last night's CIO event hosted by Harvey Nash and KPMG was held to launch their 2017 CIO Survey "Navigating Uncertainty".

In the Panel discussion afterwards, one of the key issues raised was about "knowing where your data is". GDPR is certainly driving this, for personal data in Europe and anyone who trades with organisations or consumers based there. As its difficult to implement the "right to be forgotten" if you don't know what data you hold and where it is. Similarly, SoX in the US has driven similar concerns about Financial data. The move to "Cloud First" also compounds this need, as it is core to successful integration.

So why is this such a big deal as much of GDPR is about doing things which a business really ought to be doing anyway? basically its ancient history. Most large organisations have grown partially by merging with and acquiring other organisations. Their management teams often have the tendency to declare victory before full integration occurs.

Then there are cost cutting issues. Most businesses have been through boom and bust cycles of  large investment followed by cost cutting and asset squeezing. Often this has included head count reductions or outsourcing. Each of which ensures that knowledge about where things are leaves the organisation. Many service providers tend not to document things well, if they are allowed to get away with it, as this helps keep effort and FTE (therefore costs) down. There is natural staff churn of anywhere between 5% and 20% per year in typical companies, depending upon culture, rates of pay and opportunities. Documentation does not keep pace with lost knowledge as exit processes are usually poor in knowledge transfer.

Finally, DIY activities in the business often results in unofficial applications being adopted, especially as XaaS makes this easy to do. So put this all together and it is little wonder that organisations often do not know where their data is or even what data they have. This is a situation which brings inherent risk. If an organisation does not know where its data is, how does it protect it. If no one knows what data is help and "managed", then how is it integrated, kept coherent, kept clean and timely? how does the organisation know what it is actually spending on data or even what the value of its data is. Then there is the small matter of compliance. How does the organisation know whether it is complying. These are all data hygiene issues which need to be addressed if digitisation is going to support a Digital Business Model.

So now is the time to introduce Lean Data and make sure that Data Portfolio Management (DPM) is practiced as part of any approach to Asset Portfolio Management. (Asset Portfolio Management = Application Portfolio Management + Infrastructure Portfolio Management + Data Portfolio Management).

Lean Data principles mean that:

• Organisations know what data they hold and manage;
• Data is classified according to subject area and criticality;
• Only the minimum data necessary to Add Value to the business is held;
• Data replication is kept to the minimum level necessary to optimise business performance;
• Data Value is determined by its utility in Serving the Customer, Supporting Essential Capability, Protecting the Organisation, Providing Insight for Business Decision Making.

Data Portfolio Management is concerned with:

• Knowing what data is held and where it is;
• Understanding the quality of the data;
• Knowing what technology is used to manage the data and its overall condition;
• Being able to address questions concerning issues such as criticality, protection, archiving, cost of management;
• Understanding how Master Data Management (MDM) and integration occurs;
• Knowing who has Stewardship responsibility and consumer rights for the data;
• Regularly reviewing management actions to improve Data Value and address Lean Data principles.