DevOps and AI means that Lean Data has another Principle

In an earlier post I expounded on the principles of Lean Data:

• Organisations know what data they hold and manage;
• Data is classified according to subject area and criticality;
• Only the minimum data necessary to Add Value to the business is held;
• Data replication is kept to the minimum level necessary to optimise business performance;
• Data Value is determined by its utility in Serving the Customer, Supporting Essential Capability, Protecting the Organisation, Providing Insight for Business Decision Making.

I also covered Data Portfolio Management. However, what I missed was the emerging issues around database management systems and Machine Learning. It appears that as people start to scale their Lean efforts (Strategic Alignment, Design Thinking, Agile & DevOps) they are beginning to realise that this includes data too.

What becomes increasingly important is having a handle on configuration management over data base design (i.e. schemas) and data sets themselves, ensuring that it is co-ordinated with code configuration management and integration across multiple deployments. This should have been implicit, but apparently it was not.

Therefore I suggest adding the principle that:

• Data configuration management of schemas, test data conditions and machine learning training data is co-ordinated with applications and code configuration management.

DevOps 2018 Report

Puppet has published its 2018 State of DevOps Report. This year they have examined the premises on adoption and confirmed that most successful adoptions follow a project by project approach, with a single project pioneering the way and gradual cross poliantion of practice whilst it is matured.

The project focuses on 2 aspects of adoption:

• CAMS - Culture - Automation - Mearsurement & Sharing

• a 6 layer maturity model from 0 to 5, which broadly follows a simplify and Standardise Approach, Followed by Process Practices, then Automation and finally Self Service capabilities.

One of the themes of the report is the gap between teams practicing DevOps and Senior Management. It still appears that DevOps adopters continue to struggle with cultural aspects associated with empowerment and reporting.

Anyway, for anyone interested in the Lean Aspects of Digital Business Models, this is essential reading.

Group Technology For DevOps

One of the often overlooked concepts absorbed by the umbrella term of Lean is Group Technology. The concept originated in Russia, was refined in the UK and then seized upon by North America in the 1960-1980s period, especially when robot cells and flexible manufacturing were the fashion in Manufacturing Systems.

Basically the concep was simple. Old fashioned manufacturing factories used to be laid out in "functional" organisations. So, say, all the Lathes were in one corner, the drilling machines would be in another part of the factory and the milling machines in a different part or even a different building.

A component might require that: 

1. bar metal was cut to size - done by the sawing section;
2. it was turned to shape - done by the lathes section;
3. some holes were drilled - done by the drilling section;
4. a flat surface was milled on one side - done by the milling section;
5. one end was polished extra smoothly - done by the grinding section;
6. before final inspection, packing and shipping.

Each step would be conducted under the aegis of a different foreman. Each foreman had his own priorities and was tasked with running his section efficiently. At each step, the appropriate machine would have to be set up, perhaps with a special jig as well as, with its specific set of cutting tools.

So the process was a long series of queues and delays for set up, with only a small portion of time spent actually cutting metal. Efficiency was assure by having as much work as possible queued at each section, so the machinists never ran out of work.

Process times were long, delivery dates were never guaranteed to meet promised delivery dates, but it appeared efficient, even if  lots of money (working capital) was tied up in work-in-progress inventory. It was great for the foremen, because they were never individually to blame if something was late.

Group technology, was originally focused on reducing setup times. It involved using a classification system to identify parts which were similar to manufacture, e.g. small long thin round round parts or large short squarish parts, and batching them together for manufacture at the same time.

Then the idea of the group technology cell was introduced. All the machine required to make small round parts were organised into one part of the factory, in the normal sequence in which they might be used, creating a flow line. So similar parts were all sent to the same cell and put under the control of a single foreman who was also responsible for delivery performance. This allowed parts to be batched and pipelined along the natural flow line to meet both effectiveness and efficiency needs. It also meant that there was only one queue, the one going into the cell and work in progress could be reduced. Adding TQM techniques into the mix, so that the foreman was responsible for quality as well, turned him into a mini production manager and enabled quality to be improved. Then, where practicable, automation was introduced to raise overall performance further.

So what does this mean to IT operations. Well most organisations have normally organised so that the unix/linux administrators sit in one corner reporting to a senior administrator. The NT administrators sit in another corner with their senior administrator. PC administration sits under desktop support. DBAs sit in their own little getto, with a senior DBA. The network administration people might be in another building. If global outsourcers are involved, then corners become "Competency Centres" often geographically dispersed.

In traditional "on premise" or "in data centre" operations, a large part of the time of administrators is focused on the box shifting aspects of the role with manual administration of builds and maintenance tasks. No one team is responsible for the support of an application. So when things go wrong, different disciplines point the finger at each other, saying "our bit works it must be theirs which has gone wrong". This makes reacting to incidents slow and innefficient, and slows down releases of new features. So what appears efficient is an inhibitor on value by slowing change and reducing uptime..

Modern "Everything as a Service" (XaaS) environments offer new opportunities. The job can be refocussed from box shifting and manual "oiling and adjusting" type tasks to a higher level of "End-to-End" application administration. This requires administrators to be re-organised into either application facing (if it's large like an ERP or core banking system) or business facing (if there are lots of linked smaller applications used by a business area) teams. They can then work with application and business facing development and test teams to meet the continuous delivery needs of modern digital products and applications. Tooling such as Splunk or Oracle's Management as a Service are good at pinpointing root causes of problems, ensuring that the team tracks down causes quickly and focuses on jointly fixing issues. 

That way, accountability lies with the team and incidents can be resolved quickly, whilst new releases are jointly planned to optimise value to the business. This is Group Technology for DevOps.

Update on the World of Agile

CollabNet has just published its 12 Annual Report on "The State of Agile". This makes interesting reading, although some filtering may be necessary as responses are dominated by North American participants in its survey. So for example SAFe tends to the prevalent approach taken to scaling Agile, which is more popular in the US than Europe.

To me, the most interesting bits are the high percentages of survey respondents reporting significant Business Value Oriented Benefits. Over 60% of respondents claimed:

• Better responsiveness to Changing Priorities;
• Improved Project Visibility;
• Enhanced Business/IT Alignment;
• Increased Speed to Delivery;
• Higher Productivity;
• Raised Team Morale.

The percentages far exceed the reasons for pursuing Agile. The other interesting metric is that only a fifth of respondents reported lower costs, which is in line with Agile philosophy, but still good to see confirmed.

The other key takeaway is that an increasingly high number of businesses have adopted both agile and DevOps as their mainstream approach.

Has Big Data Lost its Mojo?

A number of surveys were published during 2017 which suggested that Big Data has lost the CxO mind share that it had.

The overall impression gained was that business leaders are putting their emphasis on Artificial Intelligence (in particular Machine Learning) and IoT, whilst CIOs and Technical Leaders are worrying about Security and Lean (Agile and DevOps). 

At the same time the consumer and gadget end of things are focusing on wearables, voice and VR/AR based devices (implying other types of AI are getting important). 

Somewhere in the mix, people are worrying about culture, product management and marketing, and organisations designed around empowered product teams.

There almost appears to be an assumption that big data has been cracked and apps are easy. Also, data governance is not really getting the attention that it should do and vendors are pushing APIs for integration.

This suggests not just a gap between business vision and technical capability to deliver, but also that the consumer led boom in exploitation is going places which don't necessarily fit well with classical business environments. Open plan offices are not the place where people should all be talking to their digital assistants.

The lessons I draw from this are than business management teams need to start thinking holistically about what is needed to deliver their new business (i.e. digitally augmented) strategies and models, but also about what the future workplace looks like. Is the office finally dying?

They also need to get real about data. It needs to be managed using lean data principles. Integration and security are vital to get right. IoT and other exploitation approaches are going to emphasise Big Data's importance further. So it's vitally important to develop data specialists who understand your business and are networked with the right people. A recent article in the Harvard Business Review suggests that most organisations have focused on technicalities and need to look at a more immersive approach and organisational issues too.

The Value of Data

Today I realised that the evidence for Digital being the New Normal was compelling when I read about the way that the agricultural industry is embracing IoT, Big Data and Machine Learning (see: FoodIT: Fork to Farm ) which provides comprehensive overview of the 4th annual conference examining how to join the ecosystem from consumer to farmer up, so that farms respond to consumer tastes, improve yields, avoid waste and provide the provenance etc. that consumers now want. There are great opportunities to provide the next Deliveroo for farm products and by-pass the super markets entirely. This would greatly improve profitability for farmers and the experience for customers as they get fresh, seasonal, sustainable, quality products directly. It could also bypass the scandal of the modern monopoly of slaughter houses, if the right investments were made.

I especially enjoyed the redefining IoT by the conference as the Internet of Tomatoes, spelling out how much the industry has got into it.

There were also some interesting predictions for Data in another couple of articles on the data makes possible  site by Kirk Borne of Booz Allen Hamilton about trends for this year. Personally I found his comments on graph analytics the most compelling as this is an area of growing maturity in which it is now possible to easily uncover insights into linkages of cause and affect and networks of people and activity. He also had interesting things to say about hyper personalisation and realigning AI with a people centric model of mutual assistance to do more (similar to some of my previous observations). Finally he coined a new phrase DataOps to describe lean development approaches as applied to big data.

Does Your SOC Need Darning?

Microfocus (the new owner of much of HPE's former software division) has released the 2017 State of Security Operations Report.

This analyses the findings of analysis of the Maturity Model levels of practice in enterprise Security Operating Centres or SOCs. For those who are uninitiated, SOCs are a relatively new organisational construct within IT and are responsible for assuring that there is ongoing monitoring and analysis of an organisation's IT operations to ensure that vulnerabilities are detected, intrusions are caught and problems are rectified. Although there does seem to be a great deal of diversity in people's interpretation of the exact scope of this remit.

Maturity Models (see CMMI) typically categorises maturity in 5 levels which address process and practice standardisation as well as feedback loop control via metrics and optimisation. 1 is ad hoc, 2 is repeatable, 3 is uniformly standardised and so on. So most organisations will aspire to level 5 as an acceptable level of conformity. Though the actual scope of coverage is important too.

Many enterprises have adopted SOCs to help deal with the ongoing climate of cyber threats arising from things such as simple viruses,  spear pfishing, ransomware and Denial of Service attacks.

The report is quite sobering. Close to a quarter of the assessed organisations failed to achieve a score of even 1. only a fith appear to be making headway and the overall average score is less than 2.

The report finds that much SOC effort is wasted dealing with false positives arising from little standardisation and poor configurations of equipment. This underlines the operational hygiene issues of having accurate CMS data and consistency in build and installation. Knowing what you have and standardising as much as is practicable, does not just make it easier to operate an IT estate, but also to protect it by detecting anomalies and other problems. These are practices which not just ITIL but DevOps considers essential to robust operations and change management.

The report also shows problems with working out the right blend of insourcing and outsourcing as well as skills retention.

Overall there are signs within the report of slow but gradual maturing of approaches as well as better pooling of knowledge within organisations. But t is understandable, given the scale of issues that people face, that Splunk for instance promotes the adoption of a Lean SOC approach and gradual incremental implementation of SOC capabilities to address business priorities, 1 at a time.

Agile Is Not New, But It is Useful

Agile advocates are often quite quite siloed in their thinking, many having had little other experience of delivering projects than bespoke software development projects. Many of them are not even aware of the historical trends before the publication of the Agile Manifesto or that they are in fact just participating in an evolutionary trend, rather than a revolutionary approach. Agile owes much to Rapid Application Development or RAD (its immediate predecessor). Often, I have heard them musing about applying Agile to other technologies and industries and wandering whether it is possible. At the same time they miss the fact that DevOps, which extends Agile, is based on manufacturing theory (Total Quality Management, Just-In-Time and Flexible Automation).

Actually, project managers who subscribe to the AgilePM approach, will often use the organisational aspects of agile to run their projects and the approaches used in RAD were not new either. Prototyping was a feature enabled in manufacturing by the use of CADCAM to rapidly design and produce prototypes and Engineers have always been taught that Design is an iterative process. The concepts of Design Thinking originate from academic papers produced in the 1960s, and address many of the technique aspects of Agile including multi functional product teams and evolutionary prototyping and development of innovative products.

But if you want to see Agile organisational constructs such as sprints in action, go to a shipyard. almost all modern shipyards practice modular shipbuilding approaches. They split the overall design into modeles which are constructed and outfitted. These are combined to form blocks, which are then further outfitted. The blocks are then moved to the construction berth, combined and integrated. Work is usually organised around 1 to 2 week production units (although sometimes 4 week ones are used). Testing is continuous and progressive as systems and compartments are completed. Test conditions are part of design. (Does this sound Familiar?). 

Although this practice evolved in the late 60s in Japan and gradually spread, it was in fact learnt from high rise building construction, with each floor of office or hotel blocks treated as a sprint.

Likewise, filming has a similar approach. Up front sketches are used to design each scene. Some work is done with cameras, lighting and film to workout the best combinations to achieve the cinematic affects required (a little similar to preliminary UI design). Scenes become the sprints and are filmed iterativelly to produce what can be afforded and so on. UAT is done with test audiences.

So Agile is not new, it's just slightly tailored to software development, and just like some of the other approaches used elsewhere, it's based on producing a right quality product with the minimum of fuss.

2018 in Digital

Recently, the UK budget was set with a few old fashioned measures for increasing productivity, Investment in Infrastructure and More Money for Apprenticeships.

I then read someone's blog which suggested that this had missed the mark. His premise was that investment in technology usually fails and there is a need to continue to emphasise other traditional business change and simplicity measures such as clear strategic leadership, process design and email de-cluttering. He produced statistics to show that e-mail costs more than the UK's contributions to the EU's budget. Although I thought that this particular commentary missed the disasters caused by poor collaboration by business leaders and the failure to work together as a unified team.

At the same time, a commentary in Forbes suggested that 2018 will be the year of software automation with a sudden increase in the trading of enterprise data sets and a huge increase in the number of data scientists. Although the breaks to this are whether data sets are considered valuable IPR and the need to train up a lot more people in aspects of data science.

Contino and Gartner have identified 2018 as the year of DevOps with DevOps driving Agile (Gartner) and increasing competition between Platform Vendor services between AWS, GCS and Azure as well as the new wild card entry AliBaba. Interestingly, growth rates for all of these services are in the 50-90% p.a. area. Contino also mentioned that there are many new developments not just in the area of containerisation and serverless computing, but also in improved cloud security via projects such as Calico.

Surprisingly, given the recent frenzies, most commentators were muted on the impact of AI and Machine learning. Likewise, no one bothered to mention the rapidly maturing area of wearable technology or the new promises that quantum computing may at last start to deliver.

Overall, the impression is of Lean Digital becoming mainstream whilst other post digital technologies gradually insinuate themselves into leading edge enterprises.

Digital Patterns

Most software developers are familiar with the concept of a pattern. They use them to describe types of programming problems which can be solved in the same generic manner.

This was originally developed to support the indexing of re-usable code and trading in 3rd party off the shelf code components, with the idea of improving overall development productivity and quality. The quality aspect arising from the assumption that such code would be designed to be flexible and robust in many scenarios.

The idea was not actually that novel as it replicated the concept of group technology in manufacturing, where a group technology code defines the shape, size and manufacturing characteristics of 3D parts or components. The idea being that groups of machines could be set up to manufacture families of parts more quickly than traditional methods where machines were grouped functionally and work meandered all over a factory between the functional groups of machines. 

Anyway, patterns have become such a useful metaphor that developers will often describe what they are doing in terms of what the pattern is and therefore what tools and approach they are using. Consequently, the concept of patterns has also been adopted to describe aspects of solution architecture and encourage design re-use within enterprises.

So I have been struck recently by the fact that going digital or adopting a Digital As Usual model might be facilitated by certain organisational patterns.

For example, Lean and Agile solution development is usually product and product team based. The product team integrates the viewpoints of several functions involved in developing and exploiting the product typically including users, product owners, analysts, developers, testers and DevOps engineers. This might be called the Product Team Pattern.

Some approaches to delivering digital products also look at mass customisation via standard features. The principle being that a product is constructed from a number of features for which there may be a number of standard options. The product is then tailored to a customer's needs by selecting which features and which options the customer wants. This then gives a tailored product experience with most of the benefits of a bespoke product, but better economics and more consistent quality because each feature and each option is standardised. This might be called the Adaptive Product Pattern.

A recent article in the SLOAN MIT Management Review, Is Your Company Ready for a Digital Future? described a "Future Ready" pattern, in which it identifies a combination of Customer Experience and Process Efficiency capabilities as delivering this type of pattern resulting in low cost innovation, superlative customer experience, modular agility and the ability to exploit data as a future asset. Whilst this complements the thinking in "The Way of DAU", it struck me that perhaps there are other patterns out there which need to be identified. 

Any thoughts anyone?   

The Way of DAU - The best a business can be

Just to let you know, The Way of DAU is now available from Amazon as a paperback. (click on Amazon to see) and the e-version should be available soon. 

This is a deliberately simple book on the quite complex subject of how to adopt a sustainable digital business model. It was inspired by my personal frustration with incomplete models and advice available for Digital Business.

Digital operations have now replaced traditional Business As Usual business models. The Way of DAU promotes 10 basic principles, an iterative Framework (the DAF or Digital Adoption Framework), and positive cultural values to achieve the behaviours needed in successful digital businesses.

The book is based on a mixture of personal experience (in an organisation struggling to reinvent itself) as well as collected best practices. The current edition represents an MVP version. I hope to collect constructive feedback via a LinkedIn Group to drive future releases of the book. (see: The Way of DAU Group ).

Digital As Usual - Is Progress Stalling?

Attending a number of exhibitions this year, I have been struck by how little things have changed since last year. Indeed many of the suppliers at their stands have been quite muted in their messages and there has been little indication of much innovation. Overall the impression has been one of "more of the same" and a little "me too" in terms of what they have been promoting.

This year's State of DevOps Report showed increasing adoption of overall Lean (Agile and DevOps) practices and moved the emphasis to leadership as Digital becomes more embedded as a Digital as Usual practice.

The World Quality Report also showed higher demand on QA and testing, but a shortfall on budget and resources. Although it did express the anticipation that budgets would increase soon.

Both reports indicated maturing of accepted practice rather than new or radical changes in approach, or further innovations.

So, although technical progress may not be advancing much, adoption is definitely becomming mainstream.

This I take as confirmation that Digital is firmly mebedded as the new normal.

The Way of DAU - The New Digital Philosophy

For those of you who have come to accept that Digital has become the new normal, it it is not surprising to know that there is a new acronym DAU or Digital as Usual. This replaces the old one BAU or Business as Usual.

There is even a philosophy known as The Way of DAU (pronounced Dow). This is built around 10 guiding principles which encapsulate current best practice in the Digital world.

For those getting started, the principles are useful for driving adoption and practice of Digital. They are:

1.       Understand the Market;

2.       Identify what Changes Rewrite the Rules;

3.    Select High Priority Opportunities;

4.       Build Product Focused Culture and Teams ;

5.       Walk in your Customers’ Shoes;

6.       Embrace Opex;

7.       Go Lean;

8.       Cherish Information;

9.       Nurture Partnerships;

                    10.     Harness Fear of Obsolescence.

Although there is an assumption of a continuous iterative loop to be followed when applying them.

That's all for the weekend.

Is Your Enterprise Digital Ready? - Summer Time Reading Recommendations

Sometimes it is better to make sure that other people think that they had the idea first. As Information Professionals are not always believed when they try to persuade their business colleagues that they need to change their ways, if they are going to get the best out of their Digital Strategies and Investments.

The truth is that there are some fundamental issues to address which separate Good or Average performing companies from Excellent ones when it comes to exploiting IT or Digital investments and transformations. Proponents of Digital Strategy, Agile and DevOps often raise them, but due to the fact that these issues are being raised in a technological context, non IT colleagues tend to either listen and not hear or just dismiss them as the mad ravings of techno boffins.

Key among these are:

1. Having a "Real Business Strategy" based on deep market insight and how to disrupt or exploit it in your favour;
2. Working as a Team within a healthy Business Culture;
3. Adopting Design Thinking to help empathise with customer needs, really understand what you are trying to address and then to creatively address options and iterate design to create elegant and well targetted solutions rapidly, whilst embracing leraning from failure as a critical part of the approach;
4. Systems Thinking to understand the end-to-end process, identify and manage critical business bottlenecks and organise around product or work delivery, instead of hierarchical functional silos.

A large part of this is really concerned with taking an ego-less multi-functional team approach to addressing what is really needed and then pursuing continuous delivery, automation and improvement in small steps. 

Fortunatley there are some great business books on some of these subjects which evryone should be encouraged to read as they address the issues from a more general business perspective and introduce the key concepts that all the Digital Geeks are so keen on.

My recommendations are:

Good Strategy Bad Strategy is a great expose on how to do Strategy properly. Most readers will recognise many of the bad examples which are lacerated by Richard Rumelt (one of the leading fathers of Business Strategy) in what is a fairly easy and entertaining read.

Winning Teams Winning Cultures  - addresses a lot of the key issues in building a positive enterprise culture. This comes from Larry Senn (of Senn Delaney a culutural change consultancy) and Jim Hart who are long time practitioners in the field of cultural change. It's an interesting book, because just like democracy it is difficult to bomb culutural change from 40,000 feet into an organisation. It requires authenticity, long term commitment and sytematic sweating by the management team to achieve.

The Human Constraint - the author (Angella Montgomerry) takes the principles advocated by Demming and Goldratt and updates them to apply to all businesses (not just manufacturing) in the practical adoption of system thinking and the Theory of Constraints to business in general.

Finally there are plenty of sources on Design Thinking available on the internet. The UK Design Council publishes the Double Diamon model which provides a simple model for explaining the process. I recently saw a great webcast by Ileana Stigliani, fom Imperial College Business School, on the subject Unleash Innovation Through Design Thinking

(see: https://www.ivyexec.com/professionals/classes/details/unleash-innovation-through-design-thinking ).

Digital Adoption Framework

A lot get's talked about Digital, but there are few comprehensive approaches to adoption available for reference. 

This is why I was interested when I came across Vadrim's DAF diagram, reproduced below. Enjoy!