A Story of DevOps >> Episode 7. Jedi (Practices and Mastery)

"Everyday we push performance to its limit. Our performance. Our car's performance. We innovate, we engineer, we design. We master rules and then break them. Only to push further. Past the limits of convention. This is where performance becomes art." Jaguar. The Art of Performance. 

I bet you were wondering when I was going to bring Star Wars to this series – well, I hate to disappoint so here goes!

We have done a lot of groundwork but we haven’t ‘done’ anything yet. Many of you may ask why I have barely talked about DevOps and Agile. What’s all this got to do with delivering software?

Good question – so far it’s been all about the groundwork – creating the foundations to build a high performance teams. If the foundations are shaky, you cannot build skyscrapers. Foundations take time – you are not going to undergo a DevOps transformation in a couple of months – more like a couple of years. Check out Amazon’s letter to their Shareholders – greatness takes time.

I would recommend that you read A Story of DevOps, Origins, One Ring, Freedom, Assemble, Shield and Kryptonian before you read this.

Try not. Do. Or do not. There is no try

Yoda is training Skywalker to become a Jedi (I am assuming you have all watched the Star Wars series). Yoda has started to explain what the Force is and how to use it. Skywalker starts to get to grips with using the Force and then Yoda asks him to raise the crashed aircraft from the swamp.

Enjoy… https://youtu.be/E3-CpzZJl8w

“Try not. Do. Or do not. There is no try” – What a legendary line. Luke struggles to apply the same Force to a larger object – his answer is to ‘try’ and then he fails. Skywalker thinks that the ship was too big to move and Yoda replies with “Judge me by my size do you?”. When Yoda raises the ship from the swamp Skywalker says “I can’t believe it” to which Yoda responds “That is why you fail”.

A little while earlier, Skywalker had to face his fear and I would recommend that you read this article by Brene Brown.

Like Skywalker, we must understand the Force, overcome our fears and insecurities and practice. Once we have mastered the Force, we become a Jedi.

Are you ready Padwan?

First Principles

In previous articles, we have talked about the Force which consists of great leaders, vision, product teams, tools and environment, let’s now discuss the principles behind the practices.

Principle Suite 1 >> Agile

Core to the principles is the Agile Manifesto:

“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.”

This is supported by the following Agile principles:

“We follow these 12 principles:

  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  4. Business people and developers must work together daily throughout the project.
  5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  7. Working software is the primary measure of progress.
  8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  9. Continuous attention to technical excellence and good design enhances agility.
  10. Simplicity – the art of maximising the amount of work not done – is essential.
  11. The best architectures, requirements, and designs emerge from self-organising teams.
  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.”

Principle Suite 2 >> Scrum

The principles of Scrum (even if you do not perform full Scrum practices) are also essential. The following are the takeaways from Scrum, Doing Twice the Work in Half the Time, Sutherland.

  • Pick a Product Owner
  • Pick a team
  • Pick a Scrum Master 
  • Create and prioritise a product backlog
  • Refine and estimate the product backlog
  • Sprint planning
  • Make work visible
  • Daily stand up or daily scrum
  • Sprint review or sprint demo
  • Sprint retrospective
  • Immediately start the next sprint cycle , taking the team’s experience with impediments and process improvement into account.

https://youtu.be/58timOl2UYo

Principle Suite 3 >> DevOps; The Three Ways

If I have not mentioned this enough times already, you need to read the Phoenix Project, Gene Kim. In his book, Gene describes the Three Ways.

“The First Way emphasises the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g. a developer, system administrator).

The focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.

The outcomes of putting the First Way into practice include never passing a known defect to downstream work centres, never allowing local optimisation to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system (as per Deming).

The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.

The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.

The Third Way is about creating a culture that fosters two things: continual experimentation, taking risks and learning from failure; and understanding that repetition and practice is the prerequisite to mastery.

We need both of these equally. Experimentation and taking risks are what ensures that we keep pushing to improve, even if it means going deeper into the danger zone than we’ve ever gone. And we need mastery of the skills that can help us retreat out of the danger zone when we’ve gone too far.

The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.”

Practices

There are many ways to take the above and put them into practice and even with my Product Teams, each team has a slight variants in their application of the principles. However, there are a number of basics that they have all started with.

Warning: The following is not a step-by-step guide for implementing DevOps practices – it’s just how we started. You will need to take the principles and make this work for you in your environment:

Stage 1 >> Visualise the Workflow

This is the starting point. Build a Kanban Board. Don’t do this on paper or in a system – get yourself a big open wall space or a whiteboard. Something you can write on and keep in place – at least until you master the basics. Also, don’t do this alone – do this as a Product Team.

Now draw some columns – To Do, Doing, Done. Does this reflect what you do? Is this your actual process of delivering value? (This is not just for software by the way, I have a Kanban for household chores, DIY, etc.). Modify the Kanban lanes until they reflect what you actually do e.g. Define, Design, Build, Test, Release, Deploy, Done (You know that paper-based business management system that you have locked in some filing cabinet somewhere? That may help…).

Ok – Columns done? Now draw 4 rows. In DevOps, there are four types of work;

  • Business Projects – projects that customers want (e.g. creation of a new module or application, etc.)
  • Internal Projects – projects your product team need to do (e.g. Kaizan’s, Upgrades, Refactoring, Problem Tickets, Routine Operations, etc.)
  • Change – Modification of existing features or enhancements to existing modules (e.g. New workflow, portal enhancements, additional field creation, etc.)
  • Unplanned Work – Standard ITIL processes such as Request Fulfilment, Incident, Problem, Event and Alert Management.

Finally, agree as a team on how the work comes into the team – try and eliminate emails, phone calls, taps on the shoulder and have an agreed mechanism of how new work enters the backlog. In our case this happens via the Product Owner for Projects and Change or via the IT Service Management processes and toolset (ServiceNow) for Incident, Problem, Request, Event and Alert Management.

Stage 2 >> Visualise Workload

Now, everyone gets the infamous post-its and a marker pen. Create post-its for all the work that everyone is doing and mark your post-its with something like this:

Ideally, write the task in the form of a Story (As a <user>, I want <feature> so that <reason>). Keep a track of the Task ID as you will need this to help with measurements later on.

Everyone now populates the Kanban with all the work. In the unplanned work, you may have hundreds of incidents – perhaps categorise them and note the volume (e.g. P1 – 5, P2 – 12, P3, 2,500, etc) and place these in the relevant columns.

Finally, near the ‘Done’ column, write an agreed ‘Definition of Done’ so it’s all visible. This may include; Feature deployed to production, user acceptance, customer sign-off, documentation up-versioned, peer review complete, knowledge articles updated, etc.

When you have done this, reflect on what you have just done and what observations you all have as a team. Perhaps, you feel there is a lot of work on the board and in all columns. Perhaps you notice some people have more work than others. Maybe there is a whole load of work that you all were not aware about.

Take a bit of time to re-arrange these post-its so that the ones nearer the top of each row is the highest priority – This is where the Product Owner comes in. You should also remove any tasks that are not needed or not important. ‘Groom’ the board.

If you have time at this stage, use space around the Kanban to put up additional information:

  • Team vision and goals board
  • Release plan board – Set dates when things need to be done by
  • Observations board – What went well, what could be better, Improvement ideas
  • Questions board – There will be lots of questions, put these somewhere
  • Metrics – board… if you have service or delivery performance metrics, pin them up so everyone can see what you baseline is – talk about what needs to improve as a priority.

https://youtu.be/502ILHjX9EE

Stage 3 >> Establish Flow Management Structures

You now need to create ‘flow’. For every column on the Kanban (apart from ‘To Do’ and ‘Done’) set a Work In Progress Limit (WIP Limit). Don’t make this too large and set it to between 5 and 10 items. That is to say that there should not be more that 5 to 10 items in each column. You will find that after completing the above exercise that you will have exceeded the WIP limit in all active columns.

You may now want to look at each of the Tasks and apply estimates to each one. Traditionally, we estimate in time and if you have read Scrum (Sutherland) you will know that this is not the most effective measure of a task. Instead, I would recommend you use relative sizing, the Fibonacci sequence and a methodology called Planning Poker to estimate all the work. You could apply this to both the complexity of the work (Story Points) and to the value of the work (Value Points). Any task that cannot be completed in an agreed time box (Sprint) which could be 2 to 4 weeks, should be broken down into smaller ‘value-delivering’ segments.

Assuming you are not also recording the items in an Agile toolset such as Jira or ServiceNow, put together a spreadsheet and keep a note of each task/story:

  1. ID
  2. Task
  3. Dates that the item entered each column (Define, Build, Test, Release, etc.)
  4. Acceptance Criteria
  5. Story Points and Value Points
  6. Customer
  7. Team Owner
  8. Blockers

If you have a toolset, make sure you can capture the above in the tool and can extract the data or create reports. You should also be able to retrieve service performance metrics such as volume of incidents, SLA performance, availability, mean time to repair, incidents age, etc. from your service management toolset.

From this information you should now be able to create Cumulative Flow diagrams, Burn Up and Burn Down charts and understand your service and change delivery performance through service level metrics, throughput of change, story points and their lead and cycle time.

Stage 4 >> Visualise Flow

The Cumulative Flow Diagram shows all the work done as it passes through the stages. It provides a very visible view of the work and good ‘flow’ is represented by smooth upward curves. Spikes and flat lines show poor ‘flow’ and high variations between the various states show potential areas of blockage.

Burn Up Charts are extremely useful for a release. If you know a project is estimated at 1000 story points and your team velocity is 100 story points per sprint you can now estimate that the project will take 10 sprints to complete. You can also make decisions against scenarios where time needs to be fixed or scope needs to be increased. Furthermore, estimates can be made regarding how much value can be delivered at specific points in the project. Actual progress can be tracked against the Burn Up and this provides a very purposeful means of understanding the performance of a team against the estimated work.

A Sprint Burn Down is used as part of a sprint. The team’s velocity is known e.g. 100 Story Points per Sprint and the sprint backlog is set to 100 Story Points and at the end of each day, progress against the burn down is noted. This gives a day by day progress update to the whole team on how they are achieving against the Sprint goals.

If the Product Team supports the product, then unplanned work capacity needs to be factored in. This can be done using the ‘yesterday’s weather’ model which basically means that you take previous sprint’s capacity used for unplanned work, apply seasonal impact where appropriate, and plan this capacity into the Sprint.

https://youtu.be/XU0llRltyFM

Stage 5 >> Define Performance Metrics

Review the service measures such as SLA performance, availability and request fulfilment and review volumes of incidents and request. At your retrospective reviews, understand the root causes and submit improvement initiatives (Kaizen) into the backlog (under Internal Projects) – Do this every Sprint.

Also, review the number of changes delivered in the sprint and the corresponding Story Points. Take a look at the average time taken in each stage of the Kanban and, as a team, look at what improvements can be made. Focus on what the blockers were and optimise the constraint. The Theory of Constraints states that optimising anything other than the constraint is a waste.

Stage 6 >> Continuous Improvement (Kaizen)

The above charts and data points should give you the information needed to ask questions. As mentioned in a previous article, ask ‘why?’ 5 times on all the charts and data points. Take time getting to the root of the performance blocker. Usually, the ‘5th Why’ is a knowledge or understanding ‘thing’ so build this into the next sprint. Take at least one improvement (or Kaizen) into the next sprint. This is how you improve – get it done!

Stage 7 >> Initiate Ceremonies

Once you have visualised the workflow and the workload, defined your release plan which is visualised by a Burn Up Chart and you have all the necessary artefacts to measure and visualise performance, the team need to start the ceremonies. This needs to be done immediately after creating the Kanban board. These ceremonies include:

  • Daily Stand Ups >> A daily 15 mins stand up with the whole team – ideally face to face. Normal process is to go round the room and ask 3 questions; What did you do yesterday? What are you doing today? Do you have any blockers? We modified this in some of our teams where we start from the right hand side of the Kanban and we move tasks/stories from left to right with a brief update on each story. If a story cannot move, we start the 5 Whys. This has proved very effective. Each to their own though… Whatever you do, stop at 15 mins
  • Sprint Reviews >> At the end of the sprint, take time out as a team and go through the ‘Done’ column and take of the work that has been done. Discuss what was learnt and what could be better next time. Feel good at the achievement of all the work and start building the ‘Spirit of Success’. Reflect on whether the Sprint Goals were met and review the performance artefacts such as the Cumulative Flow Diagram, Burn Down, Burn Up and the performance metrics
  • Sprint Planning >> Following the review of the previous Sprint, plan the next Sprint. Take into account unplanned work, assign the next batch of stories that provide enough work commensurate with the velocity of the team.
  • Retrospectives >> Following the Sprint Review and Planning sessions, take time out to complete a retrospective. This is best done with a board and some post its as everyone needs to be involved. Ask the questions; What went well? What could have been better? Even better if? Take the items in the ‘Even Better If’ and if possible turn these into Stories and add them to the ‘Internal Projects’ backlog. This is the very essence of DevOps.
  • Backlog Grooming >> Mainly done by the Product Owner, Scrum Master and Product Analysts but take time to review and groom the backlog. Take things out, re-prioritise, etc. This will help with setting up the Sprint Planning Sessions.

https://youtu.be/QbPkcfzi2HI

Rinse and Repeat

Sprint after Sprint, Release after Release, Project after Project – a united stable team will improve as collaboration, team spirit and trust improves. They will understand the product and customer better and they will become more productive as they both build and support product with the right environment and support around them.

In time, they will remove technical debt thereby reduce incoming unplanned work which will improve user satisfaction as the service improves and they will create more capacity of value delivery and they will streamline the process, remove waste and accelerate the delivery of value. They will innovate and experiment (if allowed to do so) and this will lead to new ideas making their way to production, automation and orchestration of delivery and hackathons with customers.

Over time, the product will evolve beyond what you could imagine or have predicted – some thing may not be great, some ideas may have got thrown away but if you build the right teams, give them the right environment and principles to build good practices upon with time to innovate the improve, they will create a great product but more importantly they will form into an awesome team.

Mastery takes practice.

"Try not. Do. Or do not, there is no try" Yoda.

Results

We have seen amazing results which include improved service performance, increased throughput, faster and more predictable change, happier employees and more satisfied customers.

Read more of what we achieved in the Story of DevOps.

More in this series:

  • A Story of DevOps
  • Episode 1 >> Origins (Traversing the Change Curve)
  • Episode 2 >> One Ring (Alignment and Empowerment)
  • Episode 3 >> Freedom (Leadership)
  • Episode 4 >> Assemble (Productive and Teams)
  • Episode 5 >> Shield (Tools of the Trade)
  • Episode 6 >> Kryptonian (Value, Flow, Quality in a Complex World)
  • Episode 7 >> Jedi (Mastery)
  • Episode 8 >> Balrog (Confront the Brutal Facts)
  • Episode 9 >> Kryptonite (Anti-Patterns of DevOps)
  • Episode 10 >> The Suit (Digital Transformation)
  • Episode 11 >> Infinity (Automation and Orchestration)

#AStoryofDevOps #DevOps #Jedi

Raj Fowler

I am a natural, enthusiastic and authentic leader who understands the impact of IT as a differentiator for business performance and how organisational culture directly influences IT and business performance. With a strong appreciation of the changing technology environment, I have spearheaded a transformation of organisational ‘ways of working’ through adoption of the philosophies and principles that underpin DevOps, Agile and Lean. As a result, I have a track record of delivering operational excellence whilst improving IT agility, security and responsiveness enabled through close business relationships, technology-led thinking and inspirational leadership. 3 years ago we delivered about 50 to 100 changes per annum across about 80 business systems, all of which we managed using our standard plan, build and operate practices. Change was difficult and the transition from project to service was painful. I lived the opening chapters of the Phoenix Project on a regular basis. I now manage over 100 business systems and we categorised 7 of these as Products which include ServiceNow, Salesforce, Cognos, SuccessFactors, SharePoint. SAP and Bespoke Applications where we have established Product Teams. These are teams that build and support Product using DevOps philosophies. As a result, we delivered over 2500 changes last year whilst at the same time improving the service by 30%, improving customer satisfaction and employee morale. Using the "you build it, you support it" (Amazon) mentality, change is no longer a big event but a normal everyday occurrence and the usual spikes of incidents are no longer there, in fact with each change we deploy, the incident volumes reduce! Teams are not only delivering new features but are cleaning up their code and removing technical debt with every release. As a result, we have made a significant impact to the efficiency of our internal functions which in turn helps the competitiveness and profitability of the enterprise.

Have Your Say: