Tuesday, June 27, 2017

Lean Software Development

Lean thinking is a business methodology that aims to provide a new way to think about how to organize human activities to deliver more benefits to society and value to individuals while eliminating waste. The term lean thinking was coined by James P. Womack and Daniel T. Jones, in their 1996 book with the same name, to capture the essence of their in-depth study of Toyota’s fabled Toyota Production System. Lean thinking is a new way of thinking any activity and seeing the waste inadvertently generated by the way the process is organized by focusing on the concepts of:

1. Value,
2. Value streams,
3. Flow,
4. Pull,
5. Perfection.

The aim of lean thinking is to create a lean enterprise, one that sustains growth by aligning customer satisfaction with employee satisfaction, and that offers innovative products or services profitably while minimizing unnecessary over-costs to customers, suppliers and the environment. The basic insight of lean thinking is that if you train every person to identify wasted time and effort in their own job and to better work together to improve processes by eliminating such waste, the resulting enterprise will deliver more value at less expense while developing every employee’s confidence, competence and ability to work with others.

The idea of lean thinking gained popularity in the business world and has evolved in two different directions:

1. Lean thinking converts who keep seeking to understand how to seek dynamic gains rather than static efficiencies. For this group of thinkers, lean thinking continuously evolves as they seek to better understand the possibilities of the way opened up by Toyota and have grasped the fact that the aim of continuous improvement is continuous improvement. Lean thinking as such is a movement of practitioners and writers who experiment and learn in different industries and conditions, to lean think any new activity.

2. Lean manufacturing adepts who have interpreted the term “lean” as a form of operational excellence and have turned to company programs aimed at taking costs out of processes. Lean activities are used to improve processes without ever challenging the underlying thinking, with powerful low-hanging fruit results but little hope of transforming the enterprise as a whole. This “corporate lean” approach is fundamentally opposed to the ideals of lean thinking, but has been taken up by a great number of large businesses seeking to cut their costs without challenging their fundamental management assumptions.

The essence of [the Toyota system] is that each individual employee is given the opportunity to find problems in his own way of working, to solve them, and to make improvements.
The above quote underscores a vital point because over the years there have been some ostensibly ‘lean’ promoters that reduced lean thinking to a mechanistic, superficial level of management tools such as Kanban and queue management. These derivative descriptions ignore the central message of the Toyota experts who stress that the essence of successful lean thinking is “building people, then building products” and a culture of “challenge the status quo” via continuous improvement.

Lean software development is a translation of lean thinking to the software development domain. The term lean software development originated in a book by the same name, written by Mary Poppendieck and Tom Poppendieck. The book restates traditional lean principles, as well as a set of 22 tools and compares the tools to corresponding agile practices.

Lean software development can be summarized by the following nine principles, very close in concept to lean manufacturing principles:

1. Optimize the Whole
2. Eliminate Waste
3. Build Quality In
4. Deliver Fast
5. Defer Commitment
6. Create Knowledge
7. Empower People
8. Focus on Flow (of Value)
9. Continually Improve

Optimize the whole

Optimize the whole means
- to focus on the realization of business value
- take a systemic thought process.
- to take an holistic view.

Taking a systemic view means that one must attend look to the system within which people operate to understand why people are achieving the results they are.  Changing the system will help improve their results.  Effective results require good people and good systems.

The larger the system, the more organizations that are involved in its development and the more parts are developed by different teams, the greater the importance of having well defined relationships between different vendors, in order to produce a system with smoothly interacting components. During a longer period of development, a stronger subcontractor network is far more beneficial than short-term profit optimizing, which does not enable win-win relationships.

Systems thinking implies taking an holistic view since systems are more than the sum of their parts. This is one of the big differentiators of Lean and Agile. One must understand that the development team is only a part of the process.  Many of their challenges are due to upstream problems. If we can see the whole we can more easily change behavior of those upstream of the development team.

Eliminate waste

Eliminate waste by removing delays, only working on things of value that you know how to achieve and only starting work you know you can complete.  We must remove delays in our workflow because these literally create additional work to be done – redoing requirements, finding errors (while the fixing cost often stays the same, but finding bugs found late increases dramatically), integration errors (caused by the delays from when teams get out of synch until they are discovered in the integration process.

Lean philosophy regards everything not adding value to the customer as waste (muda). Such waste may include:

1. Building the wrong feature or product
2. Mismanaging the backlog
3. Rework
4. Unnecessarily complex solutions
5. Extraneous cognitive load
6. Psychological distress
7. Waiting/multitasking
8. Knowledge loss
9. Ineffective communication.

In order to eliminate waste, one should be able to recognize it. If some activity could be bypassed or the result could be achieved without it, it is waste. Partially done coding eventually abandoned during the development process is waste. Extra processes like paperwork and features not often used by customers are waste. Switching people between tasks is waste. Waiting for other activities, teams, processes is waste. Motion required to complete work is waste. Defects and lower quality are waste. Managerial overhead not producing real value is waste. A value stream mapping technique is used to identify waste. The second step is to point out sources of waste and to eliminate them. Waste-removal should take place iteratively until even seemingly essential processes and procedures are liquidated.

Removing delays

Removing delays requires not working beyond the capacity of your teams as this injects delays into the workflow because people have to switch between projects and are often waiting for other people working on other projects. People often point to the cost of task-switching here, which, while considerable, pales in comparison to the additional work created by the delays themselves.

Build quality in

Build quality in means to avoid errors, not test them out.  Shortening feedback loops is a big help here.  In particular, the adoption of acceptance test-driven development is a method that virtually every team we’ve run across in the last dozen years would benefit from.

The customer needs to have an overall experience of the System. This is the so-called perceived integrity: how it is being advertised, delivered, deployed, accessed, how intuitive its use is, its price and how well it solves problems. Conceptual integrity means that the system’s separate components work well together as a whole with balance between flexibility, maintainability, efficiency, and responsiveness.

This could be achieved by understanding the problem domain and solving it at the same time, not sequentially. The needed information is received in small batch pieces – not in one vast chunk - preferably by face-to-face communication and not any written documentation. The information flow should be constant in both directions – from customer to developers and back, thus avoiding the large stressful amount of information after long development in isolation.

One of the healthy ways towards integral architecture is refactoring. As more features are added to the original code base, the harder it becomes to add further improvements. Refactoring is about keeping simplicity, clarity, minimum number of features in the code. Repetitions in the code are signs of bad code designs and should be avoided.

The complete and automated building process should be accompanied by a complete and automated suite of developer and customer tests, having the same versioning, synchronization and semantics as the current state of the System. At the end the integrity should be verified with thorough testing, thus ensuring the System does what the customer expects it to. Automated tests are also considered part of the production process, and therefore if they do not add value they should be considered waste. Automated testing should not be a goal, but rather a means to an end, specifically the reduction of defects.

Deliver fast

Deliver fast is essentially the same as time to market.  Not a new concept.  But Lean adds the insight that if we focus on shortening the time of delivery by removing delays and focusing on quality, our productivity goes up and costs come down.  Quick delivery also allows for quick feedback and can help avoid building features that aren’t needed.

In the era of rapid technology evolution, it is not the biggest that survives, but the fastest. The sooner the end product is delivered without major defects, the sooner feedback can be received, and incorporated into the next iteration. The shorter the iterations, the better the learning and communication within the team. With speed, decisions can be delayed. Speed assures the fulfilling of the customer's present needs and not what they required yesterday. This gives them the opportunity to delay making up their minds about what they really require until they gain better knowledge.

Customers value rapid delivery of a quality product. The just-in-time production ideology could be applied to software development, recognizing its specific requirements and environment. This is achieved by presenting the needed result and letting the team organize itself and divide the tasks for accomplishing the needed result for a specific iteration. At the beginning, the customer provides the needed input. This could be simply presented in small cards or stories – the developers estimate the time needed for the implementation of each card. Thus the work organization changes into self-pulling system – each morning during a stand-up meeting, each member of the team reviews what has been done yesterday, what is to be done today and tomorrow, and prompts for any inputs needed from colleagues or the customer. This requires transparency of the process, which is also beneficial for team communication.

Defer commitment

Defer commitment means to make decisions when you have the right information yet always attend to the cost/risk of making decisions too early or too late.  Too many decisions are made too early. Deferring commitment requires taking steps so that delaying the commitment does not incur more risk while achieving better decisions when more information is available later.

As software development is always associated with some uncertainty, better results should be achieved with an options-based approach, delaying decisions as much as possible until they can be made based on facts and not on uncertain assumptions and predictions. The more complex a system is, the more capacity for change should be built into it, thus enabling the delay of important and crucial commitments.

The iterative approach promotes this principle – the ability to adapt to changes and correct mistakes, which might be very costly if discovered after the release of the system. An agile software development approach can move the building of options earlier for customers, thus delaying certain crucial decisions until customers have realized their needs better. This also allows later adaptation to changes and the prevention of costly earlier technology-bounded decisions. This does not mean that no planning should be involved – on the contrary, planning activities should be concentrated on the different options and adapting to the current situation, as well as clarifying confusing situations by establishing patterns for rapid action. Evaluating different options is effective as soon as it is realized that they are not free, but provide the needed flexibility for late decision making.

Create knowledge

Create knowledge means  to create, capture, update, and put to use the knowledge learned from product value.  Measure cycle time, work-in-progress, and which work-items have value.

Empower people

Empower people to make decisions which they are best able to make. Teams must be given clear boundaries within which to work, however. It also means to recognize the psychology of people. People are tribal, have a psychology that is sensitive (and averse) to significant change.  People also often identify with their work, so changing roles can be very difficult for them emotionally.
The lean approach follows the Agile Principle "find good people and let them do their own job", encouraging progress, catching errors, and removing impediments, but not micro-managing.

Another mistaken belief has been the consideration of people as resources. People might be resources from the point of view of a statistical data sheet, but in software development, as well as any organizational business, people do need something more than just the list of tasks and the assurance that they will not be disturbed during the completion of the tasks. People need motivation and a higher purpose to work for – purpose within the reachable reality, with the assurance that the team might choose its own commitments. The developers should be given access to the customer; the team leader should provide support and help in difficult situations, as well as ensure that skepticism does not ruin the team’s spirit.

Focus on flow 

Focus on flow of value.  We accomplish this by managing our queues so work flows from concept to consumption smoothly.  Managing work-in-progress (WIP) is an effective way of doing this.  In particular, we must pay attention to delays, which literally increases the amount of work to be done.

Continually improve

Continually improve with a scientific method.  While software development appears complex, that is often because many of us have taken a simplistic approach.

Software development is a continuous learning process based on iterations when writing code. Software design is a problem solving process involving the developers writing the code and what they have learned. Software value is measured in fitness for use and not in conformance to requirements. Instead of adding more documentation or detailed planning, different ideas could be tried by writing code and building. The process of user requirements gathering could be simplified by presenting screens to the end-users and getting their input. The accumulation of defects should be prevented by running tests as soon as the code is written. The learning process is sped up by usage of short iteration cycles – each one coupled with refactoring and integration testing.

Increasing feedback via short feedback sessions with customers helps when determining the current phase of development and adjusting efforts for future improvements. During those short sessions both customer representatives and the development team learn more about the domain problem and figure out possible solutions for further development. Thus the customers better understand their needs, based on the existing result of development efforts, and the developers learn how to better satisfy those needs. Another idea in the communication and learning process with a customer is set-based development – this concentrates on communicating the constraints of the future solution and not the possible solutions, thus promoting the birth of the solution via dialogue with the customer.
Posted on Tuesday, June 27, 2017 by Henrico Dolfing