Thursday, May 11, 2017

Agile Project Portfolio Management? Evaluation

This is the fourth article in our agile project portfolio management series. The first article explored the concept of project portfolio management and how it relates to projects that are executed with agile delivery methods. The second article discussed demand management, and the third was about categorization. This article is about portfolio evaluation.

Within project portfolio management, new projects are categorized, evaluated, selected and prioritized; existing projects may be accelerated, killed or de-prioritized; and resources are allocated and reallocated based on maximizing productivity.

Project portfolio management does not involve making project-by-project choices based on fixed acceptance criteria. Instead, decisions to add or subtract projects from the portfolio are based on the three goals of project portfolio management:

 Maximize the value of the portfolio;
2. Seek the right balance of projects, thus achieving a balanced portfolio;
3. Create a strong link to strategy, thus: the need to build strategy into the portfolio;

The reality about project portfolio management that most consultants and software vendors don't want to hear is that the necessary model for project evaluation needs to be different for different clients. That is because what clients need from their projects to be successful differs wildly, even for similar organizations within the same industry having an almost identical strategy.

We will describe a number of portfolio evaluation tools and methods in relation to the three major goals of portfolio management demonstrating how the tools could help in the evaluation, selection and prioritization process.

But before we can start with discussing these evaluation tools and methods we need to gather some additional attributes for our Project Backlog. In order to evaluate portfolio value, we should know risks and constraints. In order to balance we should know risks. The base for creating a strong link to the strategy we already build in our categorization process by linking each Project Backlog item to one or more strategic goals.

Portfolio constraints

Typically the two most important constraints are money and people. The first one is easy, the second one is harder to specify. You could argue that the only constraint here is money because you could hire specialist people with money when you have not enough people to execute a project. But in reality this is a) not economical, and b) it will cost a lot of time for them to understand the organization specific things.

One approach to constraint definition that I have taken with success is to take the headcount for each organizational unit and then take a percentage of the headcount that is available for project work. This percentage can be different for each organizational unit. Say for example a business unit has 20%, a software development unit has 80% (the rest is maintenance), and your PMO department has 100%. Now you have to identify for each item on your Project Backlog list which organizational units would be involved for how many person days (just rough estimations). These can then compared with how many days are already taken by existing and selected projects. It is far from perfect, but this will actually give you a pretty good idea if a certain unit could handle the volume of selected projects without new hires and external help.

Risks, issues, and interdependencies

Known risks and issues for all items on the Project Backlog list should available. This means that budget overruns, late delivery, resource shortage, skills mismatch, and others can be factored into the evaluation and decision-making process. Besides risk and issues, the interdependencies between projects should be known. As already discussed in the previous article you could group a set of projects into a program and evaluate this as a whole, or just identify the links between projects. You can imagine what happens when you killed a project and you were not aware that the software planned to be delivered in that project was essential for your flagship project to succeed.

Maximizing the value of the portfolio

The first goal of project portfolio management is maximizing the value of the portfolio. This means that each individual project should represent good value to the organization and the collection of projects must make efficient use of the resources available. Where projects compete for scarce resources it must be clear how the allocation between them is to be made based on value.

Below you will find a number of evaluations that can be made on your Project Backlog. In my opinion, you should select more than one for your own evaluation strategy and compare their value during the evaluation.

Net Present Value and  Return On Investment
Net Present Value (NPV) is the difference between the present value of cash inflows and the present value of cash outflows. NPV is used in capital budgeting to analyze the profitability of a projected investment or project. Basically, it represents the net result of a multiyear investment (expressed in $). The following is the formula for calculating NPV:


Ct = net cash inflow during the period t
Co = total initial investment costs
r = discount rate, and
t = number of time periods

With the NPV one must consider that this value is not used to ascertain levels of investment. It is simply a number by which the investor is aware of the amount of cash flow that he is receiving as a result of an investment. It is also used to measure (or predict) the amount of cash flow that is to come in the future; it does not look at profits and losses in a traditional sense, as it takes into consideration the discount percentage.

The Return on Investment (ROI) is a performance measure used to evaluate the efficiency of an investment or to compare the efficiency of a number of different investments. ROI measures the amount of return on an investment relative to the investment’s cost. To calculate ROI, the benefit (or return) of an investment is divided by the cost of the investment, and the result is expressed as a percentage or a ratio.

As regards the ROI, one must consider that it has no real merit when trying to calculate future cash flow based on the cash flow that an investment is accumulating today. It is simply a way to measure the amount of profit (or return) an investment is making over time. However, it is a great indicator of where your investment stands. Because of its simplicity and versatility, it is one of the most commonly used values to measure the success of an investment. If an investment returns a negative ROI or there are more lucrative opportunities using a different ROI all together, then the investment should not be considered at all.

Both values do have their downsides. In terms of the NPV, this value does consider the discount rate of the dollar value at present and in the future; however, it doesn’t exactly calculate the initial investment that must be dedicated. If two investments have an NPV of $100.000, it may seem as if both were lucrative; however, if one has requires a dedicated investment of $10.000 and the other $1.000.000 it’s obvious the former is a much more promising investment. 

A big drawback of both methods is that they need estimations as input. When you look at the cone of uncertainty you understand that at the start of a project (project proposal) the estimations are typically way off from reality. They are the most certain when the project is finished and benefits have been measured, but then it is too late to be useful in the PPM evaluation process because the project is already finished. So what to do?

I have good experiences with using the range-based estimations we discussed in previous articles. Just take the following four pairs:

1. Upper bound for the discounted cash flows and lower bound for the investment
2. Upper bound for the discounted cash flows and upper bound for the investment
3. Lower bound for the discounted cash flows and lower bound for the investment
4. Lower bound for the discounted cash flows and lower bound for the investment

You then have four different NPVs/ROIs. When you give a probability for each outcome you have everything setup for running a Monte Carlo Simulation on this value. Monte Carlo simulation, or probability simulation, is a technique used to understand the impact of risk and uncertainty in financial, project management, cost, and other forecasting models.  See my article "Estimating with Wideband Delphi and Monte Carlo Simulation" for a detailed explanation on how to do such an analysis.

Expected Commercial Value (ECV)
ECV seeks to maximize the expected value, or expected commercial worth of the portfolio,
subject to certain budget constraints.

ECV= ((PV*Pcs –C)*Pts)*D

ECV = the expected commercial value
PV = the present value of cash flow, after launch (this is strictly income stream; none of the project costs – development, capital, and so on – have been subtracted from this stream)
Pcs = the probability of commercial success (a number from 0 to 1.0)
C = commercialization or launch cost remaining to be spent on the project
Pts = the probability of technical success (a number from 0 to 1.0)
D = development costs remaining to be spent

As with the NPV, a big drawback of this method is that it needs estimations as input.

The Productivity Index (PI)
PI is a variant of the ECV method that considers risks and probabilities of projects; it shares many strengths and weaknesses. The PI tries to maximize the financial and economic value of the portfolio for given resource constraints.

PI =(ECV * Pts – Development) / Development

ECV = the expected commercial value
Pts =the probability of technical success
Development = the Development costs remaining in the project

Options Pricing Theory (OPT)
OPT is a fairly new concept but some financial experts claim it to be the only correct valuation method because they consider that NPV and DCF are “misused” and sometimes unfairly penalizes certain types of projects, especially high-risk projects. When the project is a high-risk project, i.e. a project where the probability of technical or commercial success is low and the cost to undertake the project is high then NPV and DCF considerably understate the true value of the project.

Dynamic Rank-Ordered List
The advantage of this method is that it can rank-order projects according to several criteria concurrently, without being as complex and time-consuming as a full-fledged, multiple-criteria model. NPV, ROI, strategic importance of the project, probability of technical success can be some of the criteria.

Scoring Models
Some of the more important success factors that are correlated with the probability of the new product are the following:

- Having a unique, superior product
- Targeting an attractive market
- Leveraging internal company strength (products and projects, competencies, and experience in both marketing and technology).

These strong predictors of success as tools should be used for selecting projects. For more such predictors see my articles on "Why are we actually doing this project?" and "Questions to ask before you develop your new product".

Check Lists
Some firms use checklists instead of scoring models at their review or Go/Kill meetings. The questions are similar in both methods but the scoring procedure and end result are quite different. In a checklist method, the answers are yes/no. A single “no” answer is a knockout: It kills the project. The checklist is most effective at project review to weed out poor projects but they are not good for project prioritization since there is no 0-100 total project score that facilitates rank-ordering projects.

Paired Comparison
With this method, we compare project ideas against each other, one pair at a time. Here the question, “If you had a choice, which of the two projects would you do?” There is a discussion, and a consensus vote is reached on each pair. Projects are then rank-ordered according to the number of times they receive a “yes” vote in each paired comparison. This method is very useful at the very beginning of projects when almost no information is available.

Achieving a Balanced Portfolio

Bubble Diagram
The risk-value portfolio bubble chart represents a portfolio view of all (or selection of) projects and puts projects into one of four quadrants based on value and risk; this is important for identifying projects that drive overall greater value to the organization compared to other projects as well as highlight projects that should likely be screened out.

One of the key benefits to a portfolio bubble chart is to quickly show the balance of the current portfolio.  Using portfolio bubble charts with the Portfolio Owner can focus conversations to help better manage the portfolio. When reviewing projects that are in the higher-value/ lower-risk quadrant, the Portfolio Owner should ask the question, “how can we get more of these types of projects in the portfolio?” Likewise, with the lower-value/higher-risk projects, the Portfolio Owner should ask how to avoid those types of projects. These discussions will greatly enhance the management of the portfolio and enable the Portfolio Owner to “manage the tail” and ensure that only the best projects are selected and executed.

There are four primary data elements needed to build the risk-value bubble chart: value scores for each project, risk scores for each project, categorical data, and the project cost or financial benefits of the project (commonly used for bubble size). A prioritization scoring mechanism is typically required to build the best portfolio bubble charts (we will discuss prioritization in an upcoming article).

Build Strategy into the Portfolio

Strategic fit is the first and easiest to envision because it addresses the question: “Are your projects consistent with your articulated strategy?” How to link strategy with project backlog items we discussed in the previous article. You could score each strategy link with a value between 1 (weak) to 5 (very strong) for a scoring model as described above.

Strategic Buckets Model
Strategic Bucket Model focuses more on resource allocation. It answers the questions: “If this is our strategy, then how should we be spending our development funds?” It starts with strategic goals and then moves to setting aside funds or buckets of money destined for each strategic goals. The goal of the portfolio is to fill as many buckets as possible in order to create a strategically balanced portfolio.


- you can make your portfolio evaluation as simple or complicated as you want.
- the more you rely on estimations, the more you have to think about how wrong these typically are.
- each attribute you want to use for your evaluation somehow has to be captured. This can make your demand management and categorization very complex and create a high barrier for demand collection.
- you have to match your evaluation strategy with what is important for your company. Never just blindly follow consultants our vendor recommendations.

The next article in this series will be about portfolio funding. Originally that article would be about portfolio alignment, but when I started writing it I felt that a piece of the puzzle was missing. Namely funding. That is why I decided to write that article first, and save alignment for the next.

Other articles in this series:

1. Agile Project Portfolio Management?
2. Agile Project Portfolio Management? Demand Management
3. Agile Project Portfolio Management? Categorization
4. Agile Project Portfolio Management? Evaluation
5. Agile Project Portfolio Management? Funding
6. Agile Project Portfolio Management? Alignment
7. Agile Project Portfolio Management? Authorization
8. Agile Project Portfolio Management? Monitoring
9. Agile Project Portfolio Management? Conclusion
10. The Agile Project Portfolio Management Framework Guide
Posted on Thursday, May 11, 2017 by Henrico Dolfing