Methodology Effectiveness Measurability And Optimizability

This blog addresses two questions: (a) How to determine if a methodology is better for the situation? (b) Can this methodology be further improved with cost and benefits well balanced?

A methodology without effectiveness measurement cannot be optimized with any certainty. Ambiguity will only lead to argument, ineffective spending, vendor exploitation, room for politics, and a whole slew of inefficiencies. A very simple example will be that of the traditional traveling salesman's problem in the industrial engineering discipline. The objective is to minimize traveling cost, which is linearly related to the distance. Without quantification, evaluation of effectiveness of the problem will be impossible. The keyword is linearity. However, there are apparent non-linear factors in software development. For example, risks, randomness in the delivery team's skills, and other human factors that affect the effectiveness of producing the results. As in engineering, when things are non-linear, we linearize them using approximation, which is accurate within a known range. So, this is very different from human guestimation which one cannot even quote a range with any true confidence. Of course, providers of agile service tend to be wary of measurability, because the results might not justifiably reflect the methodology itself. For instance, human factors can be muffling the benefits of a methodology. However, there is no excuse to this. Methodology and human factors need to be coupled with each other properly for any process to be successful.

Most businesses are already doing some agile practices without labeling it as such. Quarterly budgeting is a good example of short iteration. By retrospecting on the profit, loss, expenditure, etc., in quarterly cycles, better decisions can be made. This resembles integrate early and often agile principle. In this case, the integration being the external market and operations of the company. Another example will be the Stage Gate Management method in the product management field (http://www.prod-dev.com/stage-gate.php). It is not hard to see that Stage Gate and Lean's value stream mapping are compatible. The latter tends to be a bit single lined, while Stage Gate takes in the program/portfolio level view as well. The gist is that external factors like the Porter Five Forces aside, by examining the development gate, indirect measurement of overall success/failure of a software methodology can be analyzed. Again, there is no excuse. Methodology and human factors are flip sides of the same coin in the hands of management. How many coins hit the target bin is quantifiable. Methodology and how it is applied must be considered together.

Certainly, finer grain measurement helps visualization and hence improvement. Number of lines of code produced per period, or number of bugs are simple but insufficient indicators of quality or performance because these numbers change when the technology or the experience of the delivery team change. Imagine that the team now switched from low level language like C to Python. Then the number of lines of code decreases naturally. On the other hand, if there are turnover in a team, then perfomance will change but not directly reflective of the methodology itself. Also, maturity of the software also affects the "meaning" of these numbers.

In the finance world, ratios are often used to evaluate health of a company. It is definitely not my intention to go too deep on this subject. My point is that by applying ratios, or ratios of ratios, and so on in a Stage Gate fashion on Lean's value stream of software development, we can get quantitative measurement approximation linear/good enough for relative comparison. This makes continuous improvement possible. Otherwise, a change might achieve the opposite effect and get lost in the noises. In a top-down fashion, high level effectiveness measurements can be fine tuned and scoped in to gain more detailed insights into process strengths and weaknesses.

As an example, instead of using defects per line of code, we can use defects per effort hours. This prevents programming language/tools/technology from clouding the meaning of "process" or "methodology" quality. For example, if we use C language to implement a dynamic web application, then no matter what libraries or API we use, the number of lines will bound to be larger than those using frameworks in Python, PHP, or Ruby on Rails. This is similar to earned value analysis technique in project management, because effort hours is probably the highest common denominator of human intellectual factors.

As far as optimization goes, with some consistent measurement framework in place, scenario analysis using expected value techniques which take into account the risk, cost and benefits will aid better decision making. Certainly, the measurement framework might need to change too. In that case, lower scoped measurements will become obsolete.

Last but definitely not the least, measurement must be taking a global view. Optimizing just one stream and hurt ten others is unwise unless that single stream worths that much. Program/portfolio analysis must be considered at the same time. So often, agile or lean proponents talk in one dimension. For example, the classic cost of defects vs time curve is really a one dimensional view of the whole deal. On the other hand, maintaining flow is great, but if the overall cost does not justify the benefits, then we will need to find a better balance point. This is certainly no simple matter, and the number of scenarios will get exponentially larger as size of the company or number of initiatives increases. Simulations like Monte Carlo analysis, and the likes should be used together with expected value techniques in statistics to help guide the decision process.

In conclusion, to answer question (a) and (b), it is possible and necessary to use proper metrics instead of BS to measure effectiveness. A global portfolio/program level consideration must be taken into account as well. Isolated success on a stream or two does not guarantee overall success of the company.