DMO scores project maturity

The DMO's overhaul of its internal management systems has resulted in a system which enables the organisation for the first time to put hard numbers on project risk and maturity.
Have you ever wondered just how it is determined that a major defence procurement project is ready to progress further through the system? Of course, there are indications in the Defence Capability Plan as to what year a project will be put forward for first or second pass government approval, but just about everybody in Defence and industry is well aware that these times are slipped as often as they are met.

But why? Is it merely incompetence and lethargy on the part of bureaucrats, dithering and parsimony by government, or industry's failure to deliver what they promise? A cynic might suggest that it can be an unhealthy combination of all those reasons, and sometimes a cynic would be right.

However, over the past couple of years the DMO has developed a comprehensive system of reporting based on empirical judgement against key criteria for major projects; a system that generates an easy to understand number which defines whether or not the project in question is ready for the next step. This system is used internally by the DMO to keep its CEO and other senior executives up to date, across Defence in relevant committees and to keep government informed. It is so simple that politicians like it. Here's how it works.

There are seven factors that have been deemed crucial to a project's overall success. These are: schedule, cost, requirement, technical understanding, technical difficulty, commercial, and operational and support areas. Each area is broken down into a significant amount of detail and, of course, the various areas are more important at different times in the project's overall life cycle. A score of 10 is available in each area making for an overall possible score of 70.

The magic number for first pass approval is around in the low 20's, while the score needed for a project to be offered to government for second pass approval is around 35. The latter is not a mandatory figure, but it is a rule of thumb that both the DMO and the Capability Development Group see as quite significant.

It should be emphasised that this system is the DMO's way of keeping its house in order and that it is subject to outside influences. For example, an urgent operational requirement or change in strategic circumstances might see a project called forward by government before the costs and risks involved had been analysed as thoroughly as normal. Conversely, budget problems or a change in strategic priorities might see a project delayed even though its maturity score was satisfactory. In the majority of cases, however, the maturity score is what determines whether and when a project progresses through the system.

Underneath each of those seven key headings is a series of carefully chosen and defined words to explain each of the possible scores between one and ten. A score of one indicates high risk while a score of 10 means the risk is low. The terms and definitions used for each score vary under each of the seven key headings; the specific nature of each heading dictates this. However, much effort has gone into ensuring that the maturity score is a useful standardised system that can readily be applied across the broad range of projects handled by the DMO.

As an example consider the terms used under the key heading "Commercial", which has two subheadings: "Can industry deliver the solution?" for scores one to five and "How well is industry performing?" for scores six to 10. The term "not yet" indicates a score of one and is defined as "No industry sources or technology solution precedents have been identified". The term "possible" is used alongside a score of two points and is defined: "Likely industry sources have been identified and a high level acquisition strategy can be developed".

Moving further up the scale to "Industry proposal", the term used to describe a score of four is defined: "Industry proposals/DMO investigations indicate that capability and capacity exists in industry to deliver the solution". "As contracted" describes a score of six and means "Contractor has the necessary plans, skills and capacity to undertake the work and can ramp-up resources needed to commence work as planned".

Nearing the top of the scale is "Delivering" which appears alongside a score of eight and is defined: "Contractor is performing and delivering as contracted". At the very top is "Preferred supplier" which appears alongside the perfect 10 and means "Contractor's performance would merit their consideration as a preferred supplier for future similar requirements".

Each of the scores is colour coded with one being red as the highest risk score. The colour boxes go green at a score of six and continue to get brighter from there on upwards. There is a tolerance of plus or minus one for the scores awarded under each of the seven key headings.

A ty
This maturity score system recognises that much of the risk entailed in a project is commercial and in its execution rather than scientific, while still recognising the necessity of a high level of technical understanding and technical feasibility assessments. The system also makes the decision process more scientific as it is based much more heavily on data than emotion.

Maturity scores are not always an even and upward accumulation of points. Scores can go backwards for a variety of reasons including technical difficulties, contractor problems or operational support issues to name just a few.

The maturity score system is used to generate reports on the top 80 or so DMO projects every month, and these are arranged in a simple easy to read format. Key issues such as the technical aspects, project management, schedule and cost are summarised briefly on the front page. Overleaf scores appear for the seven key headings in colour-coded boxes; any more than two reds will automatically elevate it (often to the CEO) for project review.

One of the main benefits of this scoring system is that it is easy to understand, and that allows key decision makers to divorce themselves, if need be, from spin that may be put on a project by any one of a number of interested parties. All the management and technical jargon is reduced to one simple number that gives a pretty clear indication as to whether or not a project is tracking as it should. It also facilitates a more seamless hand over if new project managers take over.

This system ought to prevent a project progressing before it is ready, and thus alleviate problems downstream. It is an interesting if academic exercise to speculate just where some of the DMO's walking wounded would be, in fact if those projects had progressed at all, if such a system had been in place and rigidly enforced in the mid to late 1990s.

Another benefit of this system is that it focuses the mind of every busy project manager once a month, and puts the key issues right at the forefront of his or her mind. In addition to the discipline this brings to the human element of program management, it ought to give early warning of areas not developing or maintaining their momentum as they should.

The maturity score system can also be used to measure the effectiveness of in service arrangements. The sustainment score is also out of 70 and peak scores occur when full capability is achieved after initial introduction of the equipment, but inevitably dip as the equipment ages and more maintenance is required. Persistently low sustainment scores are an indicator of the need for fleet replacement or a mid life upgrade.

It is clearly in industry's interest to understand how this system works - after all, quicker project approval leads to cash flow sooner; a situation that would have to please most in business if they could make it happen. From the DMO's point of view industry could help project maturity scores to accumulate faster by supplying appropriate data in a timely fashion, and also by offering appropriate contractual terms and conditions.

Defence business is like any other competition with its fair share of risks and rewards, success and failure. For those who play to win it is always good to know that someone is keeping score.

By Daniel Cotterill, Canberra