Defence T&E processes under audit

Defence's Test & Evaluation (T&E) processes have come under the scrutiny of the ANAO; we can expect a report later this year.
What was planned as an examination by the Australian National Audit Office (ANAO) of the processes concerned with the acceptance and introduction into service of major capital equipment acquisitions is now likely to be broadened to include virtually all test and evaluation (T&E) activities throughout the Defence organisation.

Driven partly by acquisition problems in the past but perhaps more so by the cost imperatives of managing risk more effectively in today's budgetary climate, T&E is now widely accepted within Defence as a most important guiding principle in the capability development and acquisition process.

Colonel Brian Hall, who is one of a limited number of trained T&E experts within Defence, says T&E is an essential risk management tool that must be factored into the formative stages of capability development. "T&E is a risk management tool that is applied as a fundamental part of risk management strategy - not in blind or slavish fashion as a last minute check on the suitability or otherwise of equipment," he says.

Hall points out that because its applications range from operational test and evaluation of major platforms or weapon systems, to modelling and simulation analyses of future capability concepts which may have no current practical application, T&E may well mean different things to different people. But the principle remains the same, he adds.

While the ANAO report may include some case studies, its main focus will be on the effectiveness or otherwise of the risk reduction processes applying to the acquisition of major capital equipments to ensure their suitability and fitness for service.

Deficiences in Defence capability development and acquisition processes have resulted in under-utilised flight simulators and submarines unable to achieve anything like their planned level of operational capability. We offer as case studies in minature, three major projects where T&E and risk reduction processes - or their absence - have had pivotal roles.

Back in February 1999 Cubic Defence Systems, the preferred tenderer for Project Air 5395, Air Combat Training System, was selected to undertake a risk reduction study for the supply and management of the ACTS in a project that has now been cancelled due to its cost. Because the project involved significant system integration and software development, the risk reduction study was introduced to ensure that both Defence and Cubic Defence Systems had a clear understanding of the task before any contract for the supply of ACTS was entered into.

The ACTS was intended to provide aircrews with more realistic combat training environments and a variety of simulated scenarios, including a wide range of potential air and ground-based threats. It would also provide a T&E capability for existing and new electronic warfare equipment.

Cubic Defence Systems had teamed with Georgia Tech Research Institute to handle the EW T&E capability. Interestingly Georgia Tech is reported to be one of only two institutions in the world providing tertiary studies in T&E, the other being the University of South Australia's Systems Engineering & Evaluation Centre (SEEC). However GeorgiaTech's eminence in the field of T&E was not enough to prevent it being replaced by EW specialist Amherst at the RAAF's behest, possibly over the perceived difficulty of tying up contractually an overseas research institution.

And yet despite the risk reduction study, the careful switch to an assured EW T&E supplier, this project foundered, apparently due a major gap between what Defence was prepared to pay and what Cubic was prepared to accept.

Following their delivery, the RAAF's new C-130J-30 transport aircraft had to go through an extensive three month Operational Test & Evaluation phase before they entered operational service. The extensive evaluation involved flight and operational testing in various roles, checking supporting infrastructure, servicing and training processes, as well as developing procedural manuals. The C-130J represents a quantum leap in technology over its predecessors with Head Up Display (HUD) and glass cockpit, new computer-controlled engines and six-bladed composite propellers, a fully integrated digital avionics system and a computer-based maintenance support system.

Due to the complexity of its flight management and other systems, separate evaluation of the J-model's cockpit workload may lead to a three man flight deck crew being the norm in RAAF service instead of the two man crew the glass cockpit had promised!

In mid-1994 the government approved the purchase of two ex-USN Newport class amphibious landing ships for the ADF. HMAS Manoora and HMAS Kanimbla arrived in Australia during late-1994 and from mid-1996 underwent extensive modification and change into Amphibious Transport ships (known as LPAs) at Forgacs Dockyard in Newcastle.

However, from when the ships arrived in Australia and commenced their modification there was a significant increase to the scope of work anticipated at the time of purchase which contributed to a substantial project cost increase and subsequent schedule delay. The initial approved project cost for the two ships was $125m. When completed the final project cost amounted to some $400m, an unsatisfactory cost over-run brought about by a combination of management shortfalls and the unplanned, high risk features of this acquisition.

The damning Inspector-General's report revealed that the LPA Project was an 'opportunity buy' that reflects many of the high-risk features which typify this type of acquisition. At the time of procurement of the vessels there were:

* no defined operational requirement

* no detailed inspection of the vessels undertaken

* no life cycle costing carried out

* nor was any comprehensive risk analysis undertaken

Effective control and authority over the project was not exercised and despite these features and the overall importance attached to the LPA capability, senior level oversight of project outcomes was either limited to functional areas of responsibility, organisationally based, or absent. In any event there were no higher-level management arrangements that enabled effective control and authority to be exercised. The management of risk played little or no part in the selection and the subsequent modification of these two ships.

By Tom Muir, Canberra