Defence Business: How Defence migrated a Petabyte of information to its new Data Centres | ADM February 2013

A petabyte (PB) indicates the fifth power to 1,000, a million gigabytes or a thousand terabytes. That’s how much data was transferred by the weekend of November 17-18, 2012 to Defence’s two commissioned data centres in Sydney and Melbourne.

Some 200 business personnel were testing, testing, testing and checking, checking according to Defence’s Chief Technology Officer, Matt Yannopoulos.

“I had about 250 between Accenture and my own staff supporting the migration,” Yannopoulos recalls.

They had been engaged over a 68 week period across the majority of Defence Groups and Australian policing jurisdictions. On Thursday evening November 15, his team shut down all systems for Defence and all staff were alerted. This enabled a clear synchronisation point and back-up. The Audit Office was observing to ensure all the financial tables added up.

Thought to be the single largest data migration by Defence’s Accenture partner, the data and applications migrated to its new homes to “the delight of 136,000 satisfied customers,”  he added.

Effectively 153 application migrations across nine distinct computer platforms took the one way trip. These comprised 412 environment migrations, 3,263 server migrations with over 250 stakeholders. Completed by Sunday November 18 with Yannopoulos hosting a midday phone hook-up.

“We’re live.”

Human Resource needed more time to do their work without users coming in and putting in their leave applications. Amazingly, Yannopoulos slept well that weekend. But his dedicated team of 36 staff were on shifts throughout. He checked with his team during the critical milestones.

“I spent some 12 hours over that weekend ensuring we were working through the systems and talking with colleagues about the migration.”

As it was a Defence installation, some things were carried out earlier than planned. Defence logistics had an operation reliant on the systems in the Middle East area of operations at about 6pm that Saturday night.

“We drew that time forward. A large number of troops were on the move.”

Working with other systems Yannopoulos learnt of this a mere two weeks before the big move. Specifically, Military Integrated Logistic Information System (MILIS) which provided next generation of logistic support had to be accessible for the Middle East area of operations. It had to be sooner rather than he planned.

“If something went wrong that work could be wasted and had to be reapplied,” Yannopoulos said.

He puts the success of the move to sustained stakeholder engagement over more than a year. Come Monday, it was situation normal. More or less. Yannopoulos logged on to the HR system and found it was running slow.

“I asked my folks to have a look at it. The following morning I received advice that that was the second biggest day in history of access to our HR system.”

Shutting the system down for 24 hours was like Christmas shopping – close the shops for one day, and people buy five days’ worth of food. But the infrastructure held up and everything went fine, he said.

No fewer than four dress rehearsals were run before the real migration weekend. These occurred every six weeks prior to the weekend of 17th and 18th November. The first one was to test basic procedures. The second tuned those procedures. The third should be perfect. The fourth one was run to be sure. Murphy originally practiced his law with the Irish Army.

It was important to discover and refine the plan for all the interdependencies and the approval processes. By the time of the main event, everyone was familiar with how it was going to roll. Strictly speaking, at the time of writing, the migration was 96 per cent complete. CadetNet, Defence’s external websites (defence.gov.au) and its online services domain were to be completed by January 2013.

An immediate pay-off comes with disaster recovery times. In the past, it would have taken his team three days to bring a system up from total power and network loss, for example. A test over that last week disclosed Defence can now shorten that to just one day or 24 hours.

“The new communications architecture has data feeding all the time now to our disaster recovery centre,” he said.

Yannopoulos has been with the project for some four years now. Joining Defence at the end of 2008, he planned the move throughout 2009. The Global Switch facility was leased in mid 2010 with fit out going from mid-2010 to most of 2011. The actual data migration phase was scheduled through 2012 with the usual delays and hic ups. He doubts whether it could have been done any faster.

“It took time to get the approvals resolved for funding and support from the Whole of Government data centre panel we were using. The actual migration has gone quickly.”

The entire budget for the move was $100 million for equipment and services to move. Accenture was prime partner over the period. At its peak, Accenture had about 150 staff employed to make it happen.

One benefit of moving to the new facility is improved disaster recovery of the environment. Global Switch is located in Ultimo – where the old printing press headquarters – a well-guarded facility with imposing physical security and multi-layers of reinforced security facilities.

The security question

Is there a potential vulnerability now that all Defence data has been consolidated into just two very visible centres in Sydney and Melbourne? Have we created our own twin tower data centres that an adversary could now target as opposed to former multiple diverse targets? Yannopoulos argues it was a fallacy to believe Defence was more secure having data operations in 120 separate locations.

In practice, any one of those could be switched off and all those people could not do any work. There were also single points of failure all through the environment. If someone could switch some items off, everybody would lose that service, anyway.

“To those that that would say that centralising our data puts us at a strategic disadvantage, I can better effect security for a range of threat scenarios  perhaps at the highest threat perspective, there may be something in that argument – but I have distributed the command and control environment,” Yannopoulos said.

He revealed Defence’s big enterprise resource planning systems SAP, MILIS – were always in one system because they run on very large specialised equipment.

“That data centre was more prone already. A vulnerability was always there.”

He adds that the multiplicity of data centres, and Defence’s ability to physically secure them and configure control was more difficult. He adds that with war fighting systems, there will still be eight to ten other data centres at non-disclosed locales that provide that support.

But what of natural catastrophes, such as earthquakes or tsunamis?

Yannopoulos believes a tsunami-like event that took out the Sydney centre, near the harbour, will still enable Defence to deploy its Melbourne data centre at Victoria Barracks for those applications.

“This has been modelled,” he said “We now understand which critical applications that chief of joint operations needs to fight. We have ensured that we can provide those in a resilient way.”

The life of a data centre is shorter than you think

But what is the life of these new data centres? Defence’s downloading and generating data has grown exponentially. In practice, the data centre will have a 10 year life-span, Yannopoulos said. He plans two refresh cycles every five years where most of the equipment will be replaced with new servers and new storage

“It runs at a higher utilisation than we have had in the past,” he said.

Failures of physical equipment occur and the “sweet spot” for refreshing and upgrading is between four to five years. For now, though Defence’s new data centres have the capacity – power and floor space to take its projected workload out to the decade.