JUser: :_load: Unable to load user with ID: 3667
Monday, 08 October 2012 12:02

CSIRO wrangles extra 5TB a day


On a typical day the CSIRO adds 5 terabytes of data to its advanced scientific computer centre’s storage systems; it’s now up to 2.5 petabytes of data stored in 23 million files. To cope with the growth – but also rein in costs – the research organisation has just installed an SGI MAID (Massive Array of Idle Disk) which has the capacity to hold 1.7 petabytes of data.

Accounting for just 260 working days a year, 5 terabytes a day quickly mounts up to 1.3 petabytes a year – meaning CSIRO’s data bank will double in less than two years. While it seems an enormous growth rate – it’s smack in the middle of Gartner’s forecast for all industry sectors which is 40-60 per cent data growth per year.

Dr Robert Bell, technical services manager for the CSIRO’s advanced scientific computer centre, says that at present the CSIRO’s storage systems have enough headroom to handle that level of growth. The SGI Copan MAID system was brought into production at the CSIRO with 870 Tbyte formatted capacity in September – with the ability to build that out to 1.7 Petabytes.

Dr Bell said the MAID layer of storage helps to keep running costs under control – but also provide much faster access to data than is possible using tape. (CSIRO has access to 30 Petabytes of tape storage should it need it).

Two decades ago CSIRO started using Cray’s Data Migration Facility – a system designed to allow data storage to be sensibly tiered. That technology was inherited by SGI when it bought Cray, and since been redeveloped, but remains a fixture at the CSIRO.

Dr Bell said that it allowed the organisation to store its data collections across four levels of storage – fast expensive disk; slower cheaper disk; the new MAID; and tape. Sitting in front of these four levels of storage was an SGI UV system with 512 cores and 4 terabytes of memory to perform the initial data analysis.

One of the clear benefits of the tiered approach was the reduced power costs according to Dr Bell.

He said that the storage collection, housed in its Melbourne Docklands facility, today consumed about $13,000 worth of electricity each year. Had the entire collection been housed on fast expensive disk the electricity bill alone would have blown out to $500,000 a year.

“Power costs are not going down and demand for computer power is going up,” said Dr Bell. While the CSIRO’s data storage challenges are particularly acute, there are lessons for corporate Australia he believes.

Dr Bell said that there were significant power savings for organisations able to sensibly cascade their storage requirements.

However he acknowledged that for critical corporate applications it might not be sensible to wait the 90 seconds or so it could take for data stored in a tape library to be made available to an application. Scientists could be a little more patient he acknowledged.

But the MAID could be a half-way house, as only a quarter of the disks in the chassis are powered at any one time, leading to lower energy costs, and access to data stored in the MAID could be made accessible within 15-20 seconds.

While the hardware and software is now available to sensibly manage four tiered storage systems, many organisations will still stumble over the policies which determine when data is moved off the fastest disk, and cascaded down to slower storage systems. Even CSIRO struggles with this issue according to Dr Bell.

“We can never predict the workload – science research varies from day to day,” he said, adding that the CSIRO was still expecting “new avalanches of data”.  But there was no real pattern which controlled when scientists might need access to that data and “Uncontrolled workload is a difficult thing to do,” he said, adding that this was one area where corporate might have more predictability.


You cannot afford to miss this Dell Webinar.

With Windows 7 support ending 14th January 2020, its time to start looking at your options.

This can have significant impacts on your organisation but also presents organisations with an opportunity to fundamentally rethink the way users work.

The Details

When: Thursday, September 26, 2019
Presenter: Dell Technologies
Location: Your Computer


QLD, VIC, NSW, ACT & TAS: 11:00 am
SA, NT: 10:30 am
WA: 9:00 am NZ: 1:00 pm

Register and find out all the details you need to know below.



iTWire can help you promote your company, services, and products.


Advertise on the iTWire News Site / Website

Advertise in the iTWire UPDATE / Newsletter

Promote your message via iTWire Sponsored Content/News

Guest Opinion for Home Page exposure

Contact Andrew on 0412 390 000 or email [email protected]




Recent Comments