Lead Machine Pink 160x1200

Lead Machine Pink 160x1200

iTWire TV 705x108

Tuesday, 20 September 2022 09:28

Check the best options for scaling migration performance

By Tosin Vaithilingam, senior solution architect, BitTitan
BitTitan senior solution architect Tosin Vaithilingam BitTitan senior solution architect Tosin Vaithilingam

GUEST OPINION: In the epic spaghetti western The Good, The Bad and The Ugly, Tuco (The Ugly) says, "If you work for a living, why do you kill yourself working?"

This quote can be applied to many tasks in IT, and it can also sum up aptly many of the variables that IT professionals encounter when planning or performing a data migration.

When trying to calculate the timeframe of a migration, it is important to be aware of all the factors that might affect the migration's performance. However, while it is a worthwhile endeavour to calculate the length of time a cut-over will take, it is crucial to understand all the factors that are working for and against progress.

The Good

Having an amazing network with terabytes of throughput is certainly a checkbox in favour, but comparing bandwidth to migration performance isn't always so. Network speed tests generate random bytes in memory and check how quickly the data can be transferred.

A speed test does not convert data, encrypt data, parse data, index data, read data from disk, write data to disk, or authenticate users. Migration time and speed will depend on an organisation's network speed, server load, throttling and number of folders.

These are some average speeds that IT service providers have reported:

Low end: 250MB per hr/per user
Mid end: 750MB per hr/per user
High end: 1.25GB per hr/per user

A tier 1 migration app will open only one connection per mailbox. This allows IT to scale to a large number of concurrent mailboxes. In this way, hundreds of thousands of mailboxes can be migrated simultaneously.

Migrating more mailboxes at the same time allows parallel processing, reducing the duration of a migration. If all mailboxes are migrated at the same time, the migration duration is the time it takes to migrate the largest mailbox.

The Bad

There are certain practices that mailbox users do that make it more difficult to perform efficient migrations. The two worst examples are very large files sizes and mailboxes with many items. There is a per-item transaction overhead and time cost.

For example, in the per-item scenario, given two mailboxes of the same size but with a different number of items, the one with fewer items will migrate faster than the one with more items. And the larger the file size, the longer the migration. Four hundred users with 40MB of data will transfer far faster than one user with 4G of data.

For these larger users, we recommend breaking the data into smaller files to speed up the migration. Essentially: more smaller users are faster than a few larger users. When migrating between two hosted providers, the size of the mailbox is unlikely to overload our bandwidth, but some destinations might have capacity issues. When mailboxes exceed a certain size, speed reduction may occur because of those scalability issues.

The Ugly

Even accounting for all the known variables, both good and bad, there are still aspects of the migration that are unknown and out of our control.

Both Microsoft and Google have policies to throttle requests if they are hitting with too much frequency. Even if IT is being very careful not to send too many requests, there are other factors that come into play.

It is important to keep in mind that during a migration an organisation is using a shared network. In any given instance, there could be another migration or equally heavy traffic that causes load balancing or other network protocols that affect traffic. These factors are moving targets and are almost impossible to prepare for.

The Solution

For all migrations, if IT has scoped out the project and know there are some very large files or large number of users, the best option is to perform a few test runs or a proof of concept with a broad data set.

The type of data files should range from large to small. Then, increase concurrence until failures appear. At this point, back off the concurrence to a manageable threshold and give yourself a 50% buffer.

This manner of proof of concept is the best way to hit the moving targets of migration performance.

Read 489 times

Please join our community here and become a VIP.

Subscribe to ITWIRE UPDATE Newsletter here
JOIN our iTWireTV our YouTube Community here


Thoughtworks presents XConf Australia, back in-person in three cities, bringing together people who care deeply about software and its impact on the world.

In its fifth year, XConf is our annual technology event created by technologists for technologists.

Participate in a robust agenda of talks as local thought leaders and Thoughtworks technologists share first-hand experiences and exchange new ways to empower teams, deliver quality software and drive innovation for responsible tech.

Explore how at Thoughtworks, we are making tech better, together.

Tickets are now available and all proceeds will be donated to Indigitek, a not-for-profit organisation that aims to create technology employment pathways for First Nations Peoples.

Click the button below to register and get your ticket for the Melbourne, Sydney or Brisbane event



It's all about Webinars.

Marketing budgets are now focused on Webinars combined with Lead Generation.

If you wish to promote a Webinar we recommend at least a 3 to 4 week campaign prior to your event.

The iTWire campaign will include extensive adverts on our News Site itwire.com and prominent Newsletter promotion https://itwire.com/itwire-update.html and Promotional News & Editorial. Plus a video interview of the key speaker on iTWire TV https://www.youtube.com/c/iTWireTV/videos which will be used in Promotional Posts on the iTWire Home Page.

Now we are coming out of Lockdown iTWire will be focussed to assisting with your webinars and campaigns and assistance via part payments and extended terms, a Webinar Business Booster Pack and other supportive programs. We can also create your adverts and written content plus coordinate your video interview.

We look forward to discussing your campaign goals with you. Please click the button below.


Share News tips for the iTWire Journalists? Your tip will be anonymous




Guest Opinion

Guest Interviews

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News