[Lustre-discuss] Lustre write performances 4 times faster after removing SD_IOSTATS from kernel

[Andreas Dilger]

On 2012-02-14, at 11:55 AM, Charland, Denis wrote:
> After performing some tests, I noticed that after removing SD_IOSTATS functionality from the kernel, the write performances of Lustre was 4 times faster. On a Lustre 1.8.7 client, I untared a tarfile of a directory containing about 225,000 files (6.4GB).

That is about 28kB/file, so it is causing a LOT of IO requests to the underlying disks.

> I used time to measure the time it takes to perform the operation. I performed the tests with two Lustre 1.8.7 servers with identical hardware. One is running Fedora 12 with a 2.6.32-19 patched kernel and the other one is running Fedora 7 with a 2.6.22-14 patched kernel.
> 
> With both servers, it took about 4 minutes to untar the tarfile when running the kernel without SD_IOSTATS compared to about 16 minutes when running the kernel with SD_IOSTATS.
>  
> Is that normal that SD_IOSTATS has a so big impact on Lustre write performances?

The sd_iostats code uses spin_lock_irqsave() to ensure consistent data structure access in interrupt context, so possibly this is causing the performance impact.  The impact might be more severe if there are many cores on the server.

The sd_iostats patch is completely optional, and is no longer in the RHEL6 kernel patch series in the Lustre 2.1/2.2 releases.

Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/