[Lustre-discuss] (no subject)

[Hayes, Robert N]

While performing a single copy, single client write/read test using dd, we are finding that our Nehalem clients running
2.6.18-92.1.10.el5-lustre-1.6.5.1
write about half the speed of our Nehalem clients running
2.6.18-53.1.13.el5_lustre.1.6.4.3 to three different lustre file systems.
This is true even though the slower clients have the same processors and more RAM, 18GB for the slow writers and 12GB for the fast writers. Both systems use OFED 1.3.1. All benchmarks we use perform better on the slow-write clients and read speed from LFS is comparable across all clients.
Max_rpcs_in_flight and max_pages_per_rpc are default on both systems.
They are on the same IB network, with the same QDR cards and IB connectivity has been verified with the IB utilities. They are almost identical in bandwidth and latency.

We're also using the same modprobe.conf and openibd.conf files on both systems.
We're using 34GB file size on the 12GB and 18GB RAM systems, 137GB file on the 96GB RAM system. So it's not a matter of caching in RAM.

Are there known issues with our 2.6.18-92.1.10.el5-lustre-1.6.5.1 combination?

This is not a problem with the lustre file system as we get the same type of results no matter which of our three lustre systems the test is being written to.

Here are the summaries from several runs of ost-survey on our new Lustre system. Please comment on the worst/best deltas of the read and write operations.
Number of Active OST devices : 96
Worst Read        38.167753   38.932928   39.006537   39.782153   38.717915
Best Read         61.704534   61.832461   63.284999   65.000491   61.836016
Read Average:     51.433847   51.281630   51.297278   51.582327   51.318410
Worst Write       34.311237   49.009757   55.272744   51.532331   51.816523
Best Write       94.001170   96.033483   93.401792   93.081544   91.030717
Write Average:    74.248683   71.831019   75.179863   74.723100   74.930529

/bob


Bob Hayes
System Administrator
SSG-DRD-DP
Office:  253-371-3040
Cell:     253-441-5482
e-mail: robert.n.hayes at Intel.Com<mailto:robert.n.hayes at Intel.Com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20090511/e3469fd3/attachment.htm>