[Lustre-discuss] Poor write performance with good read performance
Hayes, Robert N
robert.n.hayes at intel.com
Mon May 11 13:42:35 PDT 2009
Adding subject
/bob
-----Original Message-----
From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of lustre-discuss-request at lists.lustre.org
Sent: Monday, May 11, 2009 1:39 PM
To: lustre-discuss at lists.lustre.org
Subject: Lustre-discuss Digest, Vol 40, Issue 9
Send Lustre-discuss mailing list submissions to
lustre-discuss at lists.lustre.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.lustre.org/mailman/listinfo/lustre-discuss
or, via email, send a message with subject or body 'help' to
lustre-discuss-request at lists.lustre.org
You can reach the person managing the list at
lustre-discuss-owner at lists.lustre.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Lustre-discuss digest..."
Today's Topics:
1. Re: [ROMIO Req #940] a new Lustre ADIO driver] (Rajeev Thakur)
2. Re: [ROMIO Req #940] a new Lustre ADIO driver] (Weikuan Yu)
3. Re: [ROMIO Req #940] a new Lustre ADIO driver] (Weikuan Yu)
4. (no subject) (Hayes, Robert N)
5. Re: no handle for file close (Nirmal Seenu)
----------------------------------------------------------------------
Message: 1
Date: Sun, 10 May 2009 22:56:43 -0500
From: "Rajeev Thakur" <thakur at mcs.anl.gov>
Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre ADIO
driver]
To: "'Weikuan Yu'" <wkyu at auburn.edu>, "'Rob Latham'"
<robl at mcs.anl.gov>
Cc: 'lustre-discuss' <lustre-discuss at lists.lustre.org>, 'David Knaak'
<knaak at cray.com>, "'Tom.Wang'" <Tom.Wang at Sun.COM>,
Martin.Audet at imi.cnrc-nrc.gc.ca, "'emoly.liu'" <Emoly.Liu at Sun.COM>,
romio-maint at mcs.anl.gov
Message-ID: <BB76BB81E65844E18589AD012266D923 at thakurlaptop>
Content-Type: text/plain; charset="us-ascii"
Weikuan,
I can't read the attachment you sent. It has all kinds of strange
characters.
Rajeev
> -----Original Message-----
> From: Weikuan Yu [mailto:wkyu at auburn.edu]
> Sent: Saturday, May 09, 2009 3:54 PM
> To: Rob Latham
> Cc: emoly.liu; Wei-keng Liao; romio-maint at mcs.anl.gov;
> Tom.Wang; David Knaak; lustre-discuss; Martin.Audet at imi.cnrc-nrc.gc.ca
> Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre
> ADIO driver]
>
> Yes, it passed compilation. But there are many errors
> reported from runtest, quite a number of them are only from
> ad_lustre driver. Attached is an output tarball (named as
> .txt though). It contains the output files from running
> romio/runtests with ad_lustre and ad_ufs drivers separately.
>
> --
> Weikuan Yu, +1 (334) 844-6330
> http://www.eng.auburn.edu/~wkyu/
>
>
> > From: Rob Latham <robl at mcs.anl.gov>
> > Date: Sat, 9 May 2009 11:18:47 -0500
> > To: Weikuan Yu <wkyu at auburn.edu>
> > Cc: "emoly.liu" <Emoly.Liu at Sun.COM>, Wei-keng Liao
> > <wkliao at ece.northwestern.edu>, <romio-maint at mcs.anl.gov>, "Tom.Wang"
> > <Tom.Wang at Sun.COM>, David Knaak <knaak at cray.com>, lustre-discuss
> > <lustre-discuss at lists.lustre.org>, <Martin.Audet at imi.cnrc-nrc.gc.ca>
> > Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre
> ADIO driver]
> >
> > On Fri, May 08, 2009 at 09:33:20PM -0500, Weikuan Yu wrote:
> >> Hi, RobL,
> >>
> >> I tried the compilation. Looks like the SVN code has some
> inconsistencies in
> >> variable names.
> >
> > Thanks. I forgot to check one change in. Do try now: revision 4480
> > should fix this.
> >
> > ==rob
> >
> > --
> > Rob Latham
> > Mathematics and Computer Science Division
> > Argonne National Lab, IL USA
>
>
------------------------------
Message: 2
Date: Mon, 11 May 2009 06:52:15 -0500
From: Weikuan Yu <wkyu at auburn.edu>
Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre ADIO
driver]
To: Rajeev Thakur <thakur at mcs.anl.gov>, "'Rob Latham'"
<robl at mcs.anl.gov>
Cc: 'lustre-discuss' <lustre-discuss at lists.lustre.org>, 'David Knaak'
<knaak at cray.com>, "'Tom.Wang'" <Tom.Wang at Sun.COM>,
Martin.Audet at imi.cnrc-nrc.gc.ca, "'emoly.liu'" <Emoly.Liu at Sun.COM>,
romio-maint at mcs.anl.gov
Message-ID: <C62D7B9F.4216%wkyu at auburn.edu>
Content-Type: text/plain; charset="US-ASCII"
Hi, Rajeev,
Yeah, that file was a tarball named as .txt. It needs to be untar'ed to get
two output files. coll_test.c, noncontig_coll.c, noncontig_coll2.c and
others are failing.
# tar zxf runtests.output.txt
--
Weikuan Yu, +1 (334) 844-6330
http://www.eng.auburn.edu/~wkyu/
> From: Rajeev Thakur <thakur at mcs.anl.gov>
> Date: Sun, 10 May 2009 22:56:43 -0500
> To: Weikuan Yu <wkyu at auburn.edu>, 'Rob Latham' <robl at mcs.anl.gov>
> Cc: "'emoly.liu'" <Emoly.Liu at Sun.COM>, 'Wei-keng Liao'
> <wkliao at ece.northwestern.edu>, <romio-maint at mcs.anl.gov>, "'Tom.Wang'"
> <Tom.Wang at Sun.COM>, 'David Knaak' <knaak at cray.com>, 'lustre-discuss'
> <lustre-discuss at lists.lustre.org>, <Martin.Audet at imi.cnrc-nrc.gc.ca>
> Subject: RE: [Lustre-discuss] [ROMIO Req #940] a new Lustre ADIO driver]
>
> Weikuan,
> I can't read the attachment you sent. It has all kinds of strange
> characters.
>
> Rajeev
>
>
>> -----Original Message-----
>> From: Weikuan Yu [mailto:wkyu at auburn.edu]
>> Sent: Saturday, May 09, 2009 3:54 PM
>> To: Rob Latham
>> Cc: emoly.liu; Wei-keng Liao; romio-maint at mcs.anl.gov;
>> Tom.Wang; David Knaak; lustre-discuss; Martin.Audet at imi.cnrc-nrc.gc.ca
>> Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre
>> ADIO driver]
>>
>> Yes, it passed compilation. But there are many errors
>> reported from runtest, quite a number of them are only from
>> ad_lustre driver. Attached is an output tarball (named as
>> .txt though). It contains the output files from running
>> romio/runtests with ad_lustre and ad_ufs drivers separately.
>>
>> --
>> Weikuan Yu, +1 (334) 844-6330
>> http://www.eng.auburn.edu/~wkyu/
>>
>>
>>> From: Rob Latham <robl at mcs.anl.gov>
>>> Date: Sat, 9 May 2009 11:18:47 -0500
>>> To: Weikuan Yu <wkyu at auburn.edu>
>>> Cc: "emoly.liu" <Emoly.Liu at Sun.COM>, Wei-keng Liao
>>> <wkliao at ece.northwestern.edu>, <romio-maint at mcs.anl.gov>, "Tom.Wang"
>>> <Tom.Wang at Sun.COM>, David Knaak <knaak at cray.com>, lustre-discuss
>>> <lustre-discuss at lists.lustre.org>, <Martin.Audet at imi.cnrc-nrc.gc.ca>
>>> Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre
>> ADIO driver]
>>>
>>> On Fri, May 08, 2009 at 09:33:20PM -0500, Weikuan Yu wrote:
>>>> Hi, RobL,
>>>>
>>>> I tried the compilation. Looks like the SVN code has some
>> inconsistencies in
>>>> variable names.
>>>
>>> Thanks. I forgot to check one change in. Do try now: revision 4480
>>> should fix this.
>>>
>>> ==rob
>>>
>>> --
>>> Rob Latham
>>> Mathematics and Computer Science Division
>>> Argonne National Lab, IL USA
>>
>>
>
>
------------------------------
Message: 3
Date: Mon, 11 May 2009 09:39:20 -0500
From: Weikuan Yu <wkyu at auburn.edu>
Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre ADIO
driver]
To: Rob Latham <robl at mcs.anl.gov>
Cc: lustre-discuss <lustre-discuss at lists.lustre.org>, David Knaak
<knaak at cray.com>, "Tom.Wang" <Tom.Wang at Sun.COM>,
Martin.Audet at imi.cnrc-nrc.gc.ca, "emoly.liu" <Emoly.Liu at Sun.COM>,
romio-maint at mcs.anl.gov
Message-ID: <C62DA2C8.423C%wkyu at auburn.edu>
Content-Type: text/plain; charset="US-ASCII"
> From: Rob Latham <robl at mcs.anl.gov>
> Date: Mon, 11 May 2009 09:28:12 -0500
> To: Weikuan Yu <wkyu at auburn.edu>
> Cc: "emoly.liu" <Emoly.Liu at Sun.COM>, Wei-keng Liao
> <wkliao at ece.northwestern.edu>, <romio-maint at mcs.anl.gov>, "Tom.Wang"
> <Tom.Wang at Sun.COM>, David Knaak <knaak at cray.com>, lustre-discuss
> <lustre-discuss at lists.lustre.org>, <Martin.Audet at imi.cnrc-nrc.gc.ca>
> Subject: Re: [Lustre-discuss] [ROMIO Req #940] a new Lustre ADIO driver]
>
> So, the real challenges are coll_test, noncontig_coll, hindexed,
> aggregation1, aggregation2, split_coll... basically, collective I/O is
> messed up.
I agree. That's why I posted all error output, along with the ones from ufs.
--
Weikuan Yu, +1 (334) 844-6330
http://www.eng.auburn.edu/~wkyu/
------------------------------
Message: 4
Date: Mon, 11 May 2009 13:35:54 -0700
From: "Hayes, Robert N" <robert.n.hayes at intel.com>
Subject: [Lustre-discuss] (no subject)
To: "lustre-discuss at lists.lustre.org"
<lustre-discuss at lists.lustre.org>
Message-ID:
<13830B75AD5A2F42848F92269B11996F44318325 at orsmsx509.amr.corp.intel.com>
Content-Type: text/plain; charset="us-ascii"
While performing a single copy, single client write/read test using dd, we are finding that our Nehalem clients running
2.6.18-92.1.10.el5-lustre-1.6.5.1
write about half the speed of our Nehalem clients running
2.6.18-53.1.13.el5_lustre.1.6.4.3 to three different lustre file systems.
This is true even though the slower clients have the same processors and more RAM, 18GB for the slow writers and 12GB for the fast writers. Both systems use OFED 1.3.1. All benchmarks we use perform better on the slow-write clients and read speed from LFS is comparable across all clients.
Max_rpcs_in_flight and max_pages_per_rpc are default on both systems.
They are on the same IB network, with the same QDR cards and IB connectivity has been verified with the IB utilities. They are almost identical in bandwidth and latency.
We're also using the same modprobe.conf and openibd.conf files on both systems.
We're using 34GB file size on the 12GB and 18GB RAM systems, 137GB file on the 96GB RAM system. So it's not a matter of caching in RAM.
Are there known issues with our 2.6.18-92.1.10.el5-lustre-1.6.5.1 combination?
This is not a problem with the lustre file system as we get the same type of results no matter which of our three lustre systems the test is being written to.
Here are the summaries from several runs of ost-survey on our new Lustre system. Please comment on the worst/best deltas of the read and write operations.
Number of Active OST devices : 96
Worst Read 38.167753 38.932928 39.006537 39.782153 38.717915
Best Read 61.704534 61.832461 63.284999 65.000491 61.836016
Read Average: 51.433847 51.281630 51.297278 51.582327 51.318410
Worst Write 34.311237 49.009757 55.272744 51.532331 51.816523
Best Write 94.001170 96.033483 93.401792 93.081544 91.030717
Write Average: 74.248683 71.831019 75.179863 74.723100 74.930529
/bob
Bob Hayes
System Administrator
SSG-DRD-DP
Office: 253-371-3040
Cell: 253-441-5482
e-mail: robert.n.hayes at Intel.Com<mailto:robert.n.hayes at Intel.Com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20090511/e3469fd3/attachment-0001.html
------------------------------
Message: 5
Date: Mon, 11 May 2009 15:38:36 -0500
From: Nirmal Seenu <nirmal at fnal.gov>
Subject: Re: [Lustre-discuss] no handle for file close
To: lustre-discuss at lists.lustre.org
Message-ID: <4A088CCC.8010903 at fnal.gov>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
I got a couple more of these errors over the weekend. One of the file in
which the error occurred was a log(ascii) file while the other was a
dynamically linked MPI binary which was getting accessed from multiple
nodes.
The PBS job that was running was a hybrid MPI/OpenMP program using 20
nodes and 6 cores per node. The PBS job got killed when its walltime
exceeded. The user confirmed that there was no corruption in any of the
output files.
The following is the error message that I found in the log files:
May 10 14:59:37 lustre3 kernel: LustreError:
7566:0:(mds_open.c:1567:mds_close()) @@@ no handle for file close ino
2570697: cookie 0xcfe66441300e06ad req at ffff81041057a800 x2975034/t0
o35->30090fc1-eb50-ca1
5-b57a-41ea32f1d9db@:0/0 lens 296/1680 e 0 to 0 dl 1241985583 ref 1 fl
Interpret:/0/0 rc 0/0
May 10 14:59:37 lustre3 kernel: LustreError:
7566:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@ processing error
(-116) req at ffff81041057a800 x2975034/t0
o35->30090fc1-eb50-ca15-b57a-41ea32f1d9db@:0/0 len
s 296/1680 e 0 to 0 dl 1241985583 ref 1 fl Interpret:/0/0 rc -116/0
May 10 14:59:37 lustre3 kernel: LustreError:
7739:0:(mds_open.c:1567:mds_close()) @@@ no handle for file close ino
2558291: cookie 0xcfe66441300e07da req at ffff81040fa46c00 x2975035/t0
o35->30090fc1-eb50-ca1
5-b57a-41ea32f1d9db@:0/0 lens 296/1680 e 0 to 0 dl 1241985583 ref 1 fl
Interpret:/0/0 rc 0/0
May 10 14:59:37 lustre3 kernel: LustreError:
7739:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@ processing error
(-116) req at ffff81040fa46c00 x2975035/t0
o35->30090fc1-eb50-ca15-b57a-41ea32f1d9db@:0/0 len
s 296/1680 e 0 to 0 dl 1241985583 ref 1 fl Interpret:/0/0 rc -116/0
Nirmal
Robin Humble wrote:
> On Thu, May 07, 2009 at 10:45:31AM -0500, Nirmal Seenu wrote:
>> I am getting quite a few errors similar to the following error on the
>> MDS server which is running the latest 1.6.7.1 patched kernel. The
>> clients are running 1.6.7 patchless client on 2.6.18-128.1.6.el5 kernel
>> and this cluster has 130 nodes/Lustre clients and uses GigE network.
>>
>> May 7 04:13:48 lustre3 kernel: LustreError: 7213:0:(mds_open.c:1567:mds_close()) @@@ no handle for file close ino 772769: cookie 0xcfe66441310829d4 req at ffff8101ca8a3800 x2681218/t0 o35->fedc91f9-4de7-c789-6bdd-1de1f5e3dd33 at NET_0x20000c0a8f109_UUID:0/0 lens 296/1680 e 0 to 0 dl 1241687634 ref 1 fl Interpret:/0/0 rc 0/0
>>
>> May 7 04:13:48 lustre3 kernel: LustreError: 7213:0:(ldlm_lib.c:1643:target_send_reply_msg()) @@@ processing error (-116) req at ffff8101ca8a3800 x2681218/t0 o35->fedc91f9-4de7-c789-6bdd-1de1f5e3dd33 at NET_0x20000c0a8f109_UUID:0/0 lens 296/1680 e 0 to 0 dl 1241687634 ref 1 fl Interpret:/0/0 rc -116/0
>>
>> I don't see the same errors on another cluster/Lustre installation with
>> 2000 Lustre clients which uses Infiniband network.
>
> we see this sometimes when a job that is using a shared library that
> lives on Lustre is killed - presumably the un-memorymapping of the .so
> from a bunch of nodes at once confuses Lustre a bit.
>
> what is your inode 772769?
> eg.
> find -inum 772769 /some/lustre/fs/
> if the file is a .so then that would be similar to what we are seeing.
>
> so we have this listed in the "probably harmless" section of the errors
> that we get from Lustre, so if it's not harmless than we'd very much
> like to know about it :)
>
> this cluster is IB, rhel5, x86_64, 1.6.6 on servers and patchless
> 1.6.4.3 on clients w/ 2.6.23.17 kernels.
>
> cheers,
> robin
> --
> Dr Robin Humble, HPC Systems Analyst, NCI National Facility
>
>> I looked at the following bugs 19328, 18946, 18192 and 19085 but I am
>> not sure if any of those bugs apply to this error. I would appreciate it
>> someone could help me understand these errors and possibly suggest the
>> solution.
>>
>> TIA
>> Nirmal
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
------------------------------
_______________________________________________
Lustre-discuss mailing list
Lustre-discuss at lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
End of Lustre-discuss Digest, Vol 40, Issue 9
*********************************************
More information about the lustre-discuss
mailing list