[Lustre-discuss] programmatic access to parameters

[Cliff White]

burlen wrote:
> System limits are sometimes provided in a header, I wasn't sure if 
> Lustre adopted that approach. The llapi_* functions are great, I see how 
> to set the stripe count and size. I wasn't sure if there was also a 
> function to query about the configuration, eg number of OST's deployed?
> 
> This would be for use in a global hybrid megnetospheric simulation that 
> runs on a large scale (1E4-1E5 cores). The good striping parameters 
> depend on the run, and could be calculated at run time. It can make a 
> significant difference in our run times to have these set correctly. I 
> am not sure if we always want a stripe count of the maximum. I think 
> this depends on how many files we are synchronously writing, and the 
> number of available OST's total. Eg if there are 256 OST's on some 
> system and we have 2 files to write would it not make sense to set the 
> stripe count to 128?
> 
> We can't rely on our user to set the Lustre parameter correctly. We 
> can't rely on the system defaults either, they typically aren't set 
> optimally for our use case. MPI hints look promising but the ADIO Lustre 
> optimization are fairly new,  as far as I understand not publically 
> available in MPICH until next release (maybe in may?). We run on a 
> variety of systems some with variety of MPI implementation (eg Cray, 
> SGI). The MPI hints will only be useful on implementation that support 
> the particular hint. From a consistency point of view we need to both 
> make use of MPI hints and direct access via the llapi so that we run 
> well on all those systems, regardless of which MPI implementation is 
> deployed.  
> 

I don't know what your constraints are, but should note that this sort
of information (number of OSTs) can be obtained rather trivially from 
any lustre client via shell prompt, to wit:
# lctl dl |grep OST |wc -l
2
or:
# ls /proc/fs/lustre/osc | grep OST |wc -l
2

probably a few other ways to do that. Not as stylish as llapi_*..

cliffw

> Thanks
> Burlen
> 
> 
> Andreas Dilger wrote:
>> On 2010-03-23, at 14:25, burlen wrote:
>>> How can one programmatically probe the lustre system an application is
>>> running on?
>> Lustre-specific interfaces are generally "llapi_*" functions, from 
>> liblustreapi.
>>
>>> At compile time I'd like access to the various lustre system limits ,
>>> for example those listed in ch.32 of operations manual.
>> There are no llapi_* functions for this today.  Can you explain a bit 
>> better what you are trying to use this for?
>>
>> statfs(2) will tell you a number of limits, as will pathconf(3), and 
>> those are standard POSIX APIs.
>>
>>> Incidentally one I didn't see listed in that chapter is the maximum 
>>> number of OST's a single file can be striped across.
>> That is the first thing listed:
>>
>>>> 32.1    Maximum Stripe Count
>>>> The maximum number of stripe count is 160. This limit is hard-coded, 
>>>> but is near the upper limit imposed by the underlying ext3 file 
>>>> system. It may be increased in future releases. Under normal 
>>>> circumstances, the stripe count is not affected by ACLs.
>>>>
>>> At run time I'd like to be able to probe the size (number of OSS, OST
>>> etc...) of the system the application is running on.
>>
>> One shortcut is to specify "-1" for the stripe count will stripe a 
>> file across all available OSTs, which is what most applications want, 
>> if they are not being striped over only 1 or 2 OSTs.
>>
>> If you are using MPIIO, the Lustre ADIO layer can optimize these 
>> things for you, based on application hints.
>>
>> If you could elaborate on your needs, there may not be any need to 
>> make your application more Lustre-aware.
>>
>> Cheers, Andreas
>> -- 
>> Andreas Dilger
>> Sr. Staff Engineer, Lustre Group
>> Sun Microsystems of Canada, Inc.
>>
> 
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss