Bulletin 157 - 2006 June 09
1. HPCCC Systems status - traffic lights
A new facility has been developed at the HPCCC and CSIRO HPSC WWW pages, showing the status of HPCCC systems as green and red 'lights'.
When systems are down, the notes section will provide links to further information, e.g. incident reports.
Your comments and feedback are welcomed.
[ page top ]
2. New NEC Parallelization Manual
The new format NEC SX-6 Fortran90/SX Shared Memory Parallelization Guide is now available on the HPCCC web site at http://www.hpccc.gov.au/hpccc/userdocs/index_user.shtml. Please note that a Distributed Memory equivalent will be available in early November 2006.
Feedback on the quality and content will be appreciated.
[ page top ]
3. CSIRO HPSC Commercial Software Access - Opportunity to use
For the last two years, CSIRO HPSC has provided users with access to a suite of commercial software, but has required a user contribution of $5000 per annum for this.
For the financial year 2006-7, CSIRO HPSC is dropping this charge for casual use of the software, but users and groups who make frequent or sustained use of the software will have to contribute to the cost of the software.
For the MSC software, each item requires tokens to be checked out, and there is a limit on the number of tokens that can be used at any time. Users who make sustained use will be expected to contribute to the cost of buying more tokens.
If this service is to be continued, it would help to be able to show broad commitment across the CSIRO HPSC user community. One clear way of demonstrating commitment is to contribute financially. Please contribute if you can, especially if you are billing customers for any of the related research.
The software covered by this scheme includes:
Some of the software can run only on the machines at the HPSC - the vector NEC SX-6, shared-memory SGI Altix, and the IBM Xeon cluster systems. For some of the software, the HPSC runs a licence server, and users can download the software and run it on their local machines.
Further information will be available at http://intra.hpsc.csiro.au/, or contact John Giovannis on 03 9669 8115, or email@example.com.
[ page top ]
4. HPCCC Seminars
Please see http://www.hpccc.gov.au/seminars/ for information about future seminars.
10.00 Wed 21 June 2006
[ page top ]
5. CSIRO Data Store enhancement - partial-state files
A new feature has been invoked under DMF on cherax for the CSIRO Data Store - partial-state files.
This feature allows DMF-managed files to have different residency states (online or offline) for different regions of a file. This means that a file can have one region that is online for immediate access and another region that is offline and will need to be recalled to online media in order to be accessed. DMF allows for up to 4 distinct file regions. A file which has more than one region is called a partial-state file. A region is simply a contiguous range of bytes which have the same residency state. The maximum number of 4 regions means that a file which is in a static state (not currently being migrated or unmigrated) can have a maximum of 2 online and a maximum of 2 offline regions.
A partial-state file is shown with the status 'PAR' in dmls -l listings, and can be matched with the dmfind command.
Currently, DMF is set to keep the first 32 kbyte of all new files in /cs/datastore on-line, although the threshold may vary.
This feature will allow commands such as "file" to access details of a file without recalling the entire file from off-line media. It may also allow access to metadata in things like netCDF files without recalling the whole file.
(This facility was recently used to allow a crawler to traverse an OPeNDAP repository without recalling the entire contents of all the files).
Please check any script that processes dmls output or uses dmfind commands, to ensure they continue to provide the correct behaviour for you. If you have any scripts that process the state of a file as to its off/online status, you should treat the partial file status (indicated by PAR by dmls) as an offline file.
Users can specify byte ranges for partial files in dmget and dmput commands - see the man pages. You can see the state of the regions of a file with the dmattr command: for example:
dmattr -a nregn,regn,state -l file
nregn : 2 regn : 0 DUL 0:40959 regn : 1 OFL 40960:EOF state : PAR
and dmattr -r -l file will show full details.
Please note that the local dmget wrapper does support byte ranges.
6. CSIRO Data Store - DMF status
We have installed a new command dmfstatus in /usr/local/bin/ on cherax.
This provides, in answer to long-standing requests from users, a clue as to the load on DMF.
DMF Status: Thursday 2006-06-08 16:06 Recalls Current Today VolGrp Queued Mb-Queued Total Mb-Total pri 5 54.3 2163 82925.9 sec 13 14950.9 12079 6410555.5 ter 0 0.0 11297 5532466.7 bu2 0 0.0 0 0.0 dcm 0 0.0 631 266714.0 Total 18 15005.1 26170 12292662.1 Migrates Current Today VolGrp Queued Mb-Queued Total Mb-Total pri 0 0.0 1931 7396.5 sec 0 0.0 2597 326242.1 ter 0 0.0 688 318916.7 bu2 0 0.0 3 37.5 dcm 1 1367.7 3472 650698.3 Total 1 1367.7 8691 1303291.0
This provides an instantaneous snapshot of the state of DMF recalls/migrates. The heading 'VolGrp' indicates volume groups - small files are written to pri and sec, other files to sec and ter: dcm is the disc cache.
We are also looking at enhancements to this facility, that will give a view of the status of your own recall requests.
Your comments on this facility are welcomed.
[ page top ]
© Copyright 2010, CSIRO Australia
Use of this web site and information available from it is subject to our Legal Notice and Disclaimer and Privacy Statement