Bulletin 157 - 2006 June 09

  1. HPCCC Systems status - traffic lights
  2. New NEC Parallelization Manual
  3. CSIRO HPSC Commercial Software Access - Opportunity to use
  4. HPCCC Seminars
  5. CSIRO Data Store enhancement - partial-state files
  6. CSIRO Data Store - DMF status

1. HPCCC Systems status - traffic lights

A new facility has been developed at the HPCCC and CSIRO HPSC WWW pages, showing the status of HPCCC systems as green and red 'lights'.

See http://www.hpccc.gov.au/hpccc/system_status/ and http://intra.hpsc.csiro.au/

When systems are down, the notes section will provide links to further information, e.g. incident reports.

Your comments and feedback are welcomed.

[ page top ]


2. New NEC Parallelization Manual

The new format NEC SX-6 Fortran90/SX Shared Memory Parallelization Guide is now available on the HPCCC web site at http://www.hpccc.gov.au/hpccc/userdocs/index_user.shtml. Please note that a Distributed Memory equivalent will be available in early November 2006.

Feedback on the quality and content will be appreciated.

[ page top ]


3. CSIRO HPSC Commercial Software Access - Opportunity to use

For the last two years, CSIRO HPSC has provided users with access to a suite of commercial software, but has required a user contribution of $5000 per annum for this.

For the financial year 2006-7, CSIRO HPSC is dropping this charge for casual use of the software, but users and groups who make frequent or sustained use of the software will have to contribute to the cost of the software.

For the MSC software, each item requires tokens to be checked out, and there is a limit on the number of tokens that can be used at any time. Users who make sustained use will be expected to contribute to the cost of buying more tokens.

If this service is to be continued, it would help to be able to show broad commitment across the CSIRO HPSC user community. One clear way of demonstrating commitment is to contribute financially. Please contribute if you can, especially if you are billing customers for any of the related research.

The software covered by this scheme includes:

MSC Master Key Software
Nastran Patran
AFEA Mark
Mentat Adams
Dytran Superform 2D
Explore GS-Mesher
SDM Easy5

Other Software
NAG Fortran subroutine Library
NAG Fortran90 Library
Mathematica
S-Plus
GaussviewW
Gaussian 2003
Ensight
Matlab
Intel Fortran compiler for Linux and Windows
Intel C compiler for Linux and Windows

Some of the software can run only on the machines at the HPSC - the vector NEC SX-6, shared-memory SGI Altix, and the IBM Xeon cluster systems. For some of the software, the HPSC runs a licence server, and users can download the software and run it on their local machines.

Further information will be available at http://intra.hpsc.csiro.au/, or contact John Giovannis on 03 9669 8115, or hpchelp@csiro.au.

[ page top ]


4. HPCCC Seminars

Please see http://www.hpccc.gov.au/seminars/ for information about future seminars.

10.00 Wed 21 June 2006
"Some points for efficient usage of the SX-6 system"
BMRC Lecture Room, 9E
Ilia Bermous

[ page top ]


5. CSIRO Data Store enhancement - partial-state files

A new feature has been invoked under DMF on cherax for the CSIRO Data Store - partial-state files.

This feature allows DMF-managed files to have different residency states (online or offline) for different regions of a file. This means that a file can have one region that is online for immediate access and another region that is offline and will need to be recalled to online media in order to be accessed. DMF allows for up to 4 distinct file regions. A file which has more than one region is called a partial-state file. A region is simply a contiguous range of bytes which have the same residency state. The maximum number of 4 regions means that a file which is in a static state (not currently being migrated or unmigrated) can have a maximum of 2 online and a maximum of 2 offline regions.

A partial-state file is shown with the status 'PAR' in dmls -l listings, and can be matched with the dmfind command.

Currently, DMF is set to keep the first 32 kbyte of all new files in /cs/datastore on-line, although the threshold may vary.

This feature will allow commands such as "file" to access details of a file without recalling the entire file from off-line media. It may also allow access to metadata in things like netCDF files without recalling the whole file.

(This facility was recently used to allow a crawler to traverse an OPeNDAP repository without recalling the entire contents of all the files).

Please check any script that processes dmls output or uses dmfind commands, to ensure they continue to provide the correct behaviour for you. If you have any scripts that process the state of a file as to its off/online status, you should treat the partial file status (indicated by PAR by dmls) as an offline file.

Users can specify byte ranges for partial files in dmget and dmput commands - see the man pages. You can see the state of the regions of a file with the dmattr command: for example:

 dmattr -a nregn,regn,state -l file

might show:

     nregn : 2
      regn : 0  DUL  0:40959
      regn : 1  OFL  40960:EOF
     state : PAR

and dmattr -r -l file will show full details.

Please note that the local dmget wrapper does support byte ranges.


6. CSIRO Data Store - DMF status

We have installed a new command dmfstatus in /usr/local/bin/ on cherax.

This provides, in answer to long-standing requests from users, a clue as to the load on DMF.

For example:

DMF Status: Thursday 2006-06-08 16:06

                   Recalls
          Current           Today
VolGrp Queued Mb-Queued  Total  Mb-Total
pri         5      54.3   2163   82925.9
sec        13   14950.9  12079 6410555.5
ter         0       0.0  11297 5532466.7
bu2         0       0.0      0       0.0
dcm         0       0.0    631  266714.0
Total      18   15005.1  26170 12292662.1

                  Migrates
          Current           Today
VolGrp Queued Mb-Queued  Total  Mb-Total
pri         0       0.0   1931    7396.5
sec         0       0.0   2597  326242.1
ter         0       0.0    688  318916.7
bu2         0       0.0      3      37.5
dcm         1    1367.7   3472  650698.3
Total       1    1367.7   8691 1303291.0

This provides an instantaneous snapshot of the state of DMF recalls/migrates. The heading 'VolGrp' indicates volume groups - small files are written to pri and sec, other files to sec and ter: dcm is the disc cache.

We are also looking at enhancements to this facility, that will give a view of the status of your own recall requests.

Your comments on this facility are welcomed.

[ page top ]



BoM Solar Help:

CSIRO ASC Help:

For urgent help at all times:
  • CSIRO users 0428 108 333
  • Bureau out of hours emergencies are managed through internal policy
HPCCC WWW Site: http://www.hpccc.gov.au/
CSIRO External ASC Site: http://www.hpsc.csiro.au/
CSIRO ASC Users' Site: http://intra.hpsc.csiro.au/

Comments to:


© Copyright 2010, CSIRO Australia
Use of this web site and information available from it is subject to our Legal Notice and Disclaimer and Privacy Statement