Bulletin 131 - 2004 Dec 23

1. HPCCC - Global File Systems and the SYNC option
2. Holidays
3. HPCCC - change of restart areas for Bureau SX-6 nodes
4. Running jobs with varying demands
5. HPCCC SX-6 diagnostics options
6. Updates to netCDF
7. CSIRO HPSC - upgrades to the cluster - burnet
8. CSIRO HPSC - new versions of the Intel compilers
9. Providing data from the CSIRO Data Store to the WWW
10. cherax - limits
11. CSIRO - compiler stacksize on clusters
12. CSIRO - Mathematica training
13. HPSC Happenings
14. Seasons Greetings

The SYNC option was introduced onto all SX-6 nodes and GFS file systems by 23rd December.

Further guidance on the setting of buffers and options to make best use of GFS can be found at http://www.hpccc.gov.au - go to User Documentation, and then to the item "User Guide - Setting Fortran default buffer values", or straight to http://www.hpccc.gov.au/hpccc/userguides/faq/fortran_default_buffer_values.php

Users should be aware that with GFS, it is more important than ever to ensure you are not running multiple jobs using the same file systems, that might interfere with each other. For example you may have an SX-6 job running, and decide to housekeep on the TX7. The housekeeping could impact the SX-6 job through renaming or deletion of files or directories, or changes in permissions. Since the system linkage that holds GFS together is based on NFS, the system does not always offer all of the locking protection you might assume for a local file system.

There was a TX7 fail-over from about 09:00 on the morning of Tuesday 21st December. Because the GFS SYNC option was in operation for most file systems, there was no need to kill SX-6 and TX7 jobs, except that the spool area used for the standard output and standard error files from CSIRO jobs was not using GFS SYNC, and so CSIRO users are advised to check the output of jobs running at the above time.

The file systems were redistributed back to both servers on the morning of Thursday 23rd December.

The days 25th December to 3rd January 2005 are not scheduled working days for the HPCCC and the CSIRO HPSC team members, and only limited support will be available, except for operational processing.

With the assistance of NEC, we plan to change the restart areas on the Bureau SX-6 nodes to be common among the existing Bureau nodes.

This will allow jobs to migrate between nodes when higher priority work arrives.

However, we will wait until the no-migrate flag becomes available in NQS II, so that jobs using local disc can be shielded from migration. This is expected by the end of February 2005.

A good strategy is to build applications with a single flag enabling the application to switch between using local disc and GFS. Users can then easily choose between local disc for speed, or migration for throughput.

On clustered systems such as the NEC SX-6 and commodity systems, there can be problems in getting high utilisation when jobs use markedly varying numbers of processors during a run. For example, suppose a job uses 7 processors on an SX-6 node, and the scheduler effectively allocates that job to a whole node. If the job then drops back to using only one processor for a substantial period, what is the scheduler to do? If it does nothing, then there will be considerable wasted time on the node. However, if it allocates other work to the node, there is the potential for interference and poor performance if the job later wants to use 7 processors again.

This problem is particularly acute with tightly coupled multi-node jobs, e.g. using MPI.

If you have a job with such widely varying resource requirements during the run (sometimes called a 'lumpy' job), then please consider breaking the job up into sub-jobs, each using a fairly constant number of processors, so that the scheduling can be more effective. Jobs with periods of relatively low CPU usage are being targeted by HPCCC staff, to see whether there can be i/o or script improvements to expedite such jobs. In recent times, some jobs have had their throughput improved by around 25% by eliminating such periods of relatively low CPU usage.

Unfortunately, there is no facility in currently used batch systems at the HPCCC to allow the job to signal to the scheduler that processor allocation should be modified during a run. The NQS II system allows the specification

 #PBS -l cpunum_prc=7

for SX-6 jobs to specify the number of CPUs per process (used by the Gang Scheduler to schedule CPUs at execution time, and used by ERS to estimate the CPU usage of a job).

NEC has undertaken to provide the specification

 #PBS -l cpunum_job=7 

which will allow the number of processors expected to be used by a job to be given, allowing the scheduler to cope better with cases like ensemble jobs, with multiple processes being backgrounded within a job.

At present, processor usage is the main concern, but we expect memory usage will grow to be a significant factor, as may the use of local disc.

Users are requested to note that the inclusion of certain diagnostic options in SX-6 scripts have very high overheads and will adversely effect your throughput.

F_FILEINF is one of the most obvious high overhead options, and sometimes most useful. Because of its overhead F_FILEINF should only be enabled for a single run when needed, and should not be left enabled as it will substantially degrade your I/O performance because of overheads added. F_FILEINF instruments your I/O, as well as uses small block I/O, to record information.

On the other extreme F_PROGINF adds no overhead as it is a sampling of hardware counters, and can be left enabled permanently.

If in doubt on the overheads added from various SX-6 options, please ask for guidance through the HPCCC.

6. Updates to netCDF

Non-beta 3.5.1 netCDF libraries for SX-6 execution have been installed on the SX-6 nodes and all the cross platforms. To link the new libraries on the cross platforms, include the flags
-L/SX/local/netcdf/3.5.1/ew/lib , and
-I/SX/local/netcdf/3.5.1/ew/include .

After a period of testing, we intend to make this or a later release the default version early in 2005.

For SX software locations (e.g. netCDF), for both the SX and SX cross platforms, refer to the HPCCC website, http://www.hpccc.gov.au/ navigate to User Documentation and then to the item "NEC SuperUX and Cross Environment default software locations".

As previously noted, if you use netCDF, please try using a statement like

 NC_BLOCKSIZE=33554432 ; export NC_BLOCKSIZE # (ksh)
 setenv NC_BLOCKSIZE 33554432 # (csh)

to make considerable gains in performance in many instances (the actual best values could be determined by experiments).

netCDF version 3.6.0 has been released by Unidata. See http://my.unidata.ucar.edu/content/software/netcdf/announce-3.6.html for the Release notes.

For cherax users, there are now two versions available to try. These are installed under /usr/local/pkgs/netcdf-3.6.0/ in somewhat awkwardly named directories reflecting the compilers and options used to build the sub directories.

ifort81-icc81-i4r4d8 was compiled using ifort version 8.1, icc version 8.1 and Fortran data types -integer_size 32 -real_size 32 -double_size 64.

ifort81-icc81-i8r8d8 was compiled using the same compiler versions but Fortran data types -integer_size 64 -real_size 64 -double_size 64.

Each has subdirectories {bin,lib,include,man} which can be prepended to PATH, LD_LIBRARY_PATH, MANPATH, and -Icompiler option to select a particular version. Contact hpchelp@hpccc.gov.au with any comments or problems found. If there are no problems, we will make this the default version at the end of February 2005.

7. CSIRO HPSC - upgrades to the cluster - burnet

The compute nodes on burnet were recently upgraded. Nodes 1-17 now have 2GB of memory, and nodes 18-34 have 4GB of memory. This will allow the running of bigger jobs on half the nodes. (Node 14 is currently unavailable due to a delay in receiving its second CPU.)

The utility pkgenv has been set up on burnet, to allow the selection of software.

To use pkgenv, run:

 pkgenv packagename

ferret, grads, R, lam and mpich have been installed and set up to use pkgenv

From 13th December, half of burnet has been dedicated to an urgent project for one research group. If you have a need for such dedicated resources, please contact us, but not now!

8. CSIRO HPSC - new versions of the Intel compilers

New versions of the Intel compilers are available on cherax and burnet. The default has not changed (icc-8.0, ifort-8.0) - but the most recent version 8.1 shared libraries will now be used by default. Please be aware that this will change the shared libraries used at runtime. No impact is expected.

To set your environment to use the latest Intel compilers do:

  pkgenv ifort-8.1 icc-8.1

To switch back to the old versions. use

  pkgenv ifort-8.0 icc-8.0

9. Providing data from the CSIRO Data Store to the WWW

There has been a long-standing request for a facility for users to allow WWW access to data at the HPCCC.

A facility has been set up on the CSIRO Data Store (hosted on cherax) and the hpsc.csiro.au WWW pages to allow this to happen.

If you wish to have external web access to data stored on the CSIRO datastore, please contact the HPCCC and we will set up that facility and provide further instructions.

10. cherax - limits

From 10th January, we intend to impose some limits on interactive usage on cherax. These changes will be made to prevent substantial and run-away tasks from impacting other users.

Each session will be limited to 30 Gbyte of memory, and individual processes to 30 minutes CPU time.

The actual limit values may be revised in future.

11. CSIRO - compiler stacksize on clusters

Users of burnet and nelson have been experiencing some problems related to the default stacksize used by executables created by the ifort compiler. The symptom of the problem is a runtime segmentation violation. Updates were made to the pthread libraries to avoid some of the problems and default stacksize limits set for users - but there are additional measures that some users may need to completely avoid the problems. Different measures are needed for executables that use static and shared libraries.

For shared library executables (the default), the user must set a sufficiently large but finite stacksize limit in the shell

• eg. For bash/sh/ksh
  ulimit -s 100000
• for tcsh
  limit stacksize 100000

For statically linked executables (compiled with -static or -fast), it seems that the stacksize can be set to be unlimited but a special version of libpthread.a must be used (/usr/lib/nptl/libpthread.a rather than /usr/lib/libpthread.a). This requires the link option -L/usr/lib/nptl .

For more information, see the release notes for the compiler at http://intra.hpsc.csiro.au/userdocs/ifort_doc/FortranReleaseNotes.htm - go to the section headed "Segmentation Violation with static link to libpthreads". The section on OpenMP has further information.

12. CSIRO - Mathematica training

The availability of Mathematica was announced in HPCbull 130, item 5.

For CSIRO staff joining the HPSC Licensed Software Agreement, there is a single licence for Mathematica 5.0 (soon 5.1) now on cherax.

If you don't know what Mathematica can do, or how to use it, an introductory training course is being offered in Melbourne by Wolfram on 11-12 January 2005.

Seats are limited, and early registration was requested by 17 December. Late registrations will be accepted subject to availability. Course details and registration are on-line at: http://www.wolfram.com/weg/calendar Any questions, please contact John Brookes at john@analytica.com.au or +61-(0)4-0304-9453, or Ian Roberts at training@quantica.com.au or +61-(0)4-1219-3041.

13. CSIRO HPSC Happenings

CSIRO High Performance Scientific Computing issues a newsletter called "HPSC Happenings"

If you wish to receive a copy, please contact Erika Stojanovic, Erika.Stojanovic@csiro.au, or 03 9669 8113.

The latest copy was issued on 23rd December.

14. Seasons Greetings

                  .     .  .      +     .      .          .
                     .      .     #       .           .
               .       .         ###            .      .      .
              .      .   "#:. .:##"##:. .:#"  .      .
                  .      . "####"###"####"  .
               .     "#:.    .:#"###"#:.    .:#"  .        .       .
          .             "#########"#########"        .        .
                    .    "#:.  "####"###"####"  .:#"   .       .
             .     .  "#######""##"##""#######"                  .
                        ."##"#####"#####"##"           .      .
            .   "#:. ...  .:##"###"###"##:.  ... .:#"     .
              .     "#######"##"#####"##"#######"      .     .
            .    .     "#####""#######""#####"    .      .
                    .     "      000      "    .     .
               .         .   .   000     .        .       .
        .. .. ..................O000O........................ ...... ...