NCI National Facility


Home	Notices and News	Accounts	Facilities, Software and Userguides	Frequently Asked Questions	Training	Annual Reports

Newsletter April 2005

End of Alphaserver SC service and start of new SGI Altix (AC) service.
The SC service will be replaced by a service on the new SGI Altix (ac.apac.edu.au) system very shortly. It is important that all users read this newsletter which outlines the planned changeover process from the SC to the AC and the major differences that users will encounter in using the SGI Altix system. Further details will be available in the AC section of the Userguide and updates mailed out as the changeover approaches.
It is not yet possible to login to the AC. Users will be emailed when this becomes possible.

Table of Contents

Transfer to AC, the SGI Altix Cluster
Files
Using Modules to replace USE_PACKAGE
AC compilers
Packaged software on the AC
AC queueing
AC Filesystems

Transfer to AC, the SGI Altix Cluster

The new SGI Altix cluster with 1664 processors is currently being installed in stages. We expect the production service on the AC to begin in mid-May on 640 processors. The exact date will be clearer once acceptance tests are complete and the NF environment is installed.
Regrettably because of machine room space limitations there will be a very short period during which a production service on both the old and new systems overlap.

We realize this short overlap may make it difficult for some users to maintain throughput of their work when the SC is turned off. To minimize this disruption, we intend to invite those users who need to compile their codes for the AC to have early access to a limited pre-production service. Further notice of this will be sent by email in the near future.

Access during the pre-production phase should not be required for package users. We hope to have commonly used packages available from the start of the AC production service so the disruption to package users should be minimal.

Time Allocations - Allocations to date have been made in SC-equivalent hours. Accounting on the AC will start when the SC production service ends. As foreshadowed, allocations will be adjusted for the AC in such a way that projects are not disadvantaged.

The SC will be unavailable on the morning of Saturday 7 May until around 1pm to allow work on power supplies in the machine room. In advance of this, jobs will not be started that could not complete before the start of the downtime. We apologise for this disruption to the service when the system is so heavily loaded.

Files Closer to the date for turning off the SC we will send out full details of the procedure for moving files but, before then, it would be helpful if users could please clean up their directories in /short and /home. In particular, remove any binaries that you do not need to use again on the SC as they will not run on the Altix and code will have to be recompiled.

Using Modules to replace USE_PACKAGE To cope with different versions of software packages and to simplify the setting up of environments for running facility provided software the AC will be using the Modules package. This provides for dynamic modification of the user's environment and will replace the practice of using setenv USE_PACKAGE required on the SC and LC.
You will need to amend batch jobscripts and .login, .cshrc or .profile files that use the USE_PACKAGE mechanism for setting paths to libraries or executables. Note that you will no longer need the line source /opt/etc/system_cshrc in your .cshrc or . /opt/etc/system_profile in your .profile.
Modules have also been installed on the LC and users are encouraged to start using them there. Further information is given here.
Example usage of modules includes:

module avail
This gives the list of all modules available to use on the AC. The list contains both version generic (our recommended default) and version specific modules for software packages, compilers and libraries.
module load gaussian/g03
This adds the default Gaussian g03 module so will adjust environment variables and add the Gaussin bin directory to the path so that g03version can be used.
module unload gaussian/g03-default
This will clear your environment of the environment variables required for g03 without you having to logoff and logon again.
module list
This will display all modules currently loaded in your environment.
module help
This will give a summary of the module commands.
All batch job scripts copied from the SC or LC which have a line setenv USE_PACKAGE will have to be modified to
module load package.
We encourage people to load a minimal number of packages through their login shells, and instead load the modules in the batch job scripts, or interactively. Example scripts for NF provided software will appear in the "How to Use" section of the software web pages.

AC compilers All code ported from the SC will also have to be recompiled with the Intel or GNU compilers.
Compilers available on the AC include the GNU compilers (gcc, g++ and g77) and the Intel compilers (icc, icpc and ifort for Intel 8.1). Further information on the compilers will be available in the AC section of the userguide and software web page in due course.
The Itanium processors on the AC are 64 bit so all code compiled on the LC (x86) with the Intel compilers will have to be recompiled. Note that the IA64 is not the same as the extended EM64T system available on some 32 bit processors (x86-64) so any code compiled using this option for 64 bit will have to be recompiled.

Packaged software on the AC Most packages will be made available on the AC. The software web pages have been expanded to include software to be installed on the AC and will be continually updated as software is made available. Several packages have already been ported, including chemistry packages Gaussian(G03), Gamess, and NWChem, and math library packages PetSC, LAPACK and ScaLAPACK, and netCDF. Others will be installed as soon as possible but if you have particular need for a package not yet available please contact help@nf.apac.edu.au.
However, there are a few packages that will not be made available since some packages do not have versions for the Itanium processor. For example Mathworks have not made Matlab available for Itanium. Matlab will continue to be available on the LC. One alternative for Matlab is Octave. An enhanced 64 bit version of Octave has been developed under the APAC Computational Tools and Techniques program. Please contact help@nf.apac.edu.au if you would like early access to this Octave version.
Some packages have access restrictions, for example due to license arrangements. These restrictions will be carried over to the AC system. However, if you have problems accessing a package for which you believe you have valid access please contact help@nf.apac.edu.au.

AC queueing The queues will be the same as on the SC with minor changes to some limits. The command nf_limits will be available to check user and project limits for memory, walltime and number of processors in each queue.

AC Filesystems Filesystems on the AC will be the same as on the SC, that is,

/home, quotas apply, backed up.
/short, larger allocation, use for reproducible executables and data, NOT backed up.
/jobfs, job scratch space, deleted at conclusion of batch job, must be explicitly requested at job submssion, make sure any essential output is copied back to /home or /short at completion of job.
/fast, used for large I/O.

Email problems, suggestions, questions to

Transfer to AC, the SGI Altix Cluster	The new SGI Altix cluster with 1664 processors is currently being installed in stages. We expect the production service on the AC to begin in mid-May on 640 processors. The exact date will be clearer once acceptance tests are complete and the NF environment is installed. Regrettably because of machine room space limitations there will be a very short period during which a production service on both the old and new systems overlap. We realize this short overlap may make it difficult for some users to maintain throughput of their work when the SC is turned off. To minimize this disruption, we intend to invite those users who need to compile their codes for the AC to have early access to a limited pre-production service. Further notice of this will be sent by email in the near future. Access during the pre-production phase should not be required for package users. We hope to have commonly used packages available from the start of the AC production service so the disruption to package users should be minimal. Time Allocations - Allocations to date have been made in SC-equivalent hours. Accounting on the AC will start when the SC production service ends. As foreshadowed, allocations will be adjusted for the AC in such a way that projects are not disadvantaged. The SC will be unavailable on the morning of Saturday 7 May until around 1pm to allow work on power supplies in the machine room. In advance of this, jobs will not be started that could not complete before the start of the downtime. We apologise for this disruption to the service when the system is so heavily loaded.

Files	Closer to the date for turning off the SC we will send out full details of the procedure for moving files but, before then, it would be helpful if users could please clean up their directories in /short and /home. In particular, remove any binaries that you do not need to use again on the SC as they will not run on the Altix and code will have to be recompiled.

Using Modules to replace USE_PACKAGE	To cope with different versions of software packages and to simplify the setting up of environments for running facility provided software the AC will be using the Modules package. This provides for dynamic modification of the user's environment and will replace the practice of using `setenv USE_PACKAGE` required on the SC and LC. You will need to amend batch jobscripts and .login, .cshrc or .profile files that use the USE_PACKAGE mechanism for setting paths to libraries or executables. Note that you will no longer need the line `source /opt/etc/system_cshrc` in your .cshrc or `. /opt/etc/system_profile` in your .profile. Modules have also been installed on the LC and users are encouraged to start using them there. Further information is given here. Example usage of modules includes: `module avail` This gives the list of all modules available to use on the AC. The list contains both version generic (our recommended default) and version specific modules for software packages, compilers and libraries. `module load gaussian/g03` This adds the default Gaussian g03 module so will adjust environment variables and add the Gaussin bin directory to the path so that g03version can be used. `module unload gaussian/g03-default` This will clear your environment of the environment variables required for g03 without you having to logoff and logon again. `module list` This will display all modules currently loaded in your environment. `module help` This will give a summary of the module commands. All batch job scripts copied from the SC or LC which have a line `setenv USE_PACKAGE` will have to be modified to `module load package`. We encourage people to load a minimal number of packages through their login shells, and instead load the modules in the batch job scripts, or interactively. Example scripts for NF provided software will appear in the "How to Use" section of the software web pages.

AC compilers	All code ported from the SC will also have to be recompiled with the Intel or GNU compilers. Compilers available on the AC include the GNU compilers (`gcc`, `g++` and `g77`) and the Intel compilers (`icc`, `icpc` and `ifort` for Intel 8.1). Further information on the compilers will be available in the AC section of the userguide and software web page in due course. The Itanium processors on the AC are 64 bit so all code compiled on the LC (x86) with the Intel compilers will have to be recompiled. Note that the IA64 is not the same as the extended EM64T system available on some 32 bit processors (x86-64) so any code compiled using this option for 64 bit will have to be recompiled.

Packaged software on the AC	Most packages will be made available on the AC. The software web pages have been expanded to include software to be installed on the AC and will be continually updated as software is made available. Several packages have already been ported, including chemistry packages Gaussian(G03), Gamess, and NWChem, and math library packages PetSC, LAPACK and ScaLAPACK, and netCDF. Others will be installed as soon as possible but if you have particular need for a package not yet available please contact help@nf.apac.edu.au. However, there are a few packages that will not be made available since some packages do not have versions for the Itanium processor. For example Mathworks have not made Matlab available for Itanium. Matlab will continue to be available on the LC. One alternative for Matlab is Octave. An enhanced 64 bit version of Octave has been developed under the APAC Computational Tools and Techniques program. Please contact help@nf.apac.edu.au if you would like early access to this Octave version. Some packages have access restrictions, for example due to license arrangements. These restrictions will be carried over to the AC system. However, if you have problems accessing a package for which you believe you have valid access please contact help@nf.apac.edu.au.

AC queueing	The queues will be the same as on the SC with minor changes to some limits. The command `nf_limits` will be available to check user and project limits for memory, walltime and number of processors in each queue.

AC Filesystems	Filesystems on the AC will be the same as on the SC, that is, /home, quotas apply, backed up. /short, larger allocation, use for reproducible executables and data, NOT backed up. /jobfs, job scratch space, deleted at conclusion of batch job, must be explicitly requested at job submssion, make sure any essential output is copied back to /home or /short at completion of job. /fast, used for large I/O.