NCI National Facility


Home	Notices and News	Accounts	Facilities, Software and Userguides	Frequently Asked Questions	Training	Annual Reports

Newsletter April 2004

Table of Contents

APAC over the next three years
Employment opportunities
Gaussian workshop
APAC NF annual reports
Recent software acquisitions and updates
MDSS changes
SC system upgrade
Intel compilers on the LC
Running parallel jobs on the LC

APAC over the next three years The Federal Government has committed $29M to APAC over 2004-2006. The main objectives of APAC for 2004-2006 are to:

develop a national strategy for advanced computing and grid infrastructure to support eResearch in Australia;
strengthen the APAC partnership to provide national cooperation on Australia's advanced computing and grid infrastructure;
improve the peak computing capabilities of the APAC National Facility to serve the demands of Australian researchers;
develop the APAC Grid to provide Australian researchers with seamless access to the resources of the National Facility and with advanced services for eResearch;
develop and deliver education, outreach and training services for users of advanced computing and grid services.

Partners will be engaged in three major programs:

National Facility program, including a Computational Tools and Techniques sub-program

APAC Grid program, to establish and apply grid infrastructure in support of research

Education, outreach and training program

Replacement of AlphaServer SC:
APAC plans to replace the AlphaServer SC with a substantially more powerful system late this year or early next year.

Employment opportunities The ANU is currently advertising several positions in support of the APAC Grid program and the Computational Tools and Techniques program as well as positions in support of the National Facility. Details can be found at http://anusf.anu.edu.au/Positions_Vacant. Please bring these to the attention of anyone who might be interested. Closing date is 19 April 2004.

Gaussian workshop Australia's first Gaussian Workshop will be held June 28 - July 1, 2004 at the University of Sydney. Registration has commenced on the Gaussian website http://www.gaussian.com/g_workshops/ws_sydney.htm. For more details see: http://nf.apac.edu.au/notices_news/GaussianWorkshop/index.html
HP and APAC have kindly provided funds to be used towards grants for National Facility users to attend the Workshop. We have not yet decided on the amounts that will be provided or who will be eligible for these grants. If you are interested in attending the Gaussian Workshop and would like to apply for an HP/APAC grant see: http://www.nf.apac.edu.au/notices_news/travel_grants.php. The Gaussian Workshop will be directly preceding MM2004 and there will be special Conference Rates for Gaussian Workshop participants.

APAC NF annual reports To date 189 annual reports have been received and these are progressively being added to the web site at http://nf.apac.edu.au/annual_reports/2003/data/.

Recent software acquisitions and updates There have been several updates to software packages on the system. (Software available is listed here.) Remember to set the PBS software flag for your package to ensure that the package is available when your job runs on a node. The PBS software keyword flag is listed for each package under the appropriate software web page. In the near future batch jobs which do not have a necessary software flag will not be able to run. Contact help@nf.apac.edu.au if you have any problems with this flag.

Recent package updates on the SC include Gaussian G03 to version b05, Fortran 90/95 compilers to V5.5A, R to version R-1.8.1, NAMD to NAMD-2.5.
On the LC Gaussian, GAMESS and R have been updated to the same versions as on the SC.

Several users are currently testing some matlab toolboxes. If you have an interest, you can get short term access by setting the environment variable USE_MATLABSP2.

MDSS changes In January 2004 the MDSS (store.apac.edu.au) was migrated to a SunFire V480 with 4 CPUs, 8GB RAM, 3x2Gb/s fibre channel, 2x1000baseTX gigabit ethernet and 2xHVD SCSI interfaces to support the 4.5TB D280 dual-path disk array and 4x9940B and 8x9840A tape drives. The new PCI-based server removes the limitations of the SBus architecture of the old E6500 server, providing much increased SAM-FS throughput.
The development environment on the new server is changing to be similar to that of the SC and LC. This includes MDSS versions of /short directories and the USE_ scheme for software installations.
We have also created a /projects/XXX directory, where XXX refers to the APAC project code. This area lives on a non-migrating filesystem, is backed up and will have project wide quotas. At the moment not all projects have directories created. If yours does not exist and you require it please contact us on help@nf.apac.edu.au and we will create it for you.

SC system upgrade The system software of the SC underwent a major upgrade (to the SCV2.6 "Eagle" release) in January and is not expected to gain any major new functionality during the remaining lifetime of the system. Hopefully this will mean that future bugfix upgrades will require only a short interruption to service.
One of the major changes in "Eagle" involves the behaviour of the MPI and Elan (low level communication) libraries of the SC. Previous versions reported a very high cpu usage when programs were waiting on communication rather than doing computation. Thus it was not always possible to distinguish communication bound or load imbalanced applications from compute bound ones. With "Eagle", users may notice a decrease in the %CPU usage of their MPI applications. The cpu usage information provided by the qps command now gives a better indication of any load imbalance problems in jobs. In particular, jobs that have one process per node reporting less cpu usage than the others are most probably wasting time waiting in collective operations.
The only definitive means of understanding your MPI applications communication and load balance behaviour is through the use of Vampir. High %CPU usage may still be masking poor parallel application performance. We strongly recommend that all MPI users make use of Vampir as much as possible. Contact help for assistance and advice.

Intel compilers on the LC Set the environment variable USE_INTEL8 to set up the paths for the latest V8 compilers. V7 can be accessed by using USE_INTEL7.
A couple of additional items are worth noting.
The compiler for the INTEL V8 Fortran compiler is called ifort whereas the compiler for V7 is called ifc;
The PEPCF90 is not needed for the V8 compiler;
For the V8 compiler to compile statically include -static -static-libcxa
The V8 compiler produces faster code, but please check the results carefully.
The lam and mpich libraries are not available for the V8 compilers at the moment.
Not all packages and libraries on the LC that have been built with the Intel compilers have been rebuilt with the new compiler. The new scheme for libraries will include an Intel8 tree which will refer to compilation with the Intel8 compiler. If you have an immediate need for one of the libraries to be rebuilt with this version then email help@nf.apac.edu.au to request that it be built. If only an Intel library subdirectory exists then you can assume that it was built with the V7 compiler.

Running parallel jobs on the LC The Linux cluster LC was added to the APAC National Facility in April 2003 to help free up the SC to run parallel applications. The LC configuration was chosen without regard to parallel use. In particular, the gigabit interconnect has a latency more than 10 times greater than that of the SC. This means that tightly coupled parallel applications may perform quite poorly on LC. Also the queueing system configuration does not favour parallel jobs to the same extent as it does on the SC. Ideally, parallel ensemble jobs should get the same throughput if they are submitted as multiple single cpu jobs as they would if submitted as a parallel job.
If you do choose to try using MPI on LC, the preferred MPI implementation is LAM which is currently available as both lam-6.5.8 and lam-7.0. It is likely that this will be rationalized to only lam-7.0 shortly.

Email problems, suggestions, questions to

APAC over the next three years	The Federal Government has committed $29M to APAC over 2004-2006. The main objectives of APAC for 2004-2006 are to: develop a national strategy for advanced computing and grid infrastructure to support eResearch in Australia; strengthen the APAC partnership to provide national cooperation on Australia's advanced computing and grid infrastructure; improve the peak computing capabilities of the APAC National Facility to serve the demands of Australian researchers; develop the APAC Grid to provide Australian researchers with seamless access to the resources of the National Facility and with advanced services for eResearch; develop and deliver education, outreach and training services for users of advanced computing and grid services. Partners will be engaged in three major programs: National Facility program, including a Computational Tools and Techniques sub-program APAC Grid program, to establish and apply grid infrastructure in support of research Education, outreach and training program Replacement of AlphaServer SC: APAC plans to replace the AlphaServer SC with a substantially more powerful system late this year or early next year.

Employment opportunities	The ANU is currently advertising several positions in support of the APAC Grid program and the Computational Tools and Techniques program as well as positions in support of the National Facility. Details can be found at http://anusf.anu.edu.au/Positions_Vacant. Please bring these to the attention of anyone who might be interested. Closing date is 19 April 2004.

Gaussian workshop	Australia's first Gaussian Workshop will be held June 28 - July 1, 2004 at the University of Sydney. Registration has commenced on the Gaussian website http://www.gaussian.com/g_workshops/ws_sydney.htm. For more details see: http://nf.apac.edu.au/notices_news/GaussianWorkshop/index.html HP and APAC have kindly provided funds to be used towards grants for National Facility users to attend the Workshop. We have not yet decided on the amounts that will be provided or who will be eligible for these grants. If you are interested in attending the Gaussian Workshop and would like to apply for an HP/APAC grant see: http://www.nf.apac.edu.au/notices_news/travel_grants.php. The Gaussian Workshop will be directly preceding MM2004 and there will be special Conference Rates for Gaussian Workshop participants.

APAC NF annual reports	To date 189 annual reports have been received and these are progressively being added to the web site at http://nf.apac.edu.au/annual_reports/2003/data/.

Recent software acquisitions and updates	There have been several updates to software packages on the system. (Software available is listed here.) Remember to set the PBS software flag for your package to ensure that the package is available when your job runs on a node. The PBS software keyword flag is listed for each package under the appropriate software web page. In the near future batch jobs which do not have a necessary software flag will not be able to run. Contact help@nf.apac.edu.au if you have any problems with this flag. Recent package updates on the SC include Gaussian G03 to version b05, Fortran 90/95 compilers to V5.5A, R to version R-1.8.1, NAMD to NAMD-2.5. On the LC Gaussian, GAMESS and R have been updated to the same versions as on the SC. Several users are currently testing some matlab toolboxes. If you have an interest, you can get short term access by setting the environment variable USE_MATLABSP2.

MDSS changes	In January 2004 the MDSS (store.apac.edu.au) was migrated to a SunFire V480 with 4 CPUs, 8GB RAM, 3x2Gb/s fibre channel, 2x1000baseTX gigabit ethernet and 2xHVD SCSI interfaces to support the 4.5TB D280 dual-path disk array and 4x9940B and 8x9840A tape drives. The new PCI-based server removes the limitations of the SBus architecture of the old E6500 server, providing much increased SAM-FS throughput. The development environment on the new server is changing to be similar to that of the SC and LC. This includes MDSS versions of /short directories and the USE_ scheme for software installations. We have also created a /projects/XXX directory, where XXX refers to the APAC project code. This area lives on a non-migrating filesystem, is backed up and will have project wide quotas. At the moment not all projects have directories created. If yours does not exist and you require it please contact us on help@nf.apac.edu.au and we will create it for you.

SC system upgrade	The system software of the SC underwent a major upgrade (to the SCV2.6 "Eagle" release) in January and is not expected to gain any major new functionality during the remaining lifetime of the system. Hopefully this will mean that future bugfix upgrades will require only a short interruption to service. One of the major changes in "Eagle" involves the behaviour of the MPI and Elan (low level communication) libraries of the SC. Previous versions reported a very high cpu usage when programs were waiting on communication rather than doing computation. Thus it was not always possible to distinguish communication bound or load imbalanced applications from compute bound ones. With "Eagle", users may notice a decrease in the %CPU usage of their MPI applications. The cpu usage information provided by the `qps` command now gives a better indication of any load imbalance problems in jobs. In particular, jobs that have one process per node reporting less cpu usage than the others are most probably wasting time waiting in collective operations. The only definitive means of understanding your MPI applications communication and load balance behaviour is through the use of Vampir. High %CPU usage may still be masking poor parallel application performance. We strongly recommend that all MPI users make use of Vampir as much as possible. Contact help for assistance and advice.

Intel compilers on the LC	Set the environment variable USE_INTEL8 to set up the paths for the latest V8 compilers. V7 can be accessed by using USE_INTEL7. A couple of additional items are worth noting. The compiler for the INTEL V8 Fortran compiler is called `ifort` whereas the compiler for V7 is called `ifc`; The `PEPCF90` is not needed for the V8 compiler; For the V8 compiler to compile statically include `-static -static-libcxa` The V8 compiler produces faster code, but please check the results carefully. The lam and mpich libraries are not available for the V8 compilers at the moment. Not all packages and libraries on the LC that have been built with the Intel compilers have been rebuilt with the new compiler. The new scheme for libraries will include an Intel8 tree which will refer to compilation with the Intel8 compiler. If you have an immediate need for one of the libraries to be rebuilt with this version then email help@nf.apac.edu.au to request that it be built. If only an Intel library subdirectory exists then you can assume that it was built with the V7 compiler.

Running parallel jobs on the LC	The Linux cluster LC was added to the APAC National Facility in April 2003 to help free up the SC to run parallel applications. The LC configuration was chosen without regard to parallel use. In particular, the gigabit interconnect has a latency more than 10 times greater than that of the SC. This means that tightly coupled parallel applications may perform quite poorly on LC. Also the queueing system configuration does not favour parallel jobs to the same extent as it does on the SC. Ideally, parallel ensemble jobs should get the same throughput if they are submitted as multiple single cpu jobs as they would if submitted as a parallel job. If you do choose to try using MPI on LC, the preferred MPI implementation is LAM which is currently available as both lam-6.5.8 and lam-7.0. It is likely that this will be rationalized to only lam-7.0 shortly.