Easily maintaining different releases of these software packages in a single machine is sometimes convenient, so I chose to install them under /opt instead of under the /usr/local hierarchy. In any case, never install anything `by hand' directly under the /usr hierarchy, which should be used only for software included in your GNU/Linux distribution or at least packaged in strict conformance with your distribution standards. Doing so can prevent seriously messing up your system; always use /usr/local or /opt instead. The Filesystem Hierarchy Standard is recommended reading.

Compiling and installing MPICH (for the ch_shmem device)

These notes concern the specific case of a 64-bit AMD Opteron system with Portland group compilers. When compiling for other platforms, carefully browse the suggestions on compiler switches, confronting them with man pages for the pfg90 and pgcc compilers. The MPICH buglist for version 1.2.6 includes more notes on compiler switches and warns about problems known in MPICH itself and in some compilers.

Downloaded the source distribution of MPICH release 1.2.6 and the installation and user's guide [PDF] specific for the ch_shmem abstract device interface (SMP systems, shared memory communication). This should be read carefully before installation. I applied the patches currently listed in the 1.2.6 buglist page (10039, 10246, 10243) and left the resulting source tree at /opt/src/mpich-1.2.6-AN.1

I chose to install this MPICH instance under /opt/mpich-1.2.6/ch_shmem; taking into account elements collected from the above sources, I configured, installed and tested MPICH as follows:

cd /opt/src/mpich-1.2.6-AN.1
env CC=pgcc CFLAGS='-Msignextend' FC=pgf77 FFLAGS= CXX=pgCC CCFLAGS= F90=pgf90 F90FLAGS= ./configure --prefix=/opt/mpich-1.2.6/ch_shmem  --with-device=ch_shmem -fc=pgf77 -lib="-L/usr/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc" -f90=pgf90 -opt=-fast --enable-debug
make
make install
cd examples/test/
./configure -mpichpath=/opt/mpich-1.2.6/ch_shmem/bin
make testing

Do not worry about a failure of the test pt2pt/structf with the message

0 - Error in MPI_ADDRESS : Invalid argument: Address of location given to MPI_ADDRESS does not fit in Fortran integer [0] Aborting program!

it is a gap in the MPI-1 definition that is fixed in MPI-2 (with the routine MPI_Get_address. This indicates that Fortran integers are not large enough to hold an address. This does indicate that MPI programs written in Fortran should not use the MPI_Address function on this system.

Since this was currently the only MPICH implementation in the target computer, it was also convenient to make the resulting commands be in the search path of any user. I added /opt/bin to the general PATH and did:

cd /opt/bin
ln -sv /opt/mpich-1.2.6/ch_shmem/bin/* .

Under different circunstances, namely if you have several MPICH versions installed, or the same version compiled for more than one device interface, it will probably be more convenient to:

• have each user adding a single /opt/mpich-1.2.6/.../bin to his/her PATH, in case he/she is only wishing to use a specific device;
• or locally adjust PATH or explicitly state program pathnames in each script used to run MPICH-dependent programs.

In any case, I created a /opt/etc/mpich-ch_shmem-environment file containing

PATH=/opt/mpich-1.2.6/ch_shmem/bin/:$PATH
MANPATH=$MANPATH:/opt/mpich-1.2.6/ch_shmem/man/
export MANPATH

Running source /opt/etc/mpich-ch_shmem-environment in the current shell adjusts the command search path and allows you to browse MPICH man pages (eventually specific for the ch_shmem device). You may of course add it to your .bash_profile...

Compiling and installing ABINIT

Downloaded the ABINIT 4.4.4 source and expanded it at /opt/src/abinit-4.4.4.

Before installation, carefully read the installation notes (also included in the source as Infos/Installation_notes/install.html) and the additional parallelism info in Infos/paral_use. One must very carefully choose the right settings for the makefile_macros file. In this particular case, makefile_macros was based on the file Machine_dept_files/Opteron/makefile_macros_ingrid_pgi5.2-64bit included in ABINIT's source and on a makefile_macros.PGIOpteron supplied by Paulo Couto (obtained from where?) which seemed to have carefully crafted PGI compiler options. We ended up with the following makefile_macros:

####################################################################
# For the sequential version 

# Machine type
MACHINE=P6

# Fortran optimized compilation.
FC=/opt/mpich-1.2.6/ch_shmem/bin/mpif90
FFLAGS=-fast -Mfree -r8 -O3 -tp k8-64 -Munroll -Mvect=sse -Mvect=assoc,cachesize:1048576 -Mcache_align -Kieee -Mvect=prefetch
FFLAGS_LIBS =-fast -r8 -O2 -tp k8-64 -Munroll -Mvect=sse -Mvect=assoc,cachesize:1048576 -Mcache_align -Kieee -Mvect=prefetch

# C preprocessor, used to preprocess the fortran source.
CPP=/lib/cpp 
CPP_FLAGS=-P -traditional -DCONTRACT

# C optimized compilation.
CC=pgcc
CFLAGS=-fastsse

# Location of perl . Used to generate the script fldiff, in ~ABINIT/Utilities .
PERL=/usr/bin/perl

# List of machine-dependent routines
MACHINE_DEP_C_SEQ_SUBS_LIST=etime.o

# Use the ABINIT default libraries.
# For dynamical link, uncomment this line
#LIBS= $(LAPACK) $(BLAS) $(NETCDF)
LIBS=-L/usr/pgi/linux86-64/5.2/lib -llapack -lblas
NC_CC=pgcc
NC_FC=pgf77
NC_F90=pgf90
NC_CPPFLAGS=-DpgiFortran

####################################################################
# For the parallel version.

# Compiler flags and definitions
FFLAGS_PAR= $(FFLAGS) -I /opt/mpich-1.2.6/ch_shmem/include/
# List of machine-dependent routines
MACHINE_DEP_C_PAR_SUBS_LIST=etime.par

# Location of the MPI library
##MPI_A=/usr/local/mpi-pgi/lib/libfmpich.a /usr/local/mpi-pgi/lib/libmpich.a
# Include blas, lapack, and any other libraries here
LIBS_PAR=$(LIBS)

# Command for automatic tests in parallel (optional)
COMMAND_PAR=/opt/mpich-1.2.6/ch_shmem/bin/mpirun -np 2 ../../abinip

####################################################################
# For integration of netcdf

# The following lines may be uncommented if the ./configure script for
# netcdf doesn't find the appropriate tools.
# The script Lib_netcdf/src/nc_configure will strip the NC_ prefix,
# set the appropriate environment variables
# and call the standard netcdf ./configure script
# Note that in this example, the CFLAGS is given to define the "compiler platform"
# as expected by netcdf (not always required, the compiler may define
# the adequate environment variable or configur may find it).
# For examples of environment variables used on different platforms see:
# http://www.unidata.ucar.edu/packages/netcdf/INSTALL.html
NC_FC=/usr/pgi/linux86-64/5.2/bin/pgf77
NC_F90=/usr/pgi/linux86-64/5.2/bin/pgf90
NC_CPPFLAGS='-DNDEBUG -DpgiFortran'
#NC_CC=
#NC_CFLAGS=
#NC_FFLAGS=
#NC_F90FLAGS=

If you are compiling for another platform and/or compiler suite, search for a seemingly appropriate makefile_macros file under Machine_dept_files and carefully check command options in the documentation for your compiler programs. You will have to make choices concerning BLAS and LAPACK libraries. With the above settings, ABINIT was linked with BLAS and LAPACK libraries supplied with the Portland group compilers being used. Notice that there was no need for explicit reference to the location of the MPICH library files, since the mpif90 script was used instead of directly invoking the pgf90 compiler.

After an appropriate makefile_macros is built, compilation is (hopefully) simple. Redirect output to `log' files so that you can carefully check everything later (you may follow progress during each step with tail -f XXX.log):

cd /opt/src/abinit-4.4.4
make allseq >& make-allseq.log
make abinip >& make-abinip.log
make test1 >& make-test1.log
make tests >& make-tests.log

The installation notes suggest additional tests. I only performed those under directory Test_paral. Running these requires some editing of the Test_paral/Run script.

[...]

I then copied all resulting executable files from /opt/src/abinit-4.4.4 to /opt/abinit-4.4.4/mpich-ch_shmem/bin/ and the documentation tree /opt/src/abinit-4.4.4/Infos to /opt/abinit-4.4.4/mpich-ch_shmem/doc/ . Since this was currently the only ABINIT implementation in the target computer, I also did:

cd /opt/bin
ln -sv /opt/abinit-4.4.4/mpich-ch_shmem/bin/* .

(remember that /opt/bin had been added to the general PATH).

Parallel execution

After the above, any user may run the parallel version of ABINIT, abinip, with

mpirun -np NUMBER_OF_PROCESSES abinip < INPUT_FILE