# GFM

##### Secções
Você está aqui: Entrada MPICH + ABINIT micro-HOWTO
J M Cerqueira Esteves
Apoio técnico
Webmaster
Telefone: +351 939838775

# MPICH + ABINIT micro-HOWTO

An abridged tale of MPICH (1.2.6) and ABINIT (4.4.4) compilation/installation in an x86_64 AMD Opteron system running GNU/Linux, using PGI (5.2) compilers.

Easily maintaining different releases of these software packages in a single machine is sometimes convenient, so I chose to install them under /opt instead of under the /usr/local hierarchy. In any case, never install anything by hand' directly under the /usr hierarchy, which should be used only for software included in your GNU/Linux distribution or at least packaged in strict conformance with your distribution standards. Doing so can prevent seriously messing up your system; always use /usr/local or /opt instead. The Filesystem Hierarchy Standard is recommended reading.

### Compiling and installing MPICH (for the ch_shmem device)

These notes concern the specific case of a 64-bit AMD Opteron system with Portland group compilers. When compiling for other platforms, carefully browse the suggestions on compiler switches, confronting them with man pages for the pfg90 and pgcc compilers. The MPICH buglist for version 1.2.6 includes more notes on compiler switches and warns about problems known in MPICH itself and in some compilers.

Downloaded the source distribution of MPICH release 1.2.6 and the installation and user's guide [PDF] specific for the ch_shmem abstract device interface (SMP systems, shared memory communication). This should be read carefully before installation. I applied the patches currently listed in the 1.2.6 buglist page (10039, 10246, 10243) and left the resulting source tree at /opt/src/mpich-1.2.6-AN.1

I chose to install this MPICH instance under /opt/mpich-1.2.6/ch_shmem; taking into account elements collected from the above sources, I configured, installed and tested MPICH as follows:

cd /opt/src/mpich-1.2.6-AN.1env CC=pgcc CFLAGS='-Msignextend' FC=pgf77 FFLAGS= CXX=pgCC CCFLAGS= F90=pgf90 F90FLAGS= ./configure --prefix=/opt/mpich-1.2.6/ch_shmem  --with-device=ch_shmem -fc=pgf77 -lib="-L/usr/pgi/linux86-64/5.2/lib -lpgftnrtl -lpgc" -f90=pgf90 -opt=-fast --enable-debugmakemake installcd examples/test/./configure -mpichpath=/opt/mpich-1.2.6/ch_shmem/binmake testing

Do not worry about a failure of the test pt2pt/structf with the message

0 - Error in MPI_ADDRESS : Invalid argument: Address of location given to MPI_ADDRESS does not fit in Fortran integer [0] Aborting program!
since this is not an error (at least in a 64-bit environment where Fortran integers have 4 bytes...):
it is a gap in the MPI-1 definition that is fixed in MPI-2 (with the routine MPI_Get_address. This indicates that Fortran integers are not large enough to hold an address. This does indicate that MPI programs written in Fortran should not use the MPI_Address function on this system.

Since this was currently the only MPICH implementation in the target computer, it was also convenient to make the resulting commands be in the search path of any user. I added /opt/bin to the general PATH and did:

cd /opt/binln -sv /opt/mpich-1.2.6/ch_shmem/bin/* .

Under different circunstances, namely if you have several MPICH versions installed, or the same version compiled for more than one device interface, it will probably be more convenient to:

• have each user adding a single /opt/mpich-1.2.6/.../bin to his/her PATH, in case he/she is only wishing to use a specific device;
• or locally adjust PATH or explicitly state program pathnames in each script used to run MPICH-dependent programs.

In any case, I created a /opt/etc/mpich-ch_shmem-environment file containing

PATH=/opt/mpich-1.2.6/ch_shmem/bin/:$PATHMANPATH=$MANPATH:/opt/mpich-1.2.6/ch_shmem/man/export MANPATH

Running source /opt/etc/mpich-ch_shmem-environment in the current shell adjusts the command search path and allows you to browse MPICH man pages (eventually specific for the ch_shmem device). You may of course add it to your .bash_profile...

### Compiling and installing ABINIT

Before installation, carefully read the installation notes (also included in the source as Infos/Installation_notes/install.html) and the additional parallelism info in Infos/paral_use. One must very carefully choose the right settings for the makefile_macros file. In this particular case, makefile_macros was based on the file Machine_dept_files/Opteron/makefile_macros_ingrid_pgi5.2-64bit included in ABINIT's source and on a makefile_macros.PGIOpteron supplied by Paulo Couto (obtained from where?) which seemed to have carefully crafted PGI compiler options. We ended up with the following makefile_macros:

##################################################################### For the sequential version # Machine typeMACHINE=P6# Fortran optimized compilation.FC=/opt/mpich-1.2.6/ch_shmem/bin/mpif90FFLAGS=-fast -Mfree -r8 -O3 -tp k8-64 -Munroll -Mvect=sse -Mvect=assoc,cachesize:1048576 -Mcache_align -Kieee -Mvect=prefetchFFLAGS_LIBS =-fast -r8 -O2 -tp k8-64 -Munroll -Mvect=sse -Mvect=assoc,cachesize:1048576 -Mcache_align -Kieee -Mvect=prefetch# C preprocessor, used to preprocess the fortran source.CPP=/lib/cpp CPP_FLAGS=-P -traditional -DCONTRACT# C optimized compilation.CC=pgccCFLAGS=-fastsse# Location of perl . Used to generate the script fldiff, in ~ABINIT/Utilities .PERL=/usr/bin/perl# List of machine-dependent routinesMACHINE_DEP_C_SEQ_SUBS_LIST=etime.o# Use the ABINIT default libraries.# For dynamical link, uncomment this line#LIBS= $(LAPACK)$(BLAS) $(NETCDF)LIBS=-L/usr/pgi/linux86-64/5.2/lib -llapack -lblasNC_CC=pgccNC_FC=pgf77NC_F90=pgf90NC_CPPFLAGS=-DpgiFortran##################################################################### For the parallel version.# Compiler flags and definitionsFFLAGS_PAR=$(FFLAGS) -I /opt/mpich-1.2.6/ch_shmem/include/# List of machine-dependent routinesMACHINE_DEP_C_PAR_SUBS_LIST=etime.par# Location of the MPI library##MPI_A=/usr/local/mpi-pgi/lib/libfmpich.a /usr/local/mpi-pgi/lib/libmpich.a# Include blas, lapack, and any other libraries hereLIBS_PAR=\$(LIBS)# Command for automatic tests in parallel (optional)COMMAND_PAR=/opt/mpich-1.2.6/ch_shmem/bin/mpirun -np 2 ../../abinip##################################################################### For integration of netcdf# The following lines may be uncommented if the ./configure script for# netcdf doesn't find the appropriate tools.# The script Lib_netcdf/src/nc_configure will strip the NC_ prefix,# set the appropriate environment variables# and call the standard netcdf ./configure script# Note that in this example, the CFLAGS is given to define the "compiler platform"# as expected by netcdf (not always required, the compiler may define# the adequate environment variable or configur may find it).# For examples of environment variables used on different platforms see:# http://www.unidata.ucar.edu/packages/netcdf/INSTALL.htmlNC_FC=/usr/pgi/linux86-64/5.2/bin/pgf77NC_F90=/usr/pgi/linux86-64/5.2/bin/pgf90NC_CPPFLAGS='-DNDEBUG -DpgiFortran'#NC_CC=#NC_CFLAGS=#NC_FFLAGS=#NC_F90FLAGS=

If you are compiling for another platform and/or compiler suite, search for a seemingly appropriate makefile_macros file under Machine_dept_files and carefully check command options in the documentation for your compiler programs. You will have to make choices concerning BLAS and LAPACK libraries. With the above settings, ABINIT was linked with BLAS and LAPACK libraries supplied with the Portland group compilers being used. Notice that there was no need for explicit reference to the location of the MPICH library files, since the mpif90 script was used instead of directly invoking the pgf90 compiler.

After an appropriate makefile_macros is built, compilation is (hopefully) simple. Redirect output to log' files so that you can carefully check everything later (you may follow progress during each step with tail -f XXX.log):

cd /opt/src/abinit-4.4.4make allseq >& make-allseq.logmake abinip >& make-abinip.logmake test1 >& make-test1.logmake tests >& make-tests.log

The installation notes suggest additional tests. I only performed those under directory Test_paral. Running these requires some editing of the Test_paral/Run script.

[...]

I then copied all resulting executable files from /opt/src/abinit-4.4.4 to /opt/abinit-4.4.4/mpich-ch_shmem/bin/ and the documentation tree /opt/src/abinit-4.4.4/Infos to /opt/abinit-4.4.4/mpich-ch_shmem/doc/ . Since this was currently the only ABINIT implementation in the target computer, I also did:

cd /opt/binln -sv /opt/abinit-4.4.4/mpich-ch_shmem/bin/* .

mpirun -np NUMBER_OF_PROCESSES abinip < INPUT_FILE