MPI Parallelism

MPI stands for 'Message Passing Interface', and over the last decade it has become the de-facto standard for running large scale computations on Linux clusters around the world. In most supercomputing centers this will be the ONLY option you have for running in parallel, and administrators may be actively hostile to trying to make use of any non-MPI software on their clusters.

PLEASE NOTE: Unfortunately, MPI was designed by computer scientists for computer scientists, and there can be a steep learning curve. That is, using MPI on any cluster is not a task for linux/unix novices. You must have a fair bit of education to understand what's involved in using MPI with any program (not just EMAN). You should be comfortable with running MPI jobs before attempting this with EMAN2. If necessary you may need to consult a cluster administrator for assistance. There is enough variation between different specific linux clusters that we cannot provide specific advice for every situation. We have tried to provide as much generic advice as possible, but this is often not going to be a cookie-cutter operation.

Installing MPI Support in EMAN2/SPARX

SPARX and EMAN2 have merged their MPI support efforts, and as of 4/19/2013, the legacy EMAN2 MPI system has been retired. Pydusa used to require separate installation. It is now installed as part of the Anaconda based installation system, so no further action should be required.

Testing EMAN2 MPI installation

There is a special script in EMAN2/examples called mpi_test.py. Once you have EMAN2 and Pydusa installed, you can use this script to test the basic MPI setup. Make a batch queuing script (described below), and in the script, put:

mpirun <path to python> $EMAN2DIR/examples/mpi_test.py

<path to python> can be found on the first line of any of the EMAN2 programs, eg - "head -1 $EMAN2DIR/bin/e2proc2d.py"

This program will produce console output describing the success or failure of the MPI job. If this script succeeds, then it is very likely that an EMAN2 program using the --parallel option will also succeed.

If you encounter errors, please see the Debugging section below

Using MPI in EMAN2

Once you have EMAN2 and pydusa installed, usage should be straightforward. EMAN2 has a modular parallelism system, supporting several types of parallel computing, not just MPI. All of these parallelism systems use a common syntax. For EMAN2 commands which can run in parallel, to use MPI parallelism, the basic syntax is:

--parallel=mpi:<nproc>:/path/to/scratch

for example:

e2refine.py ... --parallel=mpi:64:/scratch/stevel

• the number of processors MUST match your request to the batch system (see below)
• ideally, the compute nodes should have a local scratch directory (on a non-shared hard drive). However, since EMAN2.1, this is no longer a strict requirement.
• Do NOT use 'mpirun' directly. This will be called for you by the program you give the --parallel option to.

Special exception for e2refine_easy and other select programs

When using MPI parallelism, certain tasks become I/O limited, and are simply not efficient to run in parallel on multiple nodes. To handle this situation, we have added a --threads option to e2refine_easy and other select programs. If you specify --threads, then these i/o-limited tasks will all be run on the first assigned node using threads. For example, you might say:

--parallel=mpi:128:/scratch --threads=12

Which would run most of the refinement using the full 128 cores and MPI. In cases where this was inefficient, 12 cores on the first assigned node would be used instead.

One word of warning, however. On some clusters, with very tight management and job control, having a single process make use of all 12 cores could result in jobs being terminated. If you run into this problem, please let me know so we can try to come up with a solution.

Special Options to mpirun

By default, EMAN2 programs *internally* run mpirun with the --n <ncpus> option, and gets the list of available nodes from the batch queuing system (e.g., PBS or SGE). If you need to specify a different set of options (for example, if you aren't using PBS or SGE, and you want to specify a nodefile), you can set the environment variable "EMANMPIOPTS"(this variable will replace -n <ncpus> on the command line) by adding the following line in your invisible .bashrc file:

export EMANMPIOPTS="-hostfile myhosts.txt"

myhosts.txt should be a text file in your home directory listing the names of the nodes you want to use on the cluster.

For example:

 n1
 n2
 n3
 n4

You can then supply the --parallel command as usual, but instead of writing 'n' for the number of cpus to use, you need to write the number of nodes (--parallel=mpi:#nodes:/scratch/username ) listed in myhosts.txt. In the example above, there are 4 nodes listed; therefore, the --parallel parameter would become, in an EMAN2 command ran at the command line:

eman2whateverprogram.py --input=<input> --output=<output> --parallel=mpi:4:<scratch_directory>

IMPORTANT: Note that you usually cannot run a command from the head node; rather, you should log into any one node, for example

ssh n1

and then run your eman2 command from there. This is usually the way to do it since, typically, the head not will NOT have a "scratch" directory.

Batch Queuing systems

How to create a script to run jobs on the cluster

A cluster is a shared computing environment. Limited resources must be allocated for many different users and jobs all at the same time. To run a job on most clusters, you must formulate a request in the form of a text file containing the details about your job. These details include, the number of processors you want, how long the job is expected to run, perhaps the amount of RAM you will need, etc. This text file is called a 'batch script' and is submitted to the 'Batch Queuing System' (BQS) on your cluster. The BQS then decides when and where your job will be run, and communicates information about which specific processors to use to your job when launched.

The BQS (allocates resources and launches jobs) is independent of MPI (runs jobs on allocated resources). There are several common BQS systems you may encounter. We cannot cover every possibility here, so you need to consult with your local cluster policy information for details on how to submit jobs using your BQS. We will provide examples for OpenPBS and SGE, which are two of the more common BQS systems out there. Even then, the details may vary a little from cluster to cluster. These examples just give you someplace to start.

OpenPBS/Torque

Here is an example of a batch script for PBS-based systems:

{{{#!/bin/bash # All lines starting with "#PBS" are PBS commands # # The following line asks for 10 nodes, each of which has 12 processors, for a total of 120 CPUs. # The walltime is the maximum length of time your job will be permitted to run. If it is too small, your # job will be killed before it's done. If it's too long, however, your job may have to wait a looong # time before the cluster starts running it (depends on local policy). #PBS -l nodes=10:ppn=12 #PBS -l walltime=120:00:00 #PBS -q dque

# This prints the list of nodes your job is running on to the output file cat $PBS_NODEFILE

# cd to your project directory cd /home/stevel/data/myproject

# Now the actual EMAN2 command(s). Note the --parallel option at the end. The number of CPUs must match the number specified above e2refine_easy.py --input=sets/allctf_flip_hp.lst --model=initial_models/model_00_02.hdf --targetres=6.0 --sym=d7 --iter=3 --mass=800.0 --apix=2.1 --classkeep=0.9 --m3dkeep=0.8 --parallel=mpi:120:/scratch/stevel --threads 12 --speed 5 }}} If this file were called, for example, test.pbs, you would then submit the job to the cluster by saying

e2bdb.py -c
qsub test.pbs

There are additional options you can use with the qsub command as well. See your local cluster documentation for details on what is required/allowed. The e2bdb.py -c command is a good idea to make sure that the compute nodes will see any recent changes you've made to images in the project.

SGE (Sun Grid Engine)

This is another popular queuing system, which uses 'qsub' and 'qstat' commands much like OpenPBS/Torque does. Configuration, however, is completely different.

Here is an example of an SGE script to run a refinement with e2refine.py using mpich:

{{{#!/bin/bash #$ -S /bin/bash #$ -V #$ -N refine4 #$ -cwd #$ -j y #$ -pe mpich 40

cd /home/stevel/data/myproject

e2refine_easy.py --input=sets/allctf_flip_hp.lst --model=initial_models/model_00_02.hdf --targetres=6.0 --sym=d7 --iter=3 --mass=800.0 --apix=2.1 --classkeep=0.9 --m3dkeep=0.8 --parallel=mpi:40:/scratch/stevel --threads 4 --speed 5 }}}

Summary

1. Prepare the batch file appropriate for your cluster. Do not try to use 'mpirun' or 'mpiexec' directly on any EMAN programs. Instead, add the '--parallel=mpi:<n>:/path/to/scratch[:nocache]' option to an EMAN2 command like e2refine.py. Some commands do not support the --parallel option, and trying to run them using mpirun will not accomplish anything useful.

   • replace <n> with the total number of processors you have requested (these number must match exactly)

   • replace /path/to/scratch, with the path to a scratch storage directory available on each node of the cluster. Note that this directory must be local storage on each node, not a directory shared between nodes. If you use the path to a shared directory, like $HOME/scratch, you will have very very serious problems. You must use a filesystem local to the specific node. If you don't have this information, check your cluster documentation and/or consult with your system administrator.
   • Make sure that after the last e2* command in your batch script you put an 'e2bdb.py -cF' command to make sure all of the output image files have been flushed to disk.

2. If you need to pass special options to mpirun (like a hostfile), you can use the EMANMPIOPTS shell variable, but most users should not need this. A typical usage would be export EMANMPIOPTS="-hostfile myhosts.txt". You should only do this if necessary, though (note that then when supplying the parameter --parallel=mpi:n:scratch_directory, 'n' is no longer the number of cpus to use, but rather the number of nodes listed in myhosts.txt)

3. Submit your job.

4. When you run into problems (note I say when, not if), and you have exhausted any local MPI experts, please feel free to email me ( sludtke@bcm.edu ). Once you have things properly configured, you should be able to use MPI routinely, but getting there may be a painful process on some clusters. Don't get too discouraged.

5. The 'nocache' option is new as of EMAN2.04, and allows you to rely on your local filesysem sharing rather than caching data on each node. If you are using a shared Lustre or similar filesystem to store your data, or if your cluster doesn't have any significant local scratch space on the compute nodes, this may be beneficial, but the option is still experimental.
6. EMAN2 can make use of MPI very efficiently, however, as this type of image processing is VERY data intensive, in some situations, your jobs may be limited by data transfer between nodes rather than by the computational capacity of the cluster. The inherent scalability of your job will depend quite strongly on the parameters of your reconstruction. In general larger projects will scale better than smaller projects, but projects can be 'large' in several different ways (eg- large box size, large number of particles, low symmetry,...). If your cluster administrator complains that your jobs aren't using the CPUs that you have allocated for your jobs sufficiently, you can try A) running on fewer processors, which will increase the efficiency (but also increase run-times), or you can refer them to me, and I will explain the issues involved. We are also developing tools to help better measure how efficiently your jobs are taking advantage of the CPUs you are allocating, but this will be an ongoing process.

Debugging Problems

Every Linux cluster is different, and normally the best source of help will be your cluster support staff. If you show them this web page, they should be able to get the information they need to help you debug the problem. If not, you are always free to email for help on the EMAN2 list or directly to sludtke@bcm.edu .

• If you get an error about being unable to find libmpi.so.1, try this (may not work exactly like this on every cluster):

$ locate libmpi.so.1
/usr/lib64/openmpi/lib/libmpi.so.1
/usr/lib64/openmpi/lib/libmpi.so.1.0.2
/usr/local/lib/libmpi.so.1
/usr/local/lib/libmpi.so.1.0.8
/usr/local/openmpi-1.6.5/lib/libmpi.so.1
/usr/local/openmpi-1.6.5/lib/libmpi.so.1.0.8
/usr/local/openmpi-1.6.5/ompi/.libs/libmpi.so.1
/usr/local/openmpi-1.6.5/ompi/.libs/libmpi.so.1.0.8

$ which mpirun
/usr/local/bin/mpirun

$ echo $LD_LIBRARY_PATH
/home/stevel/lib:/home/stevel/EMAN2/lib

This would prompt me to add this to my .bashrc:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

• If you get errors like: "Undefined symbol", there are two major causes:
  • If you are using OpenMPI. make sure it is compiled with the '--disable-dlopen' option. If you are using a system-provided OpenMPI, and they cannot do this for you, you may have to install your own copy of OpenMPI (in your home directory).
  • Some systems have more than one MPI installed. You may have your $PATH pointing to one MPI and your LD_PRELOAD or LD_LIBRARY_PATH pointing at a different MPI. See the section about on libmpi.so.1 to debug this.

To set the LD_PRELOAD variable, add a similar line to the one below to your .bashrc file (which is an invisible file in your home directory) by following these steps:

1) Open your invisible .bashrc file by typing at the command line:

vi .bashrc

(vi is an obnoxious text editor, but you can look up how to use it on Google).

2) Add the following line anywhere in the .bashrc file: export LD_PRELOAD=<mpi_directory>/lib/libmpi.so making sure to replace <mpi_directory> with your mpi directory.

In my case, the directory of my mpi installation is /raid/home/jgalaz/openmpi-1.4.3

Thus, the line I added to my .bashrc fine was:

export LD_PRELOAD=/raid/home/jgalaz/openmpi-1.4.3/lib/libmpi.so

To find what mpi you're using (what the installation directory is for your local version of mpi), type at the command line:

which mpirun

I get:

/raid/home/jgalaz/openmpi-1.4.3/bin/mpirun

Therefore the path before '/bin' corresponds to <mpi_directory> (/raid/home/jgalaz/openmpi=1.4.3)

• Problems with the scratch directory specified in --parallel=mpi:N:/path/to/scratch. Previous versions of EMAN2 required local scratch space on each node of the cluster, and this space could not be shared. This requirement has been relaxed in EMAN2.1, and it should now be possible to run even on clusters that have no local scratch space, by specifying a shared scratch space. In any case, whatever directory you specify here must be writable by you, and should have a reasonable amount (perhaps a few GB) of free space.