MPI Parallelism

Once you have EMAN2 and pydusa installed, usage should be straightforward. EMAN2 has a modular parallelism system, supporting several types of parallel computing, not just MPI. All of these parallelism systems use a common syntax. For EMAN2 commands which can run in parallel, to use MPI parallelism, the basic syntax is:

--parallel=mpi:<nproc>:/path/to/scratch

for example:

e2refine.py ... --parallel=mpi:64:/scratch/stevel

• the number of processors MUST match your request to the batch system (see below)
• A node-local scratch directory is required !
  • That means, each compute node must have a locally attached hard drive where users can temporarily store scratch data.
  • It's location will vary by cluster

  • There are some clusters which don't have this. There is an experimental nocache option (--parallel=mpi:64:/tmp:nocache) which MAY help with this. Contact sludtke@bcm.edu if you have problems.

• Do NOT use 'mpirun' directly. This will be called for you by the program you give the --parallel option to.

Special Options to mpirun

By default, EMAN2 runs mpirun with the --n <ncpus> option, and gets the list of available nodes from the batch queuing system. If you need to specify a different set of options (for example, if you aren't using PBS or SGE, and you want to specify a nodefile), you can set the environment varaible "EMANMPIOPTS". This variable will replace -n <ncpus> on the command line.

Batch Queuing systems

How to create a script to run jobs on the cluster

A cluster is a shared computing environment. Limited resources must be allocated for many different users and jobs all at the same time. To run a job on most clusters, you must formulate a request in the form of a text file containing the details about your job. These details include, the number of processors you want, how long the job is expected to run, perhaps the amount of RAM you will need, etc. This text file is called a 'batch script' and is submitted to the 'Batch Queuing System' (BQS) on your cluster. The BQS then decides when and where your job will be run, and communicates information about which specific processors to use to your job when launched.

The BQS (allocates resources and launches jobs) is independent of MPI (runs jobs on allocated resources). There are several common BQS systems you may encounter. We cannot cover every possibility here, so you need to consult with your local cluster policy information for details on how to submit jobs using your BQS. We will provide examples for OpenPBS and SGE, which are two of the more common BQS systems out there. Even then, the details may vary a little from cluster to cluster. These examples just give you someplace to start.

OpenPBS/Torque

Here is an example of a batch script for PBS-based systems:

# All lines starting with "#PBS" are PBS commands
#
# The following line asks for 10 nodes, each of which has 12 processors, for a total of 120 CPUs. 
# The walltime is the maximum length of time your job will be permitted to run. If it is too small, your
# job will be killed before it's done. If it's too long, however, your job may have to wait a looong
# time before the cluster starts running it (depends on local policy).
#PBS -l nodes=10:ppn=12
#PBS -l walltime=120:00:00

# This prints the list of nodes your job is running on to the output file
cat $PBS_NODEFILE

# cd to your project directory
cd /home/stevel/data/myproject

# Now the actual EMAN2 command(s). Note the --parallel option at the end. The number of CPUs must match the number specified above
e2refine.py --input=bdb:sets#set-q5_phase_flipped --mass=1500 --apix=1.8 --automask3d=0.8,36,4,6,36 --iter=4 --sym=c4 --model=bdb:refine_05#threed_03 --path=refine_07 --orientgen=eman:delta=2.5:inc_mirror=0:perturb=1 --projector=standard --simcmp=frc:snrweight=1:zeromask=1:maxres=12 --simalign=rotate_translate_flip_iterative --simralign=refine --simraligncmp=ccc --simaligncmp=ccc  --classcmp=frc:snrweight=1:zeromask=1:maxres=12 --classalign=rotate_translate_flip_iterative --classralign=refine --classraligncmp=ccc --classaligncmp=ccc --classkeep=2.5 --classnormproc=normalize.edgemean --classaverager=mean --classiter=1 --sep=3 --m3diter=2 --m3dkeep=0.7 --recon=fourier --m3dpreprocess=normalize.edgemean --m3dpostprocess=filter.lowpass.gauss:cutoff_freq=.06 --pad=320 --classkeepsig --m3dsetsf --twostage=2 --parallel=mpi:120:/localscratch/stevel 

# Good idea to do this at the end
e2bdb.py -cF

If this file were called, for example, test.pbs, you would then submit the job to the cluster by saying

e2bdb.py -c
qsub test.pbs

There are additional options you can use with the qsub command as well. See your local cluster documentation for details on what is required/allowed. The e2bdb.py -c command is a good idea to make sure that the compute nodes will see any recent changes you've made to images in the project.

SGE (Sun Grid Engine)

This is another popular queuing system, which uses 'qsub' and 'qstat' commands much like OpenPBS/Torque does. Configuration, however, is completely different.

Here is an example of an SGE script to run a refinement with e2refine.py using mpich:

#$ -S /bin/bash
#$ -V
#$ -N refine4
#$ -cwd
#$ -j y
#$ -pe mpich 40

e2refine.py --input=bdb:sets#set2-allgood_phase_flipped-hp --mass=1200.0 --apix=2.9 --automask3d=0.7,24,9,9,24 --iter=1 --sym=c1 --model=bdb:refine_02#threed_filt_05 --path=refine_sge --orientgen=eman:delta=3:inc_mirror=0 --projector=standard --simcmp=frc:snrweight=1:zeromask=1 --simalign=rotate_translate_flip --simaligncmp=ccc --simralign=refine --simraligncmp=frc:snrweight=1 --twostage=2 --classcmp=frc:snrweight=1:zeromask=1 --classalign=rotate_translate_flip --classaligncmp=ccc --classralign=refine --classraligncmp=frc:snrweight=1 --classiter=1 --classkeep=1.5 --classnormproc=normalize.edgemean --classaverager=ctf.auto --sep=5 --m3diter=2 --m3dkeep=0.9 --recon=fourier --m3dpreprocess=normalize.edgemean --m3dpostprocess=filter.lowpass.gauss:cutoff_freq=.1 --pad=256 --lowmem --classkeepsig --classrefsf --m3dsetsf -v 2 --parallel=mpi:40:/scratch/username

e2bdb.py -cF

Summary

1. Prepare the batch file appropriate for your cluster. Do not try to use 'mpirun' or 'mpiexec' directly on any EMAN programs. Instead, add the '--parallel=mpi:<n>:/path/to/scratch[:nocache]' option to an EMAN2 command like e2refine.py. Some commands do not support the --parallel option, and trying to run them using mpirun will not accomplish anything useful.

   • replace <n> with the total number of processors you have requested (these number must match exactly)

   • replace /path/to/scratch, with the path to a scratch storage directory available on each node of the cluster. Note that this directory must be local storage on each node, not a directory shared between nodes. If you use the path to a shared directory, like $HOME/scratch, you will have very very serious problems. You must use a filesystem local to the specific node. If you don't have this information, check your cluster documentation and/or consult with your system administrator.
   • Make sure that after the last e2* command in your batch script you put an 'e2bdb.py -cF' command to make sure all of the output image files have been flushed to disk.

2. If you need to pass special options to mpirun (like a hostfile), you can use the EMANMPIOPTS shell variable, but most users should not need this.

3. Submit your job.

4. When you run into problems (note I say when, not if), and you have exhausted any local MPI experts, please feel free to email me (sludtke@bcm.edu). Once you have things properly configured, you should be able to use MPI routinely, but getting there may be a painful process on some clusters. Don't get too discouraged.

5. The 'nocache' option is new as of EMAN2.04, and allows you to rely on your local filesysem sharing rather than caching data on each node. If you are using a shared Lustre or similar filesystem to store your data, or if your cluster doesn't have any significant local scratch space on the compute nodes, this may be beneficial, but the option is still experimental.
6. EMAN2 can make use of MPI very efficiently, however, as this type of image processing is VERY data intensive, in some situations, your jobs may be limited by data transfer between nodes rather than by the computational capacity of the cluster. The inherent scalability of your job will depend quite strongly on the parameters of your reconstruction. In general larger projects will scale better than smaller projects, but projects can be 'large' in several different ways (eg- large box size, large number of particles, low symmetry,...). If your cluster administrator complains that your jobs aren't using the CPUs that you have allocated for your jobs sufficiently, you can try A) running on fewer processors, which will increase the efficiency (but also increase run-times), or you can refer them to me, and I will explain the issues involved. We are also developing tools to help better measure how efficiently your jobs are taking advantage of the CPUs you are allocating, but this will be an ongoing process.