Transitioning from EMAN1 to EMAN2 with command line tools

While the workflow is the correct choice for many users, we realize that some prefer to manipulate their data more directly. Many users who wish to use the command line are doing so because they are trying to update existing scripts currently using EMAN1. Note that this tutorial was written for EMAN2.03 or above.

Introduction to the EMAN2 Commmand-Line

The first thing to note is that EMAN2 uses standard unix-style command-line arguments rather than EMAN1 style arguments. For example: proc2d abc.hed def.spi clip=64,64 --> e2proc2d.py abc.hed def.spi --clip=64,64

Basic Image Processing

In direct analog to EMAN1, EMAN2 has e2iminfo.py, e2proc2d.py, e2proc3d.py. As with EMAN1, both of the proc commands can be used to convert among any of EMAN2's supported file formats, simply by using the correct extension for the output file. There is also an --outtype option to manually specify the output format.

It is important to understand how both programs deal with image stacks (a single file with multiple images). In EMAN1, only 2-D images could exist in stacks, simply because there were no 3-D file formats that supported multiple 3-D volumes at the time. That is no longer true, and HDF and BDB both support stacks of volumes. However, with 2-D images, stacks are used extensively, and with 3-D, only rarely. As a result e2proc2d.py and e2proc3d.py handle stacks differently by default. e2proc2d.py will append images to the output stack unless --inplace is specified. e2proc3d.py on the other hand will overwrite images by default, unless the --append option is specified. Finally, e2proc2d.py is capable of treating a single 3-D image as a stack of 2-D images in various ways. so:

• e2proc2d.py file2d.hdf file2d.hdf --mult=-1 - would make an inverted copy of all of the images in file.hdf and append them to the end, doubling the size of the stack

• e2proc3d.py file3d.hdf file3d.hdf --mult=-1 - would invert all of the 3-D volumes in file3d.hdf in-place. The file would be the same size when complete

Beyond this, a number of options have changed. In both e2proc2d.py and e2proc3d.py, it is now possible to construct filter chains; that is, a sequence of filters or other processors to apply to the image(s). This is done via the --process directive in either program. There are currently over 175 different processors in 10 different categories, such as filters, masks, thresholds, mathematical, etc. Any of these processors can be applied in sequence using the directive --process=procname:option=value:option=value. You can get a list of available processors using e2help.py processor or e2help.py processor -v 2 for more detail. For example:

• e2proc3d.py file3d.hdf file3d_filt.hdf --process=filter.lowpass.gauss:cutoff_freq=0.1 --process=mask.sharp:outer_radius=64

If you prefer to adjust filter parameters in a GUI, the new e2filtertool.py program allows you to graphically construct and edit filter chains.

For comparative tasks, such as computing a FSC between two maps, note that the option order has changed, so the second parameter is an output not an input:

• e2proc3d.py file3d.hdf fsc.txt --calcfsc=file3d_2.hdf

Note that like EMAN1, the --scale=scale_factor --clip=x[,y,z] operation performs sampling or interpolation, and does not do any averaging. If you wish to use scale factors <1.0, you may wish to consider the --meanshrink= or --medianshrink= options.

CTF determination/correction

All CTF processing is handled by the e2ctf.py program. To display the graphical interface, you must specify the --gui option, but this requires first having run automatic parameter determination. For example, if you have particle data in the ptcl directory, with particles from each frame in a separate HDF file:

• Important note: EMAN2's CTF correction requires a larger box size than EMAN1, both to improve correction accuracy and permit better background assessment. In EMAN2, the box size should be 1.5x - 2x the maximum dimension of a particle projection. See EMAN2/BoxSize for more.

• Make sure your input particles have been edgenormalized, and have the correct contrast (bright on a darker background). If not, correct all particle stacks with e2proc2dmulti.py with the --process=normalize.edgemean or --mult=-1 options.

• e2ctf.py --cs=4.1 --ac=15 --voltage=300 --apix=2.12 --oversamp=2 --autofit --gui ptcl/*hdf

• You may wish to manually check the CTF results quickly.
  • If you find a defocus is significantly off, adjust it manually to near the correct value and press 'refit'.
  • Be sure to press

• Compute a structure factor (you may opt to use a subset of the data instead): e2ctf.py --oversamp=2 --computesf ptcl/*hdf

• rerun : e2ctf.py --cs=4.1 --ac=15 --voltage=300 --apix=2.12 --oversamp=2 --autofit --gui ptcl/*hdf

• optionally: e2ctf.py --refinebysnr ptcl/*hdf

• generate phase-flipped and other files (using oversamp=1 is important here): e2ctf.py --oversamp=1 --phaseflip --phasefliphp --wiener --storeparm ptcl/*hdf

Reference-free Class-averages

Note that like EMAN1, these are used for initial evaluation of the data and initial model generation only. They are not used for final 3-D refinements.

• Get a list of the available wiener filtered stacks in bdb notation: e2bdb.py -s particles --filt=wiener

• Make a virtual-stack (like an EMAN1 .lst file) containing the particles you wish to use (don't have to be Wiener filtered) : e2bdb.py --makevstack=bdb:sets#stack_for_2d bdb:particles#dh3962_ctf_wiener bdb:particles#dh3965_ctf_wiener bdb:particles#dh3986_ctf_wiener bdb:particles#dh3997_ctf_wiener bdb:particles#dh4017_ctf_wiener

• Run 2-D refinement: e2refine2d.py --input=bdb:sets#stack_for_2d --iter=6 --ncls=24 --naliref=6 --nbasisfp=6 --parallel=thread:4

  • if generating a lot of classes (more than 100), use the --fastseed option

  • --parallel= option is common to many programs. Not all operations will run in parallel. See: EMAN2/Parallel

  • Many other options. This is just a representative example.

• Results will be in r2d_xx in a variety of database files. For the example above, final class-averages will be in bdb:r2d_01#classes_05

• Use e2display.py to look at the results.

Initial Model Generation

• If you don't already have an appropriate initial model EMAN2 can generate 1 (or more) for you.
• First, select a subset of the class averages. The selected averages should all be 'good' high contrast averages, and should represent as diverse a range of orientations as possible.

  • There are various ways of extracting good particles. The approach I use is to display the set of class-averages using e2display.py, middle-clicking for a control-panel, then using Del mode to delete the averages I don't want. When I'm done, I Save the results to a new file.

• Run the initial model generator: e2initialmodel.py

UNDER CONSTRUCTION