= EMAN2 Output Files =

This page documents all of the various files produced by various tasks and workflows in EMAN2. While the format of the actual files will
be one of the standard EMAN2 supported image formats in most cases, these pages will explain the contents of files with specific standard names.

== General File Information ==
 * [[Eman2DataStorage|File formats & data storage conventions]]
 * [[Eman2Metadata|Parameters/Metadata appearing in EMData attributes (and stored on disk with the image)]]

== Information on specific input/output files for different EMAN2 programs ==

== For Single Particle Analysis (SPA) ==

=== For new-style SPA refinement (e2spa_refine.py, e2spa_refine_multi.py is similar) ===
Older EMAN2 refinements made use of 2-D classification as part of the 3-D refinement process. Thanks to higher contrast data from direct detectors, the new refinement strategy determines the orientation of individual particles directly, so the results are much easier to interpret.

Each run will create a numbered ''r3d_XX'' folder.
Input files:
 * ''--ptcls'' specifies the input particles in .lst format. This input file will become ''ptcls_00.lst'' in the output folder
 * ''--ref'' specifies the initial model to start the refinement. This will be phase randomized beyond the initial resolution two times to produce ''threed_00_even.hdf'' and ''threed_00_odd.hdf''
 * The full set of input parameters from the command-line (or assumed defaults) will be saved to ''0_spa_params.json'' (https://blake.bcm.edu/emanwiki/Eman2JSStorage)

Output files (in temporal order of creation):
 * ''ptcls_XX.lst'' - LST file containing the particles and their orientations in the current iteration. Functionally similar to a STAR file in Relion, but with other benefits, since LST files can be treated as images. The orientation metadata will be treated as part of the image header when read. See [[EMAN2/ImageFormats]] for details. 
 * ''threed_XX.hdf'' - final 3-D map for iteration XX
 * ''threed_XX_even.hdf'' and ''threed_XX_odd.hdf'' - independent even/odd maps ("gold standard" refinement). These are masked and filtered.
 * ''threed_even_unmasked.hdf'' and ''threed_odd_unmasked.hdf'' - unmasked/unfiltered even/odd maps for the last completed iteration. These are overwritten after each iteration completes
 * ''fsc_masked_XX.txt'' - Fourier shell correlation curve between even/odd maps with a conservative soft mask applied. This can be used for a good, conservative resolution estimate.
 * ''fsc_maskedtight_XX.txt'' - Similar, but with a more aggressive mask, similar to Relion post-processing. This is conventionally what people would use in publication.
 * ''fsc_unmasked.txt_XX'' - Fourier shell correlation curve between even/odd maps with no applied mask. This is sometimes viewed as the most "conservative" resolution curve, however it is also susceptible to box size issues. ie - this curve will become worse if a larger box size is used, even though the structure will generally be better.
 * ''fscvol_xx.hdf'' - only present if local filtration is used in refinement. This map estimates the local resolution at each point in space. It can be used in Chimera to color the surface of the ''threed_XX.hdf'' file to highlight regions with better/worse resolution.
 * ''mask.hdf'' and ''masktight.hdf'' - mask files used in the last complete iteration of refinement 

=== For SPA refinement (single model - e2refine_easy.py and multimodel - e2refinemulti.py) ===
Input files:
 * '''files specified via ''--input'', ''--model'' for ''e2refine_easy'' '''
 * '''files specified via ''--input'' and ''--model'' or ''--models'' '''
 * '''strucfac.txt should normally be present in the project directory''' - This is a text file containing the ideal 1-D structure factor expected for the final map. Intensity as a function of spatial frequency.

Output files (in temporal order of creation), _xx denotes the iteration number:
 * '''projections_xx.hdf, proj_stg1_xx.hdf''' - [[EMAN2/ProjectionFiles|Map projection files]]
 * '''simmx_xx.hdf, simmx_stg1_xx.hdf, proj_simmx_xx.hdf''' - [[EMAN2/SimmxFiles|Similarity matrix image files]]
 * '''classmx_xx.hdf''' - [[EMAN2/ClassmxFiles|Classification matrix image files]]
 * '''cls_result_xx.hdf''' - [[EMAN2/ClsResultFiles|Class-Averaging Results matrix image files]]
 * '''classes_xx.hdf''' - [[EMAN2/ClassesFiles|Class-averages]]
 * '''threed_even_unmasked.hdf, threed_odd_unmasked.hdf''' - Reconstructions from the final completed iteration without masking or final filtration, suitable for use with ResMap
 * '''threed_xx.hdf, threed_filt_xx.hdf, threed_mask_xx.hdf''' - [[EMAN2/ThreedFiles|3-D reconstructions]]

If you are struggling with a failed refinement, look at the produced files in this order until you find something unexpected, and that may give some clues as to what went wrong. Don't be shy about posting to the Google Group for help!

=== Particle quality assessment ===
The program ''e2evalrefine.py --evalptclqual'' is normally used to assess the quality of individual particles after running ''e2refine_easy''. This produces a file in the Project Folder called '''ptclfsc_XX.txt''' where XX is the number of the refine_XX folder. This file is a multicolumn text file with the following contents. It can be displayed with ''e2display --plot'' :

 * Integrated particle/projection FSC from 100 - 30 A
  * For even marginally good data, this should always be positive, and have a pretty good spread between 0 and 1
 * Integrated particle/projection FSC from 30 - 15 A
  * If there is any hope of subnanometer resolution, this should also be mostly positive and have a good correlation with the first column
 * Integrated particle/projection FSC from 15 - 8 A
  * For _fullres or _lp5 data suitable for sub-5A resolution, you should see some correlation here with column 0, and the values should have a clear positive bias.
 * Integrated particle/projection FSC from 8 - 4 A
  * Even for very good _fullres data, this column is rarely useful. Normally it will look like a symmetric distribution about 0
 * "alt" Euler angle for this particle
 * "az" Euler angle
 * class number
 * defocus
 * particle number within file
 * file the particle is from
 * projection file number (optional)
 * projection file name (optional)

=== For SPA 2-D reference-free class-averaging (e2refine2d.py) ===
You may also wish to look at: [[EMAN2/Programs/e2refine2d|e2refine2d]]
Input files:
 * '''file specified via ''--input'' '''

Output files:
 * '''input_fp''' - rotational/translational invariants for each particle
 * '''input_fp_basis''' - MSA basis vectors (images) from input_fp
 * '''input_fp_basis_proj''' - MSA subspace projections of the input_fp invariants 
 * '''classmx_00''' - Initial classification of particles, same format as [[EMAN2/ClassmxFiles|classmx]] above
 * '''classes_init''' - Initial set of class-averages from invariant method (not very good usually)
 * '''allrefs_XX''' - All of the references (sorted and aligned) to be used for the current iteration. Other than sorting/alingment, same as classes_XX files
 * '''basis_XX''' - MSA basis from allrefs_xx
 * '''aliref_XX''' - Subset of allrefs used for alignment of raw particles
 * '''simmx_XX''' - Similarity matrix in same format as [[EMAN2/SimmxFiles|simmx]] above
 * '''input_XX_proj''' - Aligned particles projected into '''basis_XX''' subspace
 * '''classmx_XX''' - Classification matrix for the current iteration (as above)
 * '''classes_XX''' - Class averages at the end of the iteration. The highest numbered classes_XX file is the final output of the program


== For Single Particle Tomography (SPT, "subtomogram averaging") ==
 
=== Single stage alignment of subtomograms to a reference (e2spt_align.py) ===
Output Files:
 * '''align_ref.hdf''' - contains 2 volumes representing the even and odd references respectively. If --goldstandard isn't specified, then the two references will be identical
 * '''particle_parms.json''' - contains metadata for each subtomogram indexed by (filename,number) tuple
 * '''aliptcls.hdf''' - Requires specifying ''--saveali''. Stack of aligned particles. Note that particles are not split even/odd even if --goldstandard is used, which differs from e2spt_classaverage. These particles are for user-use only, they are not normally used in generating averages.


=== For SPT initial model generation by hierarchical ascendant classification (HAC, e2spt_hac.py) ===

=== For SPT initial model generation by binary tree alignment (BTA, e2spt_binarytree.py) ===

=== For SPT initial model generation by self symmetry alignment (SSA, e2symsearch3d.py) ===

=== For SPT iterative refinement runs (e2spt_classaverage.py) ===

Input files:
 * ''' ''--input'', subvolume stack in .hdf format'''
 * ''' ''--ref'', if performing reference-based refinement, reference image in .hdf format'''

Output files:
 * '''sptbt''', '''spthac''' or '''sptssa''' subdirectories if ''--ref'' is not provided and the program generates initial models automatically. The corresponding directory will appear, with the files specified above in it, depending on whether ''--btref'', ''--hacref'' or ''--ssaref'' are specified.

 * '''aliptcls.hdf''' - Requires specifying ''--saveali''. Stack of final aligned particles from last refinement iteration. If gold-standard refinement is on (by not supplying --goldstadardoff) the output includes '''aliptcls_even.hdf''' and '''aliptcls_odd.hdf'''.
 
 * '''avgs.hdf''' - Requires specifying ''--savesteps''. Stack of the averages produced in all iterations of refinement. If gold-standard refinement is on (by not supplying ''--goldstadardoff'') '''avgs.hdf'''; will be the averages of the odd and even averages for each iteration, and additional'''avgs_even.hdf''' and '''avgs_odd.hdf''' stacks will be generated containing the even and odd averages across iterations.
 
 * '''classmx_XX.hdf'''  - Classification matrix for the current iteration (as above)
 
 * '''final_avg.hdf''' - Final average of the final even and odd averages (or simply the final average of all the particles if gold-standard refinement is off).
 
 * '''fsc_XX.txt''' - FSC between even and odd averages for the current iteration.
 
 * '''initialrefs_fsc.txt''' - Initial FSC curve against the reference or between the initial even and odd models generated by the program if not reference is provided via ''--ref''.
 
 * '''parameters_sptclassavg.txt''' - Text file containing the executed command and the values used by the program for all parameters (including defaulted ones, not specified by the user).
 
 * '''spt_cccs_XX.txt''' - Text file with sorted cross correlation coefficients. If gold-standard refinement is on, '''spt_cccs_XX_even.txt''' and '''spt_cccs_XX_odd.txt''' are generated.
 
 * '''spt_meanccc.txt''' - Text file containing the mean cross correlation coefficient across iterations (this can help determine whether the mean score is improving or has plateaued or is degenerating). If gold-standard refinement is on, '''spt_meanccc_even.txt''' and  '''spt_meanccc_odd.txt''' are generated.
 
 * '''tomo_xforms_0.json''' - Json file with alignment parameters for all particles in the stack.
 
 * '''tomo_xforms_0_avgali2ref.json''' - Json file with alignment parameters for the final average aligned to the reference if gold-standard refinement is off. This becomes '''tomo_xforms_0_oddali2even.json''' if gold-standard refinement is on.

=== For SPT multiple model refinement (e2spt_refinemulti.py) ===