Diff for "EMAN2/Eman2Metadata"

Differences between revisions 7 and 31 (spanning 24 versions)

Parameters/Metadata stored in EMData Objects

The EMData object, and its representation on disk in the BDB local database, XML files, and when serialized in Python using 'pickle', supports the concept of arbitrary header parameters also known as metadata. This metadata are key/value pairs. The keys are always simple ascii text, and the values may be virtually anything represented as an EMObject, including simple ints, floats and strings, as well as more complicated objects such as Transform classes. If an EMData object is stored in some fashion other than the 3 mechanisms above, some loss of metadata is almost guaranteed. The file i/o objects will try to preserve some of the basic metadata, but most of the cryoEM formats simply don't support arbitrary header-data. The EMAN2 convention is to use BDB for most internal purposes, and to use HDF5 for data transfer/exchange.

This page will serve as a repository for the officially supported parameter key/value pairs. While you are free to set any metadata keys/values you like in an EMData object, the names listed here may be interpreted in specific ways by specific modules within EMAN, so it would be unwise to abuse them. Also, if you make up your own name for some purpose, it wouldn't hurt to register it here in the 'unofficial' section, to avoid it being used by someone else for a different purpose.

Special tags (read-only for getting image info)

These values are cached and only recomputed if the image changes

nx,ny,nz	int	Dimensions of the image, also available as get_xsize(),etc. Note: Assignments such as e["nx"] = 30 are an (not preferred) alternative to set_size(nx,ny,nz)
minimum	float	Smallest value in the image
maximum	float	Largest value in the image
mean	float	The average pixel value in the image
sigma	float	The standard deviation of the pixel values in the image
square_sum	float	Sum of the squares of the pixel values
mean_nonzero	float	The mean value of all nonzero pixels
sigma_nonzero	float	The standard deviation of the pixels ignoring pixels which are zero
is_complex	int	Flag indicating that the image is complex (R/I or A/P pairs)
is_complex_ri	int	Flag indicating that a complex image is R/I not A/P
changecount	int	An integer which is incremented every time the image is marked as changed
data_path	string	Used only in BDB files, to indicate that the binary data for an image should be read from an alternate location. Data cannot be written back to such objects.

These values are computed on the fly

skewness	float	Skewness of the pixel values
kurtosis	float	Kurtosis of the pixel values
median	float	Median value of the pixel values
nonzero_median	float	Median value of nonzero pixels

Official tags (used in EMAN2/SPARX as distributed):

apix_x,y,z	float	Angstroms per pixel on the x-axis (also _y and _z). If _y or _z are 0 or not present it will be assumed that they are the same as _x. Note that CTF objects have an independent value for A/pix
class_id	int	Set by classification routines to indicate which class number the particle is in
class_ptcl_src	string	In a class-average, this is the file containing the raw images used to create the average
class_ptcl_idxs	tuple	In a class-average, this is a list of particle numbers used in the final average (see class_ptcl_src)
ctf	Ctf subclass	A subclass of Ctf containing all CTF parameters
ctf_phase_flipped	bool	Set to 1 if the CTF phases have been flipped
eigval	float	Eigenvalue, only set for images which represent Eigenvectors
exc_class_ptcl_idxs	tuple	In a class-average, this is a list of particle numbers provided to the averager, but excluded from the final average (see class_ptcl_src)
ptcl_repr	int	If an image/volume represents the combination of one or more other images, this is the count of the number of particles that went into the average
ptcl_helix_coord	tuple	The two endpoints and a box width that defines a helix box (x1, y1, x2, y2, box_width)
ptcl_source_coord	tuple	The central coordinate of a boxed particle in terms of its source image, normally (x,y), may be (x,y,z) for subtomograms
ptcl_source_image	string	The name of the image from which the particle was extracted. Full path, may be in bdb syntax
reconstruct_norm	float	Normalization factor applied to a single projection/class-average during reconstruction
reconstruct_qual	float	Quality of a single projection/class-average relative to others during reconstruction. Unlike with comparators, larger values are better.
reconstruct_preproc	bool	Set if the image has been preprocessed for use with a reconstructor
render_min,max	float	Used when rendering an image to 8/16 bit integers. These are the values representing the minimum and maximum integer values
segment_centers	float list	Used when a volume has been segmented into regions. Set of 3*nregions floats in x1,y1,z1,x2,y2,z2,... order, indicating the center of each region as defined by the specific algorithm
subvolume_x0,y0,z0	int	Used when the EMData stores only a portion of a larger image in certain contexts (notably direct Fourier inversion. This represents the location of the origin of 'this' in the larger virtual volume
subvolume_full_nx,ny,nz	int	Used with subvolume_x0,... Specifies the size of the virtual volume that 'this' is a part of
threed_ptcl_src	string	In a 3-D map, this is the file containing the raw images used to create the reconstruction
threed_ptcl_idxs	tuple	In a 3-D map, this is a list of particle numbers used in the final reconstruction (see threed_ptcl_src)
threed_excl_ptcl_idxs	tuple	In a 3-D map, this is a list of particle numbers excluded from the final map (see threed_ptcl_src)
xform.projection	Transform	A Transform object used by Projectors. It is applied to a 3-D model prior to projecting by summing along Z. The inverse of this Transform is used by Reconstructors
xform.align2d	Transform	A Transform object representing a 2-D transformation used to align this EMData object to a reference in 2-D
xform.align3d	Transform	A Transform object representing a 3-D transformation used to align this (3-D) EMData object to a (3-D) reference

Proposed new Official tags (comments welcome):

apix_scan	float	Scan pixel size in Angstroms	(there are already parameters for apix_x/y/z above. Is this different in some way ?, you could provide this as supplementary information, but apix_x/y/z remain the 'official' values. Is this what you want ?)
box_location	int_array	4 values, x0, y0, xsize, ysize representing the location of the particle in the original (box_source) image	(Pawel suggested this be changed to the reduced image, because normally we first reduce the micrograph, the window it.) (No, the box location should be in the coordinates of the referenced 'parent' image. If you want to reference a reduced image you can, but I think it makes a lot more sense to provide the capability of refining the coordinates from the reduced image when you return to the original image.) Pawel: the coordinates in boxer refer to the image from windowing was done. This is is (or can be) a reduced micrograph. There is more confusion here: name apix_scan suggests this is pixel size of the scan, NOT THE REDUCED MICROGRAPH FROM WHICH PARTICLES WERE WINDOWED. Incidentally, both pixel sizes are needed and have to be in the header. Steve: This differs from the philosophy of e2boxer, which is that you always provide e2boxer with the original image, it may internally downscale it for boxing, but the final box positions are in terms of the original image. I don't understand why you would want to externally downscale, then have to rescale the coordinates again later ? This is very messy. Anyway, any single image has a particluar A/pix value associated with it. I have no objections to something like apix_original, but what exactly is the point ? You aren't likely to rescale reduced images back to their original size...
box_score	float	A value representing the relative quality (meaning may vary) of this particle compared to others
box_source	string	Filename (not full path) of the image this particle was extracted from
box_source_id	string	Other (database) identifier of the image the raw data came from if available

Unofficial tags (To prevent reuse, used by someone in their own code or for testing):

match_n	int	used to represent the number of a reference particle this particle best matched
match_qual	float	used to represent the quality associated with match_n, smaller is a better match

-  ⇤ ← Revision 7 as of 2008-10-02 17:11:16 → 
  Size: 5378
  Editor: SteveLudtke
  Comment:
+   ← Revision 31 as of 2010-02-12 23:17:53 → ⇥
  Size: 9015
  Editor: RossColeman
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 8:
-||nx,ny,nz||int||Dimensions of the image, also available as get_xsize(),etc.||
||minimum||float||Smallest value in the image||
||maximum||float||Largest value in the image||
||mean||float||The average pixel value in the image||
||sigma||float||The standard deviation of the pixel values in the image||
||square_sum||float||Sum of the squares of the pixel values||
||mean_nonzero||float||The mean value of all nonzero pixels||
||sigma_nonzero||float||The standard deviation of the pixels ignoring pixels which are zero||
||is_complex||int||Flag indicating that the image is complex (R/I or A/P pairs)||
||is_complex_ri||int||Flag indicating that a complex image is R/I not A/P||
||changecount||int||An integer which is incremented every time the image is marked as changed||
+||nx,ny,nz ||int ||Dimensions of the image, also available as get_xsize(),etc. Note: Assignments such as e["nx"] = 30 are an (not preferred) alternative to set_size(nx,ny,nz) ||
||minimum ||float ||Smallest value in the image ||
||maximum ||float ||Largest value in the image ||
||mean ||float ||The average pixel value in the image ||
||sigma ||float ||The standard deviation of the pixel values in the image ||
||square_sum ||float ||Sum of the squares of the pixel values ||
||mean_nonzero ||float ||The mean value of all nonzero pixels ||
||sigma_nonzero ||float ||The standard deviation of the pixels ignoring pixels which are zero ||
||is_complex ||int ||Flag indicating that the image is complex (R/I or A/P pairs) ||
||is_complex_ri ||int ||Flag indicating that a complex image is R/I not A/P ||
||changecount ||int ||An integer which is incremented every time the image is marked as changed ||
||data_path ||string ||Used only in BDB files, to indicate that the binary data for an image should be read from an alternate location. Data cannot be written back to such objects. ||
-Line 21:
+Line 23:
-||skewness||float||Skewness of the pixel values||
||kurtosis||float||Kurtosis of the pixel values||
||median||float||Median value of the pixel values||
||nonzero_median||float||Median value of nonzero pixels||
+||skewness ||float ||Skewness of the pixel values ||
||kurtosis ||float ||Kurtosis of the pixel values ||
||median ||float ||Median value of the pixel values ||
||nonzero_median ||float ||Median value of nonzero pixels ||
-Line 27:
+Line 30:
+||apix_x,y,z ||float ||Angstroms per pixel on the x-axis (also _y and _z). If _y or _z are 0 or not present it will be assumed that they are the same as _x. Note that CTF objects have an independent value for A/pix ||
||class_id ||int ||Set by classification routines to indicate which class number the particle is in ||
||class_ptcl_src ||string ||In a class-average, this is the file containing the raw images used to create the average ||
||class_ptcl_idxs ||tuple ||In a class-average, this is a list of particle numbers used in the final average (see class_ptcl_src) ||
||ctf ||Ctf subclass ||A subclass of Ctf containing all CTF parameters ||
||ctf_phase_flipped ||bool ||Set to 1 if the CTF phases have been flipped ||
||eigval ||float ||Eigenvalue, only set for images which represent Eigenvectors ||
||exc_class_ptcl_idxs ||tuple ||In a class-average, this is a list of particle numbers provided to the averager, but excluded from the final average (see class_ptcl_src) ||
||ptcl_repr ||int ||If an image/volume represents the combination of one or more other images, this is the count of the number of particles that went into the average ||
||ptcl_helix_coord ||tuple ||The two endpoints and a box width that defines a helix box (x1, y1, x2, y2, box_width) ||
||ptcl_source_coord ||tuple ||The central coordinate of a boxed particle in terms of its source image, normally (x,y), may be (x,y,z) for subtomograms ||
||ptcl_source_image ||string ||The name of the image from which the particle was extracted. Full path, may be in bdb syntax ||
||reconstruct_norm ||float ||Normalization factor applied to a single projection/class-average during reconstruction ||
||reconstruct_qual ||float ||Quality of a single projection/class-average relative to others during reconstruction. Unlike with comparators, larger values are better. ||
||reconstruct_preproc ||bool ||Set if the image has been preprocessed for use with a reconstructor ||
||render_min,max ||float ||Used when rendering an image to 8/16 bit integers. These are the values representing the minimum and maximum integer values ||
||segment_centers ||float list ||Used when a volume has been segmented into regions. Set of 3*nregions floats in x1,y1,z1,x2,y2,z2,... order, indicating the center of each region as defined by the specific algorithm ||
||subvolume_x0,y0,z0 ||int ||Used when the EMData stores only a portion of a larger image in certain contexts (notably direct Fourier inversion. This represents the location of the origin of 'this' in the larger virtual volume ||
||subvolume_full_nx,ny,nz ||int ||Used with subvolume_x0,... Specifies the size of the virtual volume that 'this' is a part of ||
||threed_ptcl_src ||string ||In a 3-D map, this is the file containing the raw images used to create the reconstruction ||
||threed_ptcl_idxs ||tuple ||In a 3-D map, this is a list of particle numbers used in the final reconstruction (see threed_ptcl_src) ||
||threed_excl_ptcl_idxs ||tuple ||In a 3-D map, this is a list of particle numbers excluded from the final map (see threed_ptcl_src) ||
||xform.projection ||Transform ||A Transform object used by Projectors. It is applied to a 3-D model prior to projecting by summing along Z. The inverse of this Transform is used by Reconstructors ||
||xform.align2d ||Transform ||A Transform object representing a 2-D transformation used to align this EMData object to a reference in 2-D ||
||xform.align3d ||Transform ||A Transform object representing a 3-D transformation used to align this (3-D) EMData object to a (3-D) reference ||
-Line 28:
+Line 56:
-||apix_x,y,z||float||Angstroms per pixel on the x-axis (also _y and _z)||
||xform.projection||Transform||A Transform object used by Projectors. It is applied to a 3-D model prior to projecting by summing along Z. The inverse of this Transform is used by Reconstructors||
||xform.align2d||Transform||A Transform object representing a 2-D transformation used to align this EMData object to a reference in 2-D||
||xform.align3d||Transform||A Transform object representing a 3-D transformation used to align this (3-D) EMData object to a (3-D) reference||
||ptcl_repr||int(float?)||If an image represents the average of one or more other images, this is the count of the number of particles that went into the average||
||class_id||int||Set by classification routines to indicate which class number the particle is in||
||eigval||float||Eigenvalue, only set for images which represent Eigenvectors||
||render_min,max||float||Used when rendering an image to 8/16 bit integers. These are the values representing the minimum and maximum integer values||
-Line 39:
+Line 60:
-||apix_scan||float||Scan pixel size in Angstroms||
||box_location||intarray||4 values, x0, y0, xsize, ysize representing the location of the particle in the original (box_source) image (Pawel suggested this be changed to the reduced image, because normally we first reduce the micrograph, the window it. -> No, the box location should be in the coordinates of the referenced 'parent' image. If you want to reference a reduced image you can, but I think it makes a lot more sense to provide the capability of refining the coordinates from the reduced image when you return to the original image.)||
||box_score||float||A value representing the relative quality (meaning may vary) of this particle compared to others||
||box_source||string||Filename (not full path) of the image this particle was extracted from||
||box_source_id||string||Other (database) identifier of the image the raw data came from if available||
||microscope_voltage||float||Microscope voltage in kV (shall we change it to just '''voltage''' to be same as sparx, what else voltage it could be?) (we are trying to define metadata dictionaries which will conflict minimally with other dictionaries. If all of the microscope parameters are prefixed by microscope_ it helps identify their source.)||
||microscope_cs||float||Cs of microscope in mm (same thing, change to '''cs'''? or '''Cs'''?) (Parameter names are case-insensitive. Same argument for using 'microscope_')||
||ctf_defocus||float||Defocus in microns, underfocus positive||
+||apix_scan ||float ||Scan pixel size in Angstroms ||(there are already parameters for apix_x/y/z above. Is this different in some way ?, you could provide this as supplementary information, but apix_x/y/z remain the 'official' values. Is this what you want ?) ||
||box_location ||int_array ||4 values, x0, y0, xsize, ysize representing the location of the particle in the original (box_source) image ||(Pawel suggested this be changed to the reduced image, because normally we first reduce the micrograph, the window it.)     (No, the box location should be in the coordinates of the referenced 'parent' image. If you want to reference a reduced image you can, but I think it makes a lot more sense to provide the capability of refining the coordinates from the reduced image when you return to the original image.) Pawel: the coordinates in boxer refer to the image from windowing was done.  This is is (or can be) a reduced micrograph. There is more confusion here: name apix_scan suggests this is pixel size of the scan, NOT THE REDUCED MICROGRAPH FROM WHICH PARTICLES WERE WINDOWED. Incidentally, both pixel sizes are needed and have to be in the header. Steve: This differs from the philosophy of e2boxer, which is that you always provide e2boxer with the original image, it may internally downscale it for boxing, but the final box positions are in terms of the original image. I don't understand why you would want to externally downscale, then have to rescale the coordinates again later ? This is very messy.  Anyway, any single image has a particluar A/pix value associated with it. I have no objections to something like apix_original, but what exactly is the point ? You aren't likely to rescale reduced images back to their original size... ||
||box_score ||float ||A value representing the relative quality (meaning may vary) of this particle compared to others ||
||box_source ||string ||Filename (not full path) of the image this particle was extracted from ||
||box_source_id ||string ||Other (database) identifier of the image the raw data came from if available ||
-Line 50:
+Line 70:
+||match_n ||int ||used to represent the number of a reference particle this particle best matched ||
||match_qual ||float ||used to represent the quality associated with match_n, smaller is a better match ||
-Line 51:
+Line 73:
-|| || ||
|| || ||