Question: I'd like to process a really big data set with EMAN. Is it up to the task ?
Answer: Gosh, I hope so! Seriously, there are a few issues to be aware of. First, one factor often of concern is the fact that EMAN generally keeps the entire set of particles in a single image file stack A really big reconstruction might cause this file to become bigger than 2 gigabytes, which is a problem for some parts of certain operating systems and a lot of other software packages. While EMAN does the proper things to support files larger than 2G, sometimes the underlying system still won't allow it. However, there is a good workaround. EMAN supports a special file format called the LST format. This format is basically a text file containing a list of images in other files. For example, if you have too much data to put into a single start.hed/img file, do the following:
- INSTEAD of combining your micrograph images into a single file, make a subdirectory called 'raw' where you intend to run 'refine'.
- move all of your hed/img files into this subdirectory
- cd back to the 'refine' directory
- lstcat.py start.hed raw/*.hed
- ln -s start.hed start.img
You now have 2 files (start.hed/img) which appear to eman to be equivalent to a big imagic file, without any of the normal 2G limitations. Note, however, that you cannot add new images to this file with proc2d. It isn't really an image file.
There are other issues. Some are of concern when there are a lot of pixels in each image and others are of concern when there are a lot of images. In the first case, memory on the computer is the biggest problem. For example, if you were trying to reconstruct a 512x512x512 volume, each volume dataset requires 512 MB. Several programs require enough memory for 2 or 3 3D models. So any machine used to process this dataset would need to have at least 2 GB of RAM. There are too many issues involved to cover all possibilities here. In general, I'd say yes, EMAN can handle really big problems. If you run into problems, email me, and I'll try to help you resolve them.