Q: How do I choose the correct arguments for the amask argument in refine? I see that it ultimately is used to run the amask2 argument in proc3d, but I'm still a little unsure as do how to choose the parameters. I take it that the 'r' is the approximate radius where the density of the molecule falls off, and thr is the value of ...? And what is N? Thank you.
A: Trial and error. This is explained a bit in the proc3d manual page, but here it is again rewritten. amask= is very much like a flood-fill (the paint-bucket icon) in most 2-D paint programs. The idea is you start with some point, then fill in anything touching that point above some threshold value. Since it's annoying to specify a specific point in 3-D, and the maps are centered, we start in the middle of the map. However, sometimes maps are hollow in the middle, so the first parameter is the radius of a sphere centered on the origin that's big enough to touch some part of your structure. It should definitely NOT be so large that it extends outside your structure at any point. It just needs to be large enough to touch your structure somewhere.
The second parameter is an isosurface threshold for flood filling. ie - if you display your map in Chimera or some other isosurface visualization program, this is the threshold you use to display your map so it looks nice. Typically, since you want to have some flexibility in how you display the map, you will actually want to use a somewhat lower value than this. ie - the lowest value you would possibly use when presenting the map. So long as any noise outside the structure isn't connected to the structure itself at the threshold you use, you're fine.
Since we don't want to put a sharp edged mask on our structure at an isosurface value this high, the 3rd parameter allows us to extend the mask we've constructed by building 1-voxel wide shells outside the existing mask. If you say 10 for the final number, it will add 10 shells to the mask. An important point is that in these added shells decay gradually (a Gaussian) to the background value. This prevents the masking from exaggerating the resolution of your map when used in an even/odd test. Typically the 3rd value should be roughly the box size/10 (in pixels).