JSON Files
(BDB Replacement)
JSON files replace the much despised BDB database mechanism for storing metadata in EMAN2. These files offer a number of advantages over BDB, but there are also a few tradeoffs.
Advantages:
- Human-readable, and human-editable.
- Can be renamed, deleted, copied, etc, just like any other file.
Standard file format, a subset of JavaScript, so interfaces easily with the web.
Persistance & threadsafety. Through use of file-locking it should be safe to read a single JSON file from multiple threads/processes.
Tradeoffs:
- Speed. Making any change to a JSON file requires re-writing the entire file. If writing is deferred, then other processes won't see the changes until the write actually happens.
- Speed. Not really designed for very large 'databases' of information. While you could have a dictionary with 1,000,000 in it, deferred writing would be critical to maintaining any sort of performance.
- Not designed for images. While it is possible to store images and/or any other (pickleable) object in a JSON file, since opening the file results in reading the ENTIRE file, storing a large stack of images in JSON format is not advisable. Use HDF instead when you need to store images.
- While technically, multiple writers should be safe, if multiple processes are all writing to .js files at high speed, it is conceivable that there could be some corruption. We have not yet ever observed this happening, but it isn't completely impossible.
Basic usage
Usage is quite similar from python (intentionally) to BDB. The basic methods for accessing JSON files:
The main object is the JSDict class. An instance of this class represents a single file on disk with a '.js' extension.
js_open_dict(filename)
- Opens a JSON file as a dict-like database object (JSDict). Writes to JDB dictionaries are inefficient unless deferred writing is used. Default behavior is to write the entire dictionary to disk when any element is changed. File locking is attempted to avoid conflicts, but may not work in all situations. This mechansim should be While it is possible to store images in JSON files it is not recommended due to inefficiency, and making files which are difficult to read.
js_close_dict(filename)
- This does not need to be called explicitly, but will free some resources associated with the database. Not associated with closing a file pointer.
js_remove_dict(filename)
- closes and deletes a database using the same specification as db_open_dict. Unlike BDB functions this will actually remove the associated file on disk.
js_check_dict(filename, readonly=True)
- Checks for the existence of the named JSON file and insures that it can be opened for reading [and writing]. It does not check the contents of the file, just for its exsistence and permissions.
js_list_dicts(path)
- Gives a list of readable json files at a given path.