User:Benbrinkmann/Multiscale electrophysiology format

Multiscale Electrophysiology Format (MEF)
Multiscale Electrophysiology Format (MEF) was developed to handle the large amounts of data produced by large-scale electrophysiology in human and animal subjects. MEF can store any time series data up to 24 bits in length, and employs lossless range encoded difference compression. Subject identifying information in the file header can be encrypted using 128-bit AES encryption in order to comply with HIPAA requirements for patient privacy when transmitting data across an open network.

Compressed data is stored in independent blocks to allow direct access to compressed data, facilitate parallel processing, and limit the effects of potential damage to file. Data fidelity is ensured by 32-bit cyclic redundancy check in each compressed data block using the Koopman polynomial (0xEB31D82E), which has a Hamming distance of 4 up to 114 kbits.

A formal specification can be found here.

Source code:

http://mayoresearch.mayo.edu/mayo/research/msel/example_code.cfm

Links:

http://en.wikipedia.org/wiki/Range_encoding

http://en.wikipedia.org/wiki/AES_encryption

http://en.wikipedia.org/wiki/CRC-32

References:

Martin, GNN. Range encoding: an algorithm for removing redundancy from a digitised message. Video & Data Recoding Conference, Southampton, 1979.

Koopman, P. 32-Bit Cyclic Redundancy Codes for Internet Applications. The International Conference on Dependable Systems and Networks (June 2002). 459.

Brinkmann, BH et al. Large-scale electrophysiology: acquisition, compression, encryption, and storage of big data. Journal of Neuroscience Methods 180 (2009) 185–192.