EMAN 2 Reference Manual

EMAN 2 Reference Manual
INSERT NAMES
Contents
1 Introduction
1.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
6
6
2 Installation
2.1 Required Libraries / Programs . . . . . . . .
2.1.1 FFTW . . . . . . . . . . . . . . . . .
2.1.2 GSL - GNU Scientific Library . . . .
2.1.3 Boost . . . . . . . . . . . . . . . . .
2.1.4 CMake . . . . . . . . . . . . . . . .
2.2 Optional Libraries / Programs . . . . . . . .
2.3 Quick Installation . . . . . . . . . . . . . . .
2.4 Advanced Installation . . . . . . . . . . . . .
2.4.1 Platform Dependent Optimization . .
2.4.2 Generating the Latest Documentation
2.5 Notes for Developers . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
8
8
8
8
9
9
9
10
11
11
11
3 Developer’s Guide
3.1 Complete Reference Guide . . . . . . .
3.2 Development Policy FAQ . . . . . . . .
3.3 Coding Style . . . . . . . . . . . . . .
3.4 Using Factory Classes . . . . . . . . . .
3.5 Exceptions . . . . . . . . . . . . . . . .
3.5.1 Using Exceptions . . . . . . . .
3.5.2 Existing Exception Types . . .
3.5.3 Defining New Exception Classes
3.6 Using the Log Class . . . . . . . . . . .
3.6.1 Typical Usage . . . . . . . . . .
3.7 Adding Components . . . . . . . . . .
3.7.1 Processors . . . . . . . . . . .
3.7.2 Aligners . . . . . . . . . . . . .
3.7.3 Averagers . . . . . . . . . . . .
3.7.4 Image Comparator . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
13
13
13
17
18
19
19
20
20
21
21
22
22
24
24
24
1
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3.8
3.9
3.10
3.11
3.12
3.13
3.14
3.15
3.16
3.7.5 IO Functionality . . . . . . . . . . . . . . . .
3.7.6 Reconstructors . . . . . . . . . . . . . . . . .
Adding Documentation . . . . . . . . . . . . . . . . .
3.8.1 Document Standards . . . . . . . . . . . . . .
3.8.2 Removing Preambles . . . . . . . . . . . . . .
3.8.3 Editing TOC.tex . . . . . . . . . . . . . . . .
3.8.4 Adding LATEX Packages . . . . . . . . . . . . .
Image Header Attribute Naming Conventions . . . . .
Testing Framework . . . . . . . . . . . . . . . . . . .
3.10.1 Testing Guidelines . . . . . . . . . . . . . . .
3.10.2 Test Example . . . . . . . . . . . . . . . . . .
CVS Help . . . . . . . . . . . . . . . . . . . . . . . .
Miscellaneous . . . . . . . . . . . . . . . . . . . . . .
3.12.1 Reading/Writing Images in Python . . . . . . .
3.12.2 Processors Usage . . . . . . . . . . . . . . . .
3.12.3 Reconstructors Usage . . . . . . . . . . . . . .
3.12.4 Using Aligner, Comparators, and Projectors . .
3.12.5 Using Pyste . . . . . . . . . . . . . . . . . . .
3.12.6 Using FFTW . . . . . . . . . . . . . . . . . .
3.12.7 Large File I/O . . . . . . . . . . . . . . . . . .
3.12.8 Euler Angles . . . . . . . . . . . . . . . . . .
3.12.9 Using numpy . . . . . . . . . . . . . . . . . .
3.12.10 Using Transformations . . . . . . . . . . . . .
3.12.11 Printing Error/Warning/Debugging Information
3.12.12 Adding Testing Groups . . . . . . . . . . . . .
How to Install Boost Python . . . . . . . . . . . . . .
How to use your own version of python . . . . . . . .
How to Install numpy . . . . . . . . . . . . . . . . . .
Writing Python Scripts . . . . . . . . . . . . . . . . .
3.16.1 Logging . . . . . . . . . . . . . . . . . . . . .
3.16.2 Command-Line Argument Parsing . . . . . . .
3.16.3 Code Modularity . . . . . . . . . . . . . . . .
4 Program Manuals
4.1 e2align3d.py . . . .
4.1.1 Usage . . .
4.1.2 Description
4.1.3 Options . .
4.2 e2aligntest.py . . .
4.2.1 Usage . . .
4.2.2 Description
4.2.3 Options . .
4.3 e2boxer.py . . . . .
4.3.1 Usage . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
24
24
24
25
25
25
25
25
27
27
27
29
30
30
31
31
31
32
32
33
33
33
34
34
34
35
36
36
36
36
37
37
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
38
38
38
38
38
38
38
38
39
39
39
4.4
4.5
4.6
4.7
4.8
4.9
4.10
4.11
4.12
4.13
4.14
4.3.2 Description
4.3.3 Options . .
e2cmmtomrc.py . .
4.4.1 Usage . . .
4.4.2 Description
4.4.3 Options . .
logfile . . . . . . .
4.5.1 Usage . . .
4.5.2 Description
4.5.3 Options . .
e2foldfitter.py . . .
4.6.1 Usage . . .
4.6.2 Description
4.6.3 Options . .
e2make3d.py . . .
4.7.1 Usage . . .
4.7.2 Description
4.7.3 Options . .
e2pdb2mrc.py . . .
4.8.1 Usage . . .
4.8.2 Description
4.8.3 Options . .
e2proc2d.py . . . .
4.9.1 Usage . . .
4.9.2 Description
4.9.3 Options . .
e2proc3d.py . . . .
4.10.1 Usage . . .
4.10.2 Description
4.10.3 Options . .
e2project3d.py . .
4.11.1 Usage . . .
4.11.2 Description
4.11.3 Options . .
e2scannereval.py .
4.12.1 Usage . . .
4.12.2 Description
4.12.3 Options . .
e2symbest.py . . .
4.13.1 Usage . . .
4.13.2 Description
4.13.3 Options . .
e2tomogram.py . .
4.14.1 Usage . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
39
39
40
40
40
40
40
40
40
40
41
41
41
41
41
41
41
41
42
42
42
43
43
43
43
43
44
44
45
45
45
45
45
45
46
46
46
47
47
47
47
47
47
47
4.14.2 Description . . . . . . . . . . . . . . . . . . . . . . . . .
4.14.3 Options . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
47
48
List of Tables
3.1
Image Attribute Naming . . . . . . . . . . . . . . . . . . . . . .
27
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
4.10
4.11
4.12
4.13
4.14
e2align3d.py Options . .
e2aligntest.py Options .
e2boxer.py Options . . .
e2cmmtomrc.py Options
logfile Options . . . . .
e2foldfitter.py Options .
e2make3d.py Options . .
e2pdb2mrc.py Options .
e2proc2d.py Options . .
e2proc3d.py Options . .
e2project3d.py Options .
e2scannereval.py Options
e2symbest.py Options . .
e2tomogram.py Options
38
39
40
40
41
41
42
43
44
45
46
47
47
48
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Chapter 1
Introduction
1.1 Abstract
EMAN 2 is an open-source Software for Single Particle Analysis and Electron
Micrograph Analysis. EMAN2 is designed to handle both small, low-symmetry
single particles and large, high symmetry viruses. It uses more object-oriented
technologies for better modular design, has convenient and flexible interfaces to
promote extendibility, emphases easy and efficient use in Python, and many new
capabilities.
1.2 Goals
1. EMAN2 should have a generic way to read/write various electronic microscopy image formats.
• Support all electron-microscopy image formats supported in EMAN
• Read an arbitrary convex 2D/3D region of any image format
• Separate reading/writing of image headers from image data
• Support averaging and shrinking when reading images
2. EMAN2 should make it easy to use and develop image processors.
• It should be simple and easy to define a new filter in either C++ or
Python.
• Allow users to define new filters that accept arbitrary numbers of parameters in an arbitrary order
• Utilize existing filters defined in EMAN1 system.
• Filters should be able to be pipelined.
• Each filter is identified with a meaningful name
6
3. The same design goals for filters should also apply to EM image aligners,
image comparators, image averagers, image 3D reconstructors.
4. EMAN2 should have a generic way to handle image translation, rotation,
Euler angles. It should support basic geometry operations like vector and
matrix.
5. EMAN2 should allow different definitions of CTFs.
6. EMAN2 should have a modular design to support different FFTW library at
compile time.
7. EMAN2 should support data processing on an arbitrary convex region of an
image.
8. EMAN2 should support a logging mechanism similar to that in EMAN1.
9. EMAN2 should be designed for easy regression tests.
10. EMAN2 should be fully documented using Doxygen style.
11. EMAN2 should support elegant integration with Phoenix software:
• Support data interchange with Phoenix CCTBX.
• Implement basic image reconstruction tasks used in Phenix GUI environment.
7
Chapter 2
Installation
2.1 Required Libraries / Programs
The following libraries are required for EMAN2 installation (the libraries should
be installed as shared-object libraries where applicable):
2.1.1 FFTW
Version 2.1.3+
Available at: http://www.fftw.org/
To install fftw from source use either configure option:
% ./configure –enable-static=no –enable-shared=yes –enable-float –enable-typeprefix
OR
% ./configure –enable-shared=yes –enable-float
Followed by:
% make
2.1.2 GSL - GNU Scientific Library
Version: 1.3+
Available at: http://www.gnu.org/software/gsl/
Installation is very straight forward:
% ./configure
% make
2.1.3 Boost
Version: 1.32 or lower
Available at: http://www.boost.org
(NOTE: EMAN2 does not currently support Boost versions above 1.32.)
8
1. Installing Boost requires Boost.Jam. Executables and source code for jam
can be found at the Boost website.
Installing Boost requires the user to identify a particular toolset to use during compliation. Most UNIX systems will probably use the ”gcc” toolset; visit
http://www.boost.org/more/getting started.html#Tools for a complete listing.
% bjam ”-sTOOLS=gcc” install
Header files from the Boost installation (located in the ”boost” subdirectory of
the Boost installation (ex. /boost 1 32 0/boost)) must now either be added to the
compilers path or copied into an existing location on the path in a subdirectory
/boost.
One possibility for this might be: % cp -r boost /usr/include/boost
2.1.4 CMake
Version: 2.0.6+
Available at: http://www.cmake.org
Executables for several platforms are available; source code can also be used
for custom installations.
2.2 Optional Libraries / Programs
- To read/write HDF5 image, use hdf5 (http://hdf.ncsa.uiuc.edu/HDF5). (NOTE:
HDF5 1.6.4 has some API compatibility issue and it doesn’t work with EMAN2
yet.)
- To read TIFF image, use libtiff (http://www.libtiff.org)
- To read PNG image, use PNG (http://www.libtiff.org)
For development the following libraries/programs are required (see 3.13 for
installation help):
- Python (version 2.2+) (http://www.python.org)
- Boost Python (version 1.32-) (http://www.boost.org)
- Numerical Python Numpy (version 22.0+) (http://www.pfdubois.com/numpy)
2.3 Quick Installation
Suppose you have source code eman2.tar.gz
1.
% cd $HOME
% mkdir -p EMAN2/src
% cd EMAN2/src
% gunzip eman2.tar.gz
% tar xf eman2.tar
9
2.
% mkdir build
% cd build
3.
% cmake ../eman2
% make
% make install
4. set up login shell
for csh/tcsh, put the following to your .cshrc or .tcshrc file:
setenv EMAN2DIR $HOME/EMAN2
setenv PATH $EMAN2DIR/bin:${PATH}
setenv LD LIBRARY PATH $EMAN2DIR/lib
setenv PYTHONPATH .:$HOME/EMAN2/lib
for bash in .bashrc add:
export EMAN2DIR=$HOME/EMAN2
export PATH=$PATH:$EMAN2DIR/bin
export LD LIBRARY PATH=$EMAN2DIR/lib
export PYTHONPATH=$PYTHONPATH:$HOME/EMAN2/lib
2.4 Advanced Installation
If your libraries (fftw, gsl, hdf, etc) are not found by Quick Installation, or if you
want to change the compilation options, the following steps help:
1. follow the first 2 steps in Quick Installation.
2. If your libraries are not installed at the default places, setup the related environment variables:
- fftw −→ FFTWDIR
- gsl −→ GSLDIR
- tiff −→ TIFFDIR
- png −→ PNGDIR
- hdf5 −→ HDF5DIR
- python −→ PYTHON ROOT and PYTHON VERSION
3. % ccmake ../eman2
- type ’c’ if it asks about ”CMAKE BACKWARDS COMPATIBILITY”.
10
- make necessary changes for compilation flags.
- developers will probably want to set BOOST-LIBRARY to a Boost.Python
object file (ex. libboost python-gcc-1 32.so)
- Then type ’c’, and type ’g’.
4.
% make
% make install
2.4.1 Platform Dependent Optimization
In CMake Configuration, enable the following option for your platform:
- Athlon: ENABLE ATHLON
- Opteron: ENABLE OPTERON
- Mac G5: ENABLE G5
2.4.2 Generating the Latest Documentation
1. Install doxygen (version 1.4.3+, http://www.doxygen.org)
2. Run “sh ./makedoc.sh” from the “EMAN2/src/eman2/” directory to generate
the documentation in the “EMAN2/src/eman2/doc/”.
2.5 Notes for Developers
1. For Emacs users, please add the following line to your $HOME/.emacs: (setq
default-tab-width 4)
2. Ensure Boost.Python is installed
3. EMAN2 uses Pyste (http://www.boost.org/libs/python/pyste/) to wrap C++
into python. Here is the way to install Pyste:
(a) get boost python.
% cd libs/python/pyste/install
% python setup.py install
(b) install elementtree
(c) install GCCXML
(d) for boost 1.32.0, apply a patch for PYSTE. (Contact EMAN2 developers for the patch.)
4. To generate new boost python wrapper, run
11
% cd eman2/libpyEm
% ./create boost python
5. Windows Installer EMAN uses ”Nullsoft Scriptable Install System” (http://nsis.sourceforge.net/)
to generate the windows installer. It also uses ”HM NIS Edit” (http://hmne.sourceforge.net/)
as the editor.
12
Chapter 3
Developer’s Guide
3.1 Complete Reference Guide
See the Complete Reference Guide (/html/index.html) for a full listing of all classes
and functions included in EMAN2.
3.2 Development Policy FAQ
Steve Ludtke, 11/26/2004
Introduction
This document summarizes the basic policy for EMAN2 development, especially
on how to contribute your code to EMAN2. The intened audiences are EMAN2
developers.
Q. What is the current eman2 development model? Anybody with cvs access
can change anything, or is there a “gatekeeper” who adds patches to the current
code, and thus has ultimate authority over what gets into the program? What I
mean is who decides whether a given change is desirable or not? Do you care if
something that you are philosophically opposed to gets added to the code?
A. There aren’t THAT many people with CVS access right now, so it isn’t a
major issue yet. As it stands now, anyone with CVS access is permitted to check
changes in. Certainly anyone is allowed to add completely new routines, like new
processors, new reconstruction routines, etc.
Q. Is there a policy about changing code submitted by somebody else? Similarly, is there anything that prevents (or facilitates) having several groups working
on the same piece of code.
A. CVS automatically allows people to work on the same code simultaneously.
If two people make conflicting changes (ie - change the same line of code), then
the person who checks their changes back into CVS last has to go through the
’merging’ process before CVS allows their changes to be committed. ie 13
1. cvs checkout by A and B
2. A and B make conflicting changes to a particular file
3. A checks his changes back in
4. B tries to check his changes in, gets a message saying the code has already
been changed, and he must resolve conflicts first
5. B does a cvs update, which will put BOTH versions of the changed code into
the appropriate file. B must then manually update the code to reflect a single
changed version.
6. B checks his changes in.
Note that, a record still exists of A’s changes, so if B changes something in a
bad way, it is still possible to check out A’s version. Regarding changing someone
elses code, I am certainly concerned about the issue of someone ’fixing’ something
and unintentionally breaking it. For now, please check with me before you spend
time changing an existing piece of code. If there is a good reason to leave it as it is,
you can simply make a different routine with the desired functionality. However,
if something is really broken, or can be substantially improved, I’m all in favor
of this happening. Just drop me a quick message explaining what you intend.
For example, the background fitting routine you described in this message was
never actually completed, and did not work, so putting in a functional version is a
welcome addition.
Q. What’s the policy of changing someone else’s code?
A. I agree in general that changing someone elses code should only be done
with permission of the original author. I would (for now) still like to act as a
clearinghouse for this sort of activity, because there may be issues that the original
author isn’t even aware of. It is always possible to find out who wrote a particular
routine through CVS, but this can be a bit annoying to accomplish. Right now we
put author stamps at the top of each source code file.
Q. Is it clear who wrote a particular piece of code?
A. It is always possible to find out who wrote a particular routine through CVS,
but this can be a bit annoying to accomplish. Right now we put author stamps at
the top of each source code file. Starting now, why don’t we set a policy that each
new function/method that is added to EMAN2 should have an author/date stamp
in the docstring for the method. I don’t think this needs to extend to small changes
within a routine. It is always possible to check CVS for detailed changes.
Q. User docs currently seem to be quite thin. Is that by design? Moreover,
do the auto-generated docs adequately allow developers to determine how and by
what a particular chunk of code is used. Also, is there a simple way to find out
whether a given procedure used by multiple pieces of code and if yes, by which
ones?
14
A. Well, there aren’t really any user docs yet because there aren’t really any
user programs yet. There are just a few that we’ve started writing over the last
few weeks. These should all be internally documented, ie e2pdb2mrc –help will
provide some documentation, but this needs to be improved, and we do need to
develop a script to collect this information into web pages. Right now our effort is
mostly focused on programming, so the docs have also focused that way.
Q. What is the reason for having monolithic source files (such as processor.cpp)
instead of having separate source files for each distinct piece of code? There are
advantages to both solutions...
A. Simply to prevent having too many source files scattered around. If every
Processor was in an individual file, there would be a tremendous number of files.
As any particular file gets too large, we generally split it into a few smaller pieces.
Since CVS takes care of the changes, and the code for a particular function can be
localized within the source file, there aren’t any major problems with this approach.
One problem with the individual file method is that it tends to dramatically increase
build and install times. This is a somewhat arbitrary policy.
Q. What is the policy on “stolen” code (such as that taken from Numerical
Recipes)?
A. Clearly taking obvious copyrighted code is a BAD thing, especially when
it has been made clear that such use is not permitted. However, I think NRC has
actually relaxed their polices a bit recently. In most cases equivalent publicly available code can be found. We have already adopted GSL (Gnu Scientific Library),
Opt++ and FFTW as dependencies. So routines from any of those toolkits can be
used freely. If there are other toolkits you find you need, you need to talk to me
before adding new dependencies to EMAN2 as a whole. More dependencies are
generally a bad thing.
Q. May one incorporate pure C (or even pure Fortran) routines, particularly
numerical routines, into eman2?
A. No Fortran. This is strictly due to the added compilation difficulties it poses.
Pure C is a bit trickier, I certainly have no objections to C-like C++, but to work in
the framework, the ’functions’ need to at least be defined as static methods in some
classes, for example, in Util class or EMUtil class.
Q. Is there a specification somewhere that distinguishes fatal from recoverable
errors?
A. We are trying to adopt try/except exception processing for errors. The details
for this are still being worked out. Fatal errors should be avoided if possible in the
libraries, but when they occur, we should have a central routine to log the error and
exit (EMAN1 did). This is in flux right now I’d say.
Q. Are operations on arrays / matrices defined anywhere in the code? Similarly,
why does the EMData class define an image as a low-level C chunk of memory,
with the user constantly performing on-the-fly index calculations?
A. Using C++ abstractions for the low level image storage (like an array of
arrays) generally imposes a substantial speed penalty. There IS a higher level abstraction for accessing the image data get value at, set value at, etc. These are
15
generally inlined functions, so the ones that don’t check limits are pretty much the
same speed as doing the index calculations yourself. Currently there aren’t any
routines for treating EMData objects as vectors/matrices/tensors, but this could be
done. Liwei did make a matrix class, but this is used strictly for transformation
matrices (3x3 or 4x4). GSL has a large set of matrix math routines. The main
reason we haven’t settled on anything is that we haven’t completely decided on a
particular linear algebra suite. We could use numpy array on the python side, but
this wouldn’t provide C++ functionality. We could use GSL, but then we need to
expose all of the appropriate routines on the python side, or we could decide to go
full bore and adopt something like Lapack. This is still open for debate...
Q. Is there a testing policy or framework?
A. We would like to have a full set of unit-tests, so every method in EMAN2
will have a corresponding test. We are working on this. Bugfixes are still a higher
priority right now. If you guys are actively working on EMAN2, I would strongly
suggest doing frequent CVS updates. Wen and I have begun writing user-level
programs now, and finding many bugs, which are rapidly getting fixed. Still, there
are probably quite a few problems.
Q. How does eman2 report results (such as a goodness-of-fit, etcetera) to the
user when manipulating images? The LOG facility appears to exist mainly for error/warning reporting. Some commands can produce copious output, foe example
the ramp removal can print the coeffs of the fit. Should they be printed, and if yes,
where?
A. Here is what I would propose:
1. In general, low level algorithms should not produce textual output aimed at
the user. They should return appropriate values, then the user-level program
can decide what to display.
2. If you really need, at least optionally, to produce output from within a C++
algorithm, use the LOG mechanism. By changing the LOG priority before
making the function call, the higher level program can then decide whether
the output should be generated or not. Lots of printing in high performance algorithms can have a severe impact on performance, so this should
be avoided where possible. 3.6
3. most user level programs should provide a –verbose=<int> option to set the
verbosity level. The user level program would then do appropriate things to
implement this. This would generall be on a 0-9 scale with 0 being completely or almost completely silent and 9 being extremely verbose. In many
cases only one or two levels may exist.
4. Note that image objects have an attribute mechanism, which can be used to
store relevant, but indirect results. For example, say you called a routine
which aligns one 2D image to another. The resulting aligned image might
16
have attributes set indicating the alignment parameters and the goodness of
fit.
5. A note to 4, above, we are gradually adopting the PDB/EBI standard attribute
dictionary for many parameters, so we may require that algoritm specific
attributes would be named in such a way that they are unique.
formatted by Liwei Peng on 11/30/2004
3.3 Coding Style
1. Introduction
• This document summarizes the basic coding and naming style in EMAN2.
The intended audiences are EMAN2 developers.
• This document is not a programming tutorial. You may refer the references at the end of document for good books.
2. Overview
(a) Coding:
i. EMAN2 follows the GNU coding style with minor changes. We
use indent at http://www.gnu.org/software/indent/indent.html for
beautifying EMAN2 code.
(b) Naming:
i. All source code files use lower cases.
ii. All classes and data types use uppercase in the first letter;
iii. All functions use lower cases with ’ ’ if necessary.
3. indent HowTo
(a) install indent. (for linux, rpm is available from standard distribution.)
(b) save file .indent.pro to your home directory.
(c) say you have a file called “foo.C”, run indent like this: indent foo.C
(d) because indent is designed for C code, it is not perfect for C++ code.
Read your new source and fix the following possible errors:
i. change ’const const’ to ’const’
4. Comments
Use Doxygen JavaDoc style:
17
/** Brief description which ends at this dot. Details follow here.\\
*/\\
class Test\\
{\\
public:\\
/** The constructor’s brief description in one line.\\
* A more elaborate description of the constructor.\\
*/\\
Test();\\
\\
/** do some test.\\
* @author Liwei Peng <lpeng@bcm.tmc.edu>\\
* @date 1/20/2005\\
* @param low the low threshold.\\
* @param high the high threshold.\\
* @return 0 if do_test succeeds; 1 if do_test fails.\\
*/\\
int do_test(float low, float high);\\
\\
/** Calculate the sum of an array.\\
* @param[in] data Data array\\
* @param[in] nitems Number of items in the array.\\
* @param[out] sum The sum of the array.\\
*/\\
void calc_sum(int[] data, int nitems, int *sum);\\
\\
}\\
5. Samples
(a) emdata.h
(b) emdata.cpp
6. References
• Doxygen: http://www.stack.nl/˜dimitri/doxygen/
Last modifiedby Liwei Peng (lpeng@bcm.tmc.edu)
3.4 Using Factory Classes
EMAN has many functions that implement specific algorithms to do image processing. These algorithms are relatively independent and a user chooses which one
18
to use. EMAN2 the following factory classes are defined to manage these algorithms:
• Processor (image processor)
• Aligner (2D/3D image alignment)
• Averager (averaging a set of images)
• Cmp (comparing 2 same-size Images)
• Projector (3D image projection)
• Reconstructor (3D image reconstruction)
3.5 Exceptions
3.5.1 Using Exceptions
• Here is an example on throwing an exception:
vector <float>EMData::calc_fourier_shell_correlation(EMData * with){
if (!with) {
throw NullPointerException("NULL input image");
}
if (!EMUtil::is_same_size(this, with)) {
LOGERR("images not same size");
throw ImageFormatException( "images not same size");
}
//...
}
• Here is an example on catching all possible exception
void foo()
{
EMData* e1 = new EMData();
EMData* e2 = new EMData();
try {
e1->read_image("test1.mrc");
e2->read_image("test2.mrc");
vector<float> v = e1->calc_fourier_shell_correlation(e2);
}
catch (E2Exception & exception) {
printf("%s\n", exception.what());
19
}
}
• Here is an example on catching a specific exception:
void foo()
{
EMData* e1 = new EMData();
EMData* e2 = new EMData();
try {
e1->read_image("test1.mrc");
e2->read_image("test2.mrc");
vector<float> v = e1->calc_fourier_shell_correlation(e2);
}
catch (_NullPointerException & exception) {
printf("%s\n", exception.what());
}
}
Note the “ ” before NullPointerException.
3.5.2 Existing Exception Types
• See the complete EMAN2 Reference Guide
3.5.3 Defining New Exception Classes
• A XYZ Exception class is defined in the following way:
– It will extend E2Exception class.
– The class is named XYZException.
– The class has a function to return its name ”XYZException”.
– A macro called ”XYZException” is defined to simplify the usage of
XYZException class.
• Here is the code for NullPointerException.
class _NullPointerException : public E2Exception {
public:
_NullPointerException(const string& file = "unknown",
int line = 0,
const string& desc_str = "")
: E2Exception(file, line, desc_str) {}
20
const char *name() const{ return "NullPointerException"; }
};
#define NullPointerException(desc) _NullPointerException(__FILE__,\
__LINE__,desc)
3.6 Using the Log Class
EMAN2 includes a built in class to handle writing log messages. It is highly recommended that all textual output be handled by this class.
3.6.1 Typical Usage
It is sometimes necessary to specify what output level of the message should be.
There are 4 log levels to choose from
• ERROR LOG - used for error messages
• WARNING LOG - used for warning messages
• DEBUG LOG - used for debugging messages
• VARIABLE LOG - highly detailed debugging messages
The default value used is ERROR LOG; to change the level use:
Log::logger()->set_level(Log::NEW_LOG_LEVEL);
Next, output the log message using:
Log::logger()->OUTPUT_FUNC;
where OUTPUT FUNC is one of the following:
• error(string) - log an error message
• warn(string) - log a warning message
• debug(string) - log a debugging message
• variable(string) - log a very detailed debugging message
21
3.7 Adding Components
There are two general procedures for adding new components to EMAN2. The
first option involves editing templates in order to integrate the new component into
the core while the second option involves directly altering the existing core files.
It is generally recommended that components should be added first using templates and later directly added to existing core if desired. When testing or refining
a new components build times are much faster if templates are used.
- General Overview for Using Templates:
1. Edit the template.h and template.cpp files for the type of component
that is being added
2. From EMAN2/src/build run:
% make
3.7.1 Processors
Using Templates:
Located in the plugin directory of the EMAN2 source (i.e. EMAN2/src/eman2/plugins)
are various template files including processor template.h and processor template.cpp.
These are the files that will be used for new processor installation. Begin by editing
processor template.h.
1. Change the occurrences of ”XYZ” in ”XYZProcessor” with the name of the
new processor
• Don’t forget to change the string in get name() to the name of the processor (this is the name that will be used to call the processor)
2. Edit the string in get desc() with a brief description of the processor. Place
a more detailed descriptions elsewhere such as before the class or before the
functions (see 3.3 for coding style information)
3. Define the processor’s parameters in get param types()
• A description string can be added as a third param to the TypeDict::put()
function to describe the variables
4. In the class constructor of FilterFactorExt uncomment the line ”Factory <Processor>
::add(&dProcessor::NEW);”
Now edit processor template.cpp
• In processor() add the implementation code of the new processor.
22
– The existing template has sample code showing how to access the variables that where defined in get para types()
– Note that the sample code included in the template is enclosed in a conditional statement that essentially causes all of the code tobe skipped.
Finally rebuild EMAN:
% cd ../../build
% make
The new processor should now be available using the name that was specified in
get name().
Adding Directly to the Core:
If the new processor code has already been created using the supplied templates,
then adding to the core can be done as follows:
1. Open processor.h in src/eman2/libEM
2. Copy the class you defined in processor template.h and paste it in the file
3. Open processor.cpp (also located in src/eman2/libEM)
4. Copy and paste the class from processor template.cpp to processor.cpp. Header
file include statements may also need to be copied.
5. In the template class Factory located in the begining of processor.cpp add a
line ”force add(&newProcessor::NEW)” where ”newProcessor” is the name
of the processor class that is being added
6. Rebuild EMAN2
The instructions for adding a new processor without first using templates go as
follows:
1. In src/eman2/libEM open processor.h
2. Towards the end of the file there is an example class called XYZProcessor.
Follow the first 3 steps listed in ”Using Templates”.
3. Open processor.cpp and write an implementation for the process() function
that was just defined in processor.h
4. Repeat the last two steps (5 and 6) from the template installation
23
3.7.2 Aligners
The instructions are the same as those for adding a new processor (3.7.1) except
the files of interest are aligner template.h and aligner template.cpp.
Also, when adding to the core, aligner.h and aligner.cpp should be altered.
3.7.3 Averagers
The instructions are the same as those for adding a new processor (3.7.1) except
the files of interest are averager template.h and averager template.cpp.
Also, when adding to the core, averager.h and averager.cpp should be altered.
3.7.4 Image Comparator
The instructions are the same as those for adding a new processor (3.7.1) except
the files of interest are cmp template.h and cmp template.cpp.
Also, when adding to the core, cmp.h and cmp.cpp should be altered.
3.7.5 IO Functionality
The instructions are the same as those for adding a new processor (3.7.1) except
the files of interest are io template.h and io template.cpp.
Also, when adding to the core, *****NEED TO CHECK THIS***** should
be altered.
3.7.6 Reconstructors
The instructions are the same as those for adding a new processor (3.7.1) except
the files of interest are reconstructor template.h and reconstructor template.cpp.
Also, when adding to the core, reconstructor.h and reconstructor.cpp should be
altered.
3.8 Adding Documentation
The majority of the documentation for EMAN2 is made using LATEX (see the
LATEX homepage for more information). No preamble should be included in the
new documentation; preamble information can removed either manually or using
the python script rm preamble.py (see 3.8.2 for more information).
Documentation can be created using a WYSIWYG LATEX editor such as LyX
or TeXmacs. When using these utilities do not create title pages, abstract sections,
or tables of content as this will cause integration problems with the rest of documentation.
The recommended steps for adding new documentation go as follows:
1. Create a new stand alone document (see 3.8.1 for standards information)
24
2. Remove the preamble information from the new file (3.8.2)
3. Edit TOC.tex (3.8.3)
4. Run “make” to create the new documentation file
3.8.1 Document Standards
Please do not include the following sections/commands in new documentation:
• \title,\maketitle
• \tableofcontents
Edit the existing title and table of content information instead.
3.8.2 Removing Preambles
The preamble of a LATEX document is used to specify various global parameters of
the document. In a stand alone document, all commands coming before and including \begin{document} are part of the preamble. There is also a \end{document}
at the very end of the document that should be removed.
Use the “%” character at the start of a line to comment it out.
The python script rm preamble.py can usually be used to automatically remove
the preamble and other tags that should not be used.
% python rm preamble.py file1 file2 ...
The output will be placed in the file1.tex, file2.tex, ...
3.8.3 Editing TOC.tex
TOC.tex contains the table of contents information for this document. Add “\input{new
file name}” in the appropriate chapter to insert a new documentation file. Of course
a new chapter can always be added using the “\chapter{new chapter name}” command.
3.8.4 Adding LATEX Packages
The package inclusion tag for the documentation is located in EMAN2.tex. Add
the package that is needed to the \usepackage command.
3.9 Image Header Attribute Naming Conventions
EMAN2 uses a unique name to reference each image header attribute. The following table lists the conventions:
25
Attribute Name
apix x
apix y
apix z
nx
ny
nz
origin row
origin col
origin sec
minimum
maximum
mean
sigma
square sum
Data Type
float
float
float
int
int
int
float
float
float
float
float
float
float
float
mean nonzero
float
sigma nonzero
float
kurtosis
skewness
orentation convention
rot alt
rot az
rot phi
datatype
float
float
string
float
float
float
int
is complex
int
is ri
int
avgnimg
int
label
string
image filename
string
image index
DM3.exposure number
DM3.exposure time
int
int
double
Attribute Description
pixel x size in
pixel y size in
pixel z size in
image x-direction size in number of pixels
image y-direction size in number of pixels
image z-direction size in number of pixels
origin location along row (x) in number of pixels
origin location along column (y) in number of pixels
origin location along section (z) in number of pixels
minimum pixel density value in the image
maximum pixel density value in the image
mean pixel density value in the image
sigma pixel density value in the image
sum of the squares of each pixel density value in the
image
mean pixel density value in the image, ignoring pixel
whose value =0.
sigma pixel density value in the image, ignoring pixel
whose value =0.
kurtosis pixel density value in the image
skewness pixel density value in the image
Euler orientation Convention, ”EMAN”, ”MRC”, etc
EMAN convention: alt
EMAN convention: az
EMAN convention: phi
pixel data type in the image physical file (e.g., float,
int, double, etc). This is an enumeration type.
1 if this image is a complex image. 0 if it is a real
image.
1 if this is a complex image and is in real/imaginary
format; 0 otherwise.
If the image is the result of averaging, avgnimg is the
number of images used in averaging. Otherwise, avgnimg is 0.
user-defined label for this image. It is like ’name’ in
EMAN1.
the physical image filename where this image comes
from.
image number in file pointed to by path.
DM3 image only. exposure number.
DM3 image only. exposure time in seconds.
Continued on next page
26
Attribute Name
DM3.zoom
DM3.antiblooming
DM3.magnification
DM3.frame type
DM3.camera x
Data Type
double
int
double
string
int
DM3.camera y
int
DM3.binning x
DM3.binning y
MRC.nlabels
int
int
int
MRC.labelN
string
LST.filenum
LST.reffile
LST.refn
LST.comment
micrograph id
particle center x
partcle center y
ctf dimension
int
string
int
string
string
float
float
int
Attribute Description
zoom
antiblooming
Indicated magnification
frame type
DM3 image only. camera size along x direction in
number of pixels.
DM3 image only. camera size along y direction in
number of pixels.
DM3 image only. Binning #0
DM3 image only. Binning #1
MRC image only. Number of comment labels in a
MRC image.
MRC image only. The Nth label, where N is an integer
from 1 to 10.
LIST image only, file number in FileSystem server
LIST image only, reference file
LIST image only, reference image number
LIST image only, comment
Micrograph ID.
x coordinate of particle location in the micrograph
y coordinate of particle location in the micrograph
CTF dimension, nD, where n is 1, 2, or 3.
Table 3.1: Image Attribute Namimg
3.10 Testing Framework
EMAN2 uses python unit test and regression test as its testing framework. For
features not testable or not easily to be tested in python, they will be tested in C++.
3.10.1 Testing Guidelines
• Each unit test must be self contained.
• Each test module (.py file) should be named as ”test ” + featurename.
• Each test method should be named as ”test ” + method-name.
• unittest module should be used to do the unit testing. doctest is discouraged.
3.10.2 Test Example
• Example of a unit test and a regression test:
27
import unittest
from test import test_support
class MyTestCase1(unittest.TestCase):
# Only use setUp() and tearDown() if necessary
def setUp(self):
... code to execute in preparation for tests ...
def tearDown(self):
... code to execute to clean up after tests ...
def test_feature_one(self):
# Test feature one.
... testing code ...
def test_feature_two(self):
# Test feature two.
... testing code ...
... more test methods ...
class MyTestCase2(unittest.TestCase):
... same structure as MyTestCase1 ...
... more test classes ...
def test\_main():
test\_support.run\_unittest(MyTestCase1,
MyTestCase2,
... list other tests ...
)
if __name__ == ’__main__’:
test_main()
• Here is a more detailed example:
import random
import unittest
class TestSequenceFunctions(unittest.TestCase):
28
def setUp(self):
self.seq = range(10)
def tearDown(self):
# do clean up here
def testshuffle(self):
# make sure the shuffled sequence does not lose any elements
random.shuffle(self.seq)
self.seq.sort()
self.assertEqual(self.seq, range(10))
def testchoice(self):
element = random.choice(self.seq)
self.assert_(element in self.seq)
def testsample(self):
self.assertRaises(ValueError, random.sample, self.seq, 20)
for element in random.sample(self.seq, 5):
self.assert_(element in self.seq)
if __name__ == ’__main__’:
unittest.main()
• Asserting Values and Conditions
The crux of each test is a call to assertEqual() to check for an expected
result; assert () to verify a condition; or assertRaises() to verify that an
expected exception gets raised.
3.11 CVS Help
CVS is a version control system. Using it, you can record the history of sources
files, and documents. For detailed information, you may refer http://www.gnu.org/software/cvs/.
This document gives the typical usage of CVS under Unix/Linux platforms.
The intended audience are EMAN2 developers.
• Before you use CVS, set up environmental variable CVSROOT. For EMAN/EMAN2,
set up the following in your shell startup script:
– for csh/tcsh
% setenv CVS RSH ssh
% setenv CVSROOT ”blake.3dem.bioch.bcm.tmc.edu:/usr/local/CVS/CVS”
29
– for bash/sh/zsh
% export CVS RSH=ssh
% export CVSROOT=”blake.3dem.bioch.bcm.tmc.edu:/usr/local/CVS/CVS”
NOTE: The following supposes you put EMAN2 under $HOME/EMAN2.
• To check out EMAN2 source code, run
% cd $HOME/EMAN2/src
% cvs co eman2
• To add new files, run
% cd $HOME/EMAN2/src/eman2
% cd YOUR-DIRECTORY
% cvs add YOUR-FILES
% cvs commit
• To remove files, run
% cd $HOME/EMAN2/src/eman2
% cd YOUR-DIRECTORY
% cvs remove YOUR-FILES
% cvs commit
• To check in modified existing files, run
% cd $HOME/EMAN2/src/eman2
% cvs ci
• To update your source code to the latest version in the CVS tree, run
% cd $HOME/EMAN2/src/eman2
% cvs update
NOTE: read the output from the above command. If you see merging conflicts, you must resolve them first.
3.12 Miscellaneous
3.12.1 Reading/Writing Images in Python
Example: Converting from IMAGIC to MRC or SPIDER:
30
from EMAN2 import *
e = EMData()
e.read_image(\u201cproj.img\u201d)
e.write_image(\u201cproj.mrc\u201d)
# output image format is determined by file extension
e.write_image(\u201cproj.spi\u201d, 1, SPIDER)
# explicitly specifiy the output image format
3.12.2 Processors Usage
Example: Using Processors in Python
from EMAN2 import
e = EMData()
e.read_image("test1.mrc")
e.process("math.sqrt")
e.process("threshold.binaryrange", {"low" : 5.2, "high" : 10})
e.write_image("output.mrc")
3.12.3 Reconstructors Usage
Example: Using Reconstructors in Python
from EMAN2 import *
import math
e1 = EMData()
e1.read_image(TestUtil.get_debug_image("samesize1.mrc"))
e2 = EMData()
e2.read_image(TestUtil.get_debug_image("samesize2.mrc"))
r = Reconstructors.get("back_projection")
r.set_params({"size":100, "weight":1})
r.setup()
r.insert_slice(e1, Transform(EULER_EMAN, 0,0,0))
r.insert_slice(e2, Transform(EULER_EMAN, math.pi/2,0,0))
result = r.finish()
result.write_image("reconstructor.mrc")
3.12.4 Using Aligner, Comparators, and Projectors
Using any of these objects is very similar to the process described in Reconstructor
Usage above. The general process is
31
obj = CLASS.get(OBJECT_TYPE, {arg1:value1,
arg2:value2, ...})
data = obj.OBJECT_COMMAND(args)
Use the EMAN2 html documentation or use the “dump” command in python to see
a list of the available object types for each class and their corresponding argument
lists.
3.12.5 Using Pyste
EMAN2 uses Pyste to automatically parse C++ code to generate boost python
wrappers. To use Pyste:
1. Install Pyste libraries/tools:
(a) Pyste in boost library
(b) elementtree
(c) gccxml
2. Create or modify the pyste file (e.g., eman2/libpyEM/processor.pyste). For
a function that return a pointer, a return-policy must be defined in the pyste
file. The typical cases are:
(a) If the function returns a pointer allocated in this function, do:
set policy(YOUR FUNCTION, return value policy(manage new object))
(b) If the function returns a static pointer , do:
set policy(YOUR FUNCTION, return value policy(reference existing object))
(c) For other cases, do:
set policy(YOUR FUNCTION, return internal reference())
3. Run script: eman2/libpyEM/create boost python
3.12.6 Using FFTW
EMAN2 works with both fftw2 and fftw3. A user makes the choice at compile
time. A standard interface is defined to do fft:
class EMfft {
public:
static int real_to_complex_1d(float *real_data, float *complex_data,
int n);
static int complex_to_real_1d(float *complex_data, float *real_data,
int n);
static int real_to_complex_nd(float *real_data, float *complex_data,
int nx, int ny, int nz);
32
static int complex_to_real_nd(float *complex_data, float *real_data,
int nx, int ny, int nz);
};
3.12.7 Large File I/O
1. portable fseek() should be used for fseek.
2. portable ftell() should be used for ftell.
3.12.8 Euler Angles
• Euler angles are implemented in Rotation class.
• Rotation r = Rotation(alt, az, phi, Rotation::EMAN);
float alt2 = r.eman_alt();
float az2 = r.eman_az();
float phi2 = r.eman_phi();
float theta = r.mrc_theta();
float phi = r.mrc_phi();
float omega = r.mrc_omega();
3.12.9 Using numpy
• In EMAN2, Numeric array and the corresponding EMData object shares the
same memory block.
• Example: Converting EMData object to numpy array
from EMAN2 import *
e = EMData()
e.read_image("e.mrc"))
array = EMNumPy.em2numpy(e)
• Example: Converting Numerc numpy array to EMData object
from EMAN2 import *
import numpy
n= 100
numbers= range(2*n*n)
array= numpy.reshape(numpy.ndarray(numbers,numpy.Float32),(2*n,n))
e = EMNumpy.numpy2em(array)
e.write_image("numpy.mrc")
33
3.12.10 Using Transformations
Transform defines a transformation, which can be rotation, translation, scale, and
their combinations.


a b c
 e f g 


Internally a transformation is stored in a 4x3 matrix. 

 i j k 
m n o


a b c


The left-top 3x3 submatrix  e f g  provides rotation, scaling and skewi j k
ing.
h
i
Post translation is stored in m n o




A separate vector containing the pretranslation, with an implicit column 
0
0
0
1



at

the end when 4x4 multiplies are required.
The ’center of rotation’ is NOT implemented as a separate vector, but as a
combination of pre and post translations.
3.12.11 Printing Error/Warning/Debugging Information
• Using the Log Class
1. In your main() file, set log level: Log::logger() −→set log level(WARNING LOG);
2. Log message in different levels: (log functions use the same argument
format like printf()).
LOGERR(”out of memory”);
LOGWARN(”invalid image size”);
LOGDEBUG(”image mean density = %f\n”, mean);
LOGVAR(”image size = (%d, %d, %d)\n”, nx, ny, nz);
3. To log function enter point, use ENTERFUNC; To log function exit
point, use EXITFUNC.
3.12.12 Adding Testing Groups
• These group tags are already defined in file, ”eman2doc.h”:
– tested0 : code not yet complete
– tested1 : code complete but untested
– tested2 : code complete but contains bugs
– tested3 : tested
34
– tested3a : manual testing
– tested3b : unit test in C++
– tested3c : unit test in Python
– tested3d : incorporated into sucessful regression test
• How to use these tag to label testing group:
– add /**@ingroup tested3c*/ to the beginning of a class tested in Python,
then the corresponding class will be labeled ”unit test in Python” in
doxygen generated document.
– you can also define other grouping tag, just follow the testing group
example in ”eman2doc.h”
– a single unit can be labeled for multiple group
3.13 How to Install Boost Python
1. Download ’bjam’ for your platform.
2. Download boost source from http://www.boost.org. Assume the version is
boost 1 32 0.
% cd /usr/local/src
% tar zxf boost 1 32 0.tar.gz
% cd boost 1 32 0.
3. Set up environment variables ”PYTHON ROOT” and ”PYTHON VERSION”.
For example, if your python is at /usr/bin/python then PYTHON ROOT is
”/usr”. If your python version is 2.2.X, then PYTHON VERSION is ’2.2’.
(a) check your shell: % echo $SHELL
(b) if you are using bash/zsh, do
% export PYTHON VERSION=2.2
% export PYTHON ROOT=/usr
if you are using csh/tcsh, do
% setenv PYTHON VERSION 2.2
% setenv PYTHON ROOT /usr
4. cd libs/python/build
5. run ’bjam’ with your options:
- linux-x86: % bjam
35
- SGI Irix: % bjam ”-sTOOLS=mipspro”
6. login as root
7.
% cp -df bin-stage/libboost python.so* /usr/local/lib
% cd ../../..; cp -rf boost /usr/local/include
3.14 How to use your own version of python
If the python you want to use in your computer is not found by CMake, you may
set up environment variables ”PYTHON ROOT” and ”PYTHON VERSION”. For
example, if your python is at /usr/local/python2.4/bin/python. PYTHON ROOT is
”/usr/local/python2.4”. if your python is 2.4.X, PYTHON VERSION is ’2.4’.
3.15 How to Install numpy
From the website http://sourceforge.net/projects/numpy and download version 1.0.1
of numpy.
For windows, run the binary installer and the installation is complete. Other
users must download the source code and install manually as follows:
- get source code numpy-XX.Y.tar.gz
- % gunzip numpy-XX.Y.tar.gz
% tar xf numpy-XX.Y.tar
- login as root
- % cd numpy-XX.Y;
% python setup.py install
3.16 Writing Python Scripts
3.16.1 Logging
EMAN2 includes built-in functionality to create a record of what commands are
executed, using what parameters. This information is stored in the file “.eman2log”
located in the directory the commands are run.
In order to include records of new programs into the log, use the functions
E2init([args]) to initiate an entry and the function E2end() to finalize the entry.
For example:
logger = E2init (args)
.
. # interesting code here
36
.
E2end (logger)
3.16.2 Command-Line Argument Parsing
For convenience, the OptionParse module is generally used to handle commandline argument parsing. If another parser is used, it is asked that the input argument
format conform to the standards of OptionParser. Generally this means using a “-”
in front of single letter options such as “-v” and using “–” in front of longer options
names such as “–verbose” or “–limit=15.4”
3.16.3 Code Modularity
It is recommended that all program code be separated into 1 or more functions
rather directly written at the base level. This allows programs to be run either
individually or imported as modules into other programs that wish to use their
functionality.
Normally the base level of the program only includes
if __name__=="__main__":
main()
where main() is the function that actually begins completing the program’s work.
37
Chapter 4
Program Manuals
4.1 e2align3d.py
4.1.1 Usage
usage: e2align3d.py [options] fixed.mrc input.mrc output.mrc
4.1.2 Description
Locates the best ’docking’ locations for a small probe in a large target map.
4.1.3 Options
Argument
–version
-h, –help
–csym
Description
show program’s version number and exit
show this help message and exit
Restrict axes for c/d symmetric objects
Table 4.1: e2align3d.py Options
4.2 e2aligntest.py
4.2.1 Usage
usage: e2aligntest.py [options] input1 input2
4.2.2 Description
Locates the best ’docking’ locations for a small probe in a large target map.
38
4.2.3 Options
Argument
–version
-h, –help
–axes=AXES
Description
show program’s version number and exit
show this help message and exit
String list 3 axes from xyzaqp
Table 4.2: e2aligntest.py Options
4.3 e2boxer.py
4.3.1 Usage
usage: e2boxer.py [options] <image>
4.3.2 Description
Automatic and manual particle selection. This version is specifically aimed at
square boxes
for single particle analysis.
4.3.3 Options
Argument
–version
-h, –help
–gui
-B
-P
-R
-V
-S
-A
-T
Description
show program’s version number and exit
show this help message and exit
Start the GUI for interactive boxing
BOX, –box=BOX Box size in pixels
PTCLSIZE, –ptclsize=PTCLSIZE Approximate size (diameter) of the particle in pixels. Not required if reference
particles are provided.
REFPTCL, –refptcl=REFPTCL A stack of reference images. Must have the same scale as the image being boxed.
REFVOL, –refvol=REFVOL A 3D model to use as a reference for autoboxing
SYM, –sym=SYM Symmetry of the 3D model
AUTO, –auto=AUTO Autobox using specified method:
circle, ref
THRESHOLD, –threshold=THRESHOLD Threshold for
keeping particles. 0-4, 0 excludes all, 4 keeps all.
Continued on next page
39
Argument
–nretest=NRETEST
Description
Number of reference images (starting with the first) to use
in the final test for particle quality.
Comma separated list of image numbers for retest cycle
Edgenormalize boxed particles
Stores intermediate aligned particle images in boxali.hdf.
Mainly for debugging.
filename or ’next’, name of an aligned far from focus image for preliminary boxing
–retestlist=RETESTLIST
–norm
–savealiref
–farfocus=FARFOCUS
Table 4.3: e2boxer.py Options
4.4 e2cmmtomrc.py
4.4.1 Usage
usage: e2cmmtomrc.py [options] input.cmm output.mrc
4.4.2 Description
4.4.3 Options
Argument
–version
-h, –help
-A
-R
-B
Description
show program’s version number and exit
show this help message and exit
APIX, –apix=APIX A/voxel
RES, –res=RES Resolution in A, equivalent to Gaussian
lowpass with 1/e width at 1/res
BOX, –box=BOX Box size in pixels
Table 4.4: e2cmmtomrc.py Options
4.5 logfile
4.5.1 Usage
no logfile
4.5.2 Description
4.5.3 Options
40
Argument
Description
Table 4.5: logfile Options
4.6 e2foldfitter.py
4.6.1 Usage
usage: e2foldfitter.py [options] target.mrc probe.mrc
4.6.2 Description
Locates the best ’docking’ locations for a small probe in a large target map. Note
that the probe
should be in a box barely large enough for it. The target may be arbitrarily
padded. For best speed
both box sizes should be multiples of 8.
4.6.3 Options
Argument
–version
-h, –help
-S
-E
Description
show program’s version number and exit
show this help message and exit
SHRINK, –shrink=SHRINK shrink factor for initial
search, default=auto
EPSILON, –epsilon=EPSILON final target accuracy, default=.01
Table 4.6: e2foldfitter.py Options
4.7 e2make3d.py
4.7.1 Usage
usage: e2make3d.py <input file> [options]
4.7.2 Description
4.7.3 Options
41
Argument
–version
-h, –help
–out=FILENAME
–lowmem
–sym=SYM
–pad=PAD
–mode=MODE
–recon=RECON TYPE
–quiet
–snrfile=SNRFILE
–goodbad
–log
–inorm
–fftmerge=FFTMERGE
–savenorm
–hard=HARD
–noweight
–mask=MASK
–apix=APIX
–keep=SIGM
–resmap=RESMAP
Description
show program’s version number and exit
show this help message and exit
Output 3D MRC file
Read in images only when needed instead of preloading
all in memory
Set the symmetry; if no value is given then the model is
assumed to have no symmetry. Choices are: i, c, d, t, icos,
or oct
To reduce Fourier artifacts, the model is typically padded
by 25%
Specifies the interpolation size (1 to 6), 2 is the default
Reconstructor to use
Quiet output
Use a SNR file (as generated by classalignall) for proper
3D Wiener filtration
Saves the used and unused class averages in 2 files
Averages the log of the projections
This will use a special weighting scheme to compensate
for poor SNR sampling on the unit sphere
Fourier model to merge with real model
Saves the normalization map to norm.mrc
This specifies how well the class averages must match the
model to be included, 25 is typical
Normally the class averages are weighted by the number
of raw particles used, this disables that
Real-space mask radius
Set the sampling (angstrom/pixel)
An alternative to ’hard’
Generates a ’resolution map’ in another 3D MRC file
Table 4.7: e2make3d.py Options
4.8 e2pdb2mrc.py
4.8.1 Usage
usage: e2pdb2mrc.py [options] input.pdb output.mrc
4.8.2 Description
Converts a pdb file into an electron density map. 0,0,0 in PDB space will
42
map to the center of the volume. Use e2procpdb.py to adjust coordinates,
apply symmetry, etc. Resolution is equivalent to standard cryoEM definition,
using 1/2 width of Gaussian in Fourier space.
4.8.3 Options
Argument
Description
–version
-h, –help
-A
-R
-B
–het
–chains=CHAINS
–quiet
show program’s version number and exit
show this help message and exit
APIX, –apix=APIX A/voxel
RES, –res=RES Resolution in A, equivalent to Gaussian
lowpass with 1/e width at 1/res
BOX, –box=BOX Box size in pixels, <xyz> or
<x>,<y>,<z>
Include HET atoms in the map
String list of chain identifiers to include, eg ’ABEFG’
Verbose is the default
Table 4.8: e2pdb2mrc.py Options
4.9 e2proc2d.py
4.9.1 Usage
usage: e2proc2d.py options inputfile outputfile
4.9.2 Description
4.9.3 Options
Argument
Description
–version
-h, –help
–apix=APIX
–average
show program’s version number and exit
show this help message and exit
A/pixel for S scaling
Averages all input images (without alignment) and writes
a single (normalized) output image
outputfile calculate a radial structure factor for the image
and write it to the output file, must specify apix. divide
into <n> angular bins
ysize Define the output image size. CLIP=xsize ysize
Continued on next page
–calcsf=n
–clip=xsize
43
Argument
–ctfsplit
–exclude=exclude-list-file
–fftavg=filename
–filter=filtername:param1=val1:param2=val2
–first=n
–inplace
–interlv=interleave-file
–last=n
–list=listfile
–meanshrink=n
–mraprep
–norefs
–outtype=image-type
–plt=plt-file
–radon
–rfp
–scale=f
–selfcl=steps
–setsfpairs
–shrink=n
–split=n
–verbose=n
Description
Splits the input file into output files with the same CTF
parameters
Excludes image numbers in EXCLUDE file
Incoherent Fourier average of all images and write a single
power spectrum image
apply a filter named ’filtername’ with all its parameters/values.
the first image in the input to process [0 - n-1])
Output overwrites input, USE SAME FILENAME, DO
NOT ’clip’ images.
Specifies a 2nd input file. Output will be 2 files interleaved.
the last image in the input to process
Works only on the image numbers in LIST file
Reduce an image size by an integral scaling factor using
average. Clip is not required.
this is an experimental option
Skip any input images which are marked as references
(usually used with classes.*)
output image format, mrc, imagic, hdf, etc
output the orientations in IMAGIC .plt file format
Do Radon transform
this is an experimental option
Scale by specified scaling factor. Clip must also be specified to change the dimensions of the output map.
mode Output file will be a 180x180 self-common lines
map for each image.
Applies the radial structure factor of the 1st image to the
2nd, the 3rd to the 4th, etc
Reduce an image size by an integral scaling factor, uses
median filter. Clip is not required.
Splits the input file into a set of n output files
verbose level [1-4]
Table 4.9: e2proc2d.py Options
4.10 e2proc3d.py
4.10.1 Usage
usage: e2proc3d.py options inputfile outputfile
44
4.10.2 Description
4.10.3 Options
Argument
Description
-h, –help
–shrink=n
–scale=n
–clip=x
–fftclip=x
show this help message and exit
Shrinks the image by integer n using median filter
Rescales the image by ’n’, generally used with clip option.
y z xc yc zc Make the output have this size, no scaling.
y z Make the output have this size, rescaling by padding
FFT.
Apply a filter named ’filtername’ with all its parameters/values.
A/pixel for S scaling
y z Set the coordinates for the pixel (0,0,0).
Scales the densities by ’f’ in the output
Adds a constant ’f’ to the densities
Calculate a radial structure factor. Must specify apix.
The output only keeps the top half map
The input is the icos 5f top half map generated by the
’tophalf’ option
Set output image format, mrc, imagic, hdf, etc
–filter=filtername:param1=val1:param2=val2
–apix=APIX
–origin=x
–mult=f
–add=f
–calcsf=outputfile
–tophalf
–icos5fhalfmap
–outtype=image-type
Table 4.10: e2proc3d.py Options
4.11 e2project3d.py
4.11.1 Usage
usage: e2project3d.py <input file> [options]
4.11.2 Description
4.11.3 Options
Argument
–version
-h, –help
–out=OUTFILE
–projector=PROJECTOR
Description
show program’s version number and exit
show this help message and exit
Output file. Default is ’e2proj.img’
Projector to use
Continued on next page
45
Argument
–prop=PROP
–prop start=PROP START
–prop end=PROP END
–euler=<az>
–sym=SYM
–mode=MODE
–angletype=ANGLETYPE
–dump angles
–mask=MASK
–randomphi
–phicomp
–phitoo=PHITOO
–smear
–grid=GRID
–angle input file=FILE NAME
–pad=PAD
Description
Generates projections with a relatively uniform projection
density in the unit triangle.
Start projections at the specified alt (in degrees). Default=0.0
Create projections up to the specified alt (in degrees). Default=90.0
<alt> <phi> Generate a single projection with the given
orientation
Set the symmetry; choices are: c<n>, d<n>, h<n>, i, t,
icos, or oct
Default is real-space projection, this specifies various
Fourier modes
Angle convention to use: [EMAN, SPIDER]. EMAN is
the default
Dumps Euler angles to a text file
Specify a circular mask radius for the projections
Randomize phi
Roughly compensate for in-plane rotation from az with
phi
This will also vary phi in the final file. Warning: This
works only with ’–prop=’ and generates a LOT of projections.
Used in conjunction with ’–phitoo=’, this will rotationally
smear between phi steps.
Generate projections on an rectangular alt/az mesh,
nonuniform projection density
Us an input file to specify what angles to project
Pad image
Table 4.11: e2project3d.py Options
4.12 e2scannereval.py
4.12.1 Usage
usage: e2scannereval.py [options] input.mrc
4.12.2 Description
Designed for use on scanned micrographs. Overlays image with a grid of
2D power spectra.
46
4.12.3 Options
Argument
–version
-h, –help
-S
Description
show program’s version number and exit
show this help message and exit
BOX, –box=BOX size in pixels of the power spectra
Table 4.12: e2scannereval.py Options
4.13 e2symbest.py
4.13.1 Usage
usage: e2symbest.py options inputfile outputfile
4.13.2 Description
4.13.3 Options
Argument
–version
-h, –help
–nkeep=N
–sym=Cn
–mirror=outputfile
–rtp
–mask=rad
–imask=rad
Description
show program’s version number and exit
show this help message and exit
Number of particles to keep
Symmetry to search for
search for particles with mirror symmetry and write them
out to outputfile.
make a rotational footprint
Mask radius
Inside mask radius
Table 4.13: e2symbest.py Options
4.14 e2tomogram.py
4.14.1 Usage
usage: e2tomogram.py [options] input stack.hed output.hed
4.14.2 Description
Processes a tomographic tilt series
47
4.14.3 Options
Argument
–version
-h, –help
-T
-M
-B
–highpass=HIGHPASS
–lowpass=LOWPASS
–mode=MODE
–localavg=LOCALAVG
–tiltaxis=TILTAXIS
–twopass
–nozero
Description
show program’s version number and exit
show this help message and exit
TILT, –tilt=TILT Angular spacing between tilts (fixed)
MAXSHIFT, –maxshift=MAXSHIFT Maximum translational error between images (pixels), default=64
BOX, –box=BOX Box size for alignment probe (pixels),
default=96
Highpass Gaussian processor radius (pixels), default none
Lowpass Gaussian processor radius (pixels), default none
centering
mode
’modeshift’,
’censym’
or
’region,<x>,<y>,<clipsize>,<alisize>
Average several images for the alignment
Skip automatic tilt axis location, use fixed angle from x
Skip automatic tilt axis location, use fixed angle from x
Do not allow 0-translations between images
Table 4.14: e2tomogram.py Options
48
Index
Abstract, 6
Adding Components, 22
Aligners, 24
Averagers, 24
CMP, 24
IO, 24
Processors, 22
Reconstructors, 24
Adding Documentation, 24
Aligners
Adding New, 24
Usage, 31
Averagers
Adding New, 24
Boost, 8
Boost Python
Installation, 35
CMake, 9
CMP
Adding New, 24
Usage, 31
CVS Help, 29
Debugging
Printing Error Information, 34
Developer’s Guide, 11, 13
Coding Style, 17
Doxygen, 17
Factory Classes, 18
FAQ, 13
Reference Guide, 13
Exceptions, 19
Adding New, 20
Usage, 19
FFTW, 8
Usage, 32
Images
Attrib. Naming, 25
Installation, 8
Advanced Installation, 10
Optional Libraries, 9
Quick Installation, 9
Required Libraries, 8
Boost, 8
CMake, 9
FFTW, 8
Introduction, 6
IO
Adding New, 24
Misc., 30
Euler Angles, 33
FFTW, 32
Large File I/O, 33
Pyste, 32
Reading/Writing Images, 30
numpy
Installation, 36
Usage, 33
Processors
Adding, 22
Usage, 31
Program Manuals, 38
Program Manuals
e2align3d.py, 38
e2aligntest.py, 38
e2boxer.py, 39
e2cmmtomrc.py, 40
49
e2foldfitter.py, 41
e2make3d.py, 41
e2pdb2mrc.py, 42
e2proc2d.py, 43
e2proc3d.py, 44
e2project3d.py, 45
e2scannereval.py, 46
e2symbest.py, 47
e2tomogram.py, 47
logfile, 40
Projectors
Usage, 31
Python
Writing Scripts, 36
Reconstructors
Adding, 24
Usage, 31
Testing Groups, 34
Transformations, 34
50
Download PDF