Differences between revisions 2 and 15 (spanning 13 versions)
⇤ ← Revision 2 as of 2017-06-01 21:43:43 →
Size: 6522
Editor: TunayDurmaz
Comment:
← Revision 15 as of 2022-11-07 04:44:29 → ⇥
Size: 6669
Editor: TunayDurmaz
Comment:
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
Line 22: Line 23:
''':TODO:''' Advantages/disadvantages/comparison of the strategies.

''':TODO:''' Resulting file hierarchies, EMAN2DIR
Line 30: Line 27:
 1. Install dependencies with {{{conda install}}}.
 1. Build and install EMAN2 manually into '''root/home directory''' with {{{cmake}}}, {{{make}}} and {{{make install}}}.		  1. Install dependencies
    a. Manually with {{{
conda install <package>
}}}
    a. Or with a single command{{{
conda install eman-deps -c cryoem -c defaults -c conda-forge
}}}

 2. Build and install EMAN2 manually into '''home directory''' with {{{cmake}}}, {{{make}}} and {{{make install}}}.

 3. Resulting installation is under '''$HOME/EMAN2''' by default.
Line 37: Line 43:
 1. Install dependencies with {{{conda install}}}.
 2. Set environment variables, {{{shell export CONDA_BUILD_STATE=BUILD export PREFIX=<path-to-anaconda-installation-directory> # $HOME/miniconda2/ or $HOME/anaconda2/ export SP_DIR=$PREFIX/lib/python2.7/site-packages }}}.		  1. Install dependencies
    a. Manually with {{{
conda install <package>
}}}
    a. Or with a single command{{{
conda install eman-deps -c cryoem -c defaults -c conda-forge
}}}

 2. Set environment variables. {{{
export CONDA_BUILD_STATE=BUILD
export PREFIX=<path-to-anaconda-installation-directory> # $HOME/miniconda2/ or $HOME/anaconda2/
export SP_DIR=$PREFIX/lib/python2.7/site-packages
}}}
Line 40: Line 58:

 4. Resulting installation is under Anaconda/Miniconda installation, '''$HOME/anaconda2/''' or '''$HOME/miniconda2/''' by default.
Line 45: Line 65:
conda install eman2 --use-local -c cryoem -c defaults -c conda-forge
Line 47: Line 68:
Resulting installation is under Anaconda/Miniconda installation, '''$HOME/anaconda2/''' or '''$HOME/miniconda2/''' by default.

''':TODO:''' Advantages/disadvantages/comparison of the strategies.
Line 48: Line 73:
The build strategies described in [section Build Strategies](#build-strategies) are tested on CI (Continuous Integration) servers for MacOSX ([TravisCI](https://travis-ci.org/cryoem/eman2/builds)) and Linux ([CircleCI](https://circleci.com/gh/cryoem/eman2)). For Windows (Appveyor), only the recipe strategy is tested. The tests are triggered for every commit that is pushed to GitHub.
Line 50: Line 74:
The build strategies described in section Build Strategies are tested on CI (Continuous Integration) servers for MacOSX ([[https://travis-ci.org/cryoem/eman2/builds|TravisCI]]) and Linux ([[https://circleci.com/gh/cryoem/eman2|CircleCI]]). For Windows (Appveyor), only the recipe strategy is tested. The tests are triggered for every commit that is pushed to GitHub.

=== Local Build Tests and Automated Daily Snapshot Binaries with Jenkins ===
Line 52: Line 79:
Line 53: Line 81:
Line 54: Line 83:
Packaging is done with `constructor`, a tool for making installers from conda packages. In order to slighltly customize the installers the project was forked. The customized project is at https://github.com/cryoem/constructor. The input files for `constructor` are maintained at https://github.com/cryoem/docker-images.
Line 56: Line 84:
The installer includes additional tools such as `conda`, `conda-build`, `pip`. The installer is setup so that the packages are kept in the installed EMAN2 conda environment cache for convenience. Packaging is done with {{{constructor}}}, a tool for making installers from conda packages. In order to slightly customize the installers the project was forked. The customized project is at https://github.com/cryoem/constructor. The input files for {{{constructor}}} are maintained at --(https://github.com/cryoem/docker-images )-- https://github.com/cryoem/build-scripts.

The installer has additional tools like {{{conda}}}, {{{conda-build}}}, {{{pip}}} bundled. The installer is setup so that the packages are kept in the installed EMAN2 conda environment cache for convenience.
Line 59: Line 90:
The binary packages are built on three physical machines. The operating systems are `Mac OSX 10.10`, `CentOS 7` and `Windows 10`. `CentOS 6` binaries are built in a `CentOS 6` docker container on the `CentOS 7` machine.
Line 61: Line 91:
https://github.com/cryoem/buils-scripts The binary packages are built on three physical machines. The operating systems are Mac OSX 10.10, CentOS 7 and Windows 10. CentOS 6 binaries are built in a CentOS 6 docker container on the CentOS 7 machine. Various build related scripts are on https://github.com/cryoem/build-scripts.
Line 67: Line 97:
[[Windows]] [Windows]
Line 70: Line 100:
Docker images and helper scripts are at https://github.com/cryoem/docker-images. Docker images and helper scripts are at --(https://github.com/cryoem/docker-images )-- https://github.com/cryoem/build-scripts.
Line 72: Line 102:
Runs as root on Linux. `chown` doesn't work, the resulting installer has root ownership. Command to run docker with GUI support, CentOS7: {{{
xhost + local:root
Line 74: Line 105:
:TODO: More explanation. docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY cryoem/eman-nvidia-cuda8-centos7
Line 76: Line 107:
=== Pydusa, OpenMPI and Fftw-mpi ===
The version of `OpenMPI` provided with EMAN2 installer may not work with user's batch queueing system, meaning they would not be able to run jobs on more than one node at a time. In that case, installer-provided `OpenMPI` needs to be removed, `pydusa` and `fftw-mpi` need to be installed against `OpenMPI` that is present in the system. Two scripts, one that uninstalls installer provided `OpenMPI`~~, `pydusa` and `fftw-mpi`,~~ and another that builds `fftw-mpi` and `pydusa` against the system `OpenMPI` are provided. The installer prints a detailed message regarding this issue.		 # When done with eman
xhost - local:root
}}}
Line 79: Line 111:
Only OpenMPI is force-removed. 1. So the system or newly installed version is used 1. Force-removing fftw-mpi and, specifically, pydusa leaves the system broken. 1. Even if not removed, later installations overwrite, so there isn't any need to uninstall anyways. :FIXME: Runs as root on Linux. `chown` doesn't work, the resulting installer has root ownership.

Under Construction

EMAN2 is built with conda-build using binaries from https://anaconda.org, packaged into an installer with constructor as of v2.2.

  1. conda is the package manager.

  2. https://anaconda.org is the online repository of binaries.

  3. conda-build is the tool to build from source.

  4. constructor is the tool to package eman2 and dependency binaries into a single installer file.

EMAN2 is distributed as a single installer which includes all its dependencies. However, EMAN2 is not available as a conda-package on https://anaconda.org. In other words it is not possible to install EMAN2 by typing conda install eman2.

Conda

Packages that are available on https://anaconda.org can be installed into any conda environment by issuing the command conda install <package>. Conda installs the package along with its dependencies. In order for packages to benefit from this automation, they need to be packaged in a specific way. That can be done with conda-build. conda-build builds packages according to instructions provided in a recipe. A recipe consists of a file with package metadata, meta.yaml, and any other necessary resources like build scripts, (build.sh, bld.bat), patches and so on.

Recipes, Feedstocks and anaconda.org channel: cryoem

Most of EMAN2 dependencies can be found on anaconda's channels, defaults and conda-forge. A few that do not exist or need to be customized have been built and uploaded to channel cryoem. The recipes are hosted in separate repositories on GitHub. Every recipe repository follows the feedstock approach of conda-forge. See here for a complete list.

Build Strategies

It is possible to utilize conda for building and installing EMAN2 in a few ways. One way is to just install binaries with conda and point to the right locations of dependencies during cmake configuration. Another way is to make use of the newly added features in EMAN2's cmake which find the dependencies automatically. These features are activated only when the build is performed by conda-build. CMake knows the build is a conda-build build only through an environment variable. So, it is possible to set the specific environment variable manually and still activate those features without actually using conda-build. Third way is to use a recipe to run conda-build. Basic instructions for all three strategies follow.

1. Use conda for binaries only

  1. Install dependencies

    1. Manually with 

      conda install <package>

    2. Or with a single command

      conda install eman-deps -c cryoem -c defaults -c conda-forge

  2. Build and install EMAN2 manually into home directory with cmake, make and make install.

  3. Resulting installation is under $HOME/EMAN2 by default.

This is detailed on EMAN WIKI. Also, see build_no_envars.sh.

2. Use conda to install EMAN2 into a conda environment

  1. Install dependencies

    1. Manually with 

      conda install <package>

    2. Or with a single command

      conda install eman-deps -c cryoem -c defaults -c conda-forge

  2. Set environment variables. 

    export CONDA_BUILD_STATE=BUILD
export PREFIX=<path-to-anaconda-installation-directory> # $HOME/miniconda2/ or $HOME/anaconda2/
export SP_DIR=$PREFIX/lib/python2.7/site-packages

  3. Build and install EMAN2 manually into conda environment with cmake, make and make install. See build_with_envars.sh.

  4. Resulting installation is under Anaconda/Miniconda installation, $HOME/anaconda2/ or $HOME/miniconda2/ by default.

3. Use recipe for a fully automated conda build

conda build <path-to-eman-recipe-directory>
conda install eman2 --use-local -c cryoem -c defaults -c conda-forge

Resulting installation is under Anaconda/Miniconda installation, $HOME/anaconda2/ or $HOME/miniconda2/ by default.

:TODO: Advantages/disadvantages/comparison of the strategies.

Tests

The build strategies described in section Build Strategies are tested on CI (Continuous Integration) servers for MacOSX (TravisCI) and Linux (CircleCI). For Windows (Appveyor), only the recipe strategy is tested. The tests are triggered for every commit that is pushed to GitHub.

Local Build Tests and Automated Daily Snapshot Binaries with Jenkins

:TODO: Jenkins.

Binary Distribution

Constructor

Packaging is done with constructor, a tool for making installers from conda packages. In order to slightly customize the installers the project was forked. The customized project is at https://github.com/cryoem/constructor. The input files for constructor are maintained at https://github.com/cryoem/docker-images https://github.com/cryoem/build-scripts.

The installer has additional tools like conda, conda-build, pip bundled. The installer is setup so that the packages are kept in the installed EMAN2 conda environment cache for convenience.

Build Machines

The binary packages are built on three physical machines. The operating systems are Mac OSX 10.10, CentOS 7 and Windows 10. CentOS 6 binaries are built in a CentOS 6 docker container on the CentOS 7 machine. Various build related scripts are on https://github.com/cryoem/build-scripts.

Cron

:TODO:

Windows

[Windows]

Docker

Docker images and helper scripts are at https://github.com/cryoem/docker-images https://github.com/cryoem/build-scripts.

Command to run docker with GUI support, CentOS7: 

xhost + local:root

docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY cryoem/eman-nvidia-cuda8-centos7

# When done with eman
xhost - local:root

:FIXME: Runs as root on Linux. chown doesn't work, the resulting installer has root ownership.