Build System

Overview

The page goes into details about how the build system is configured.

Since 5.5, OMERO decouples many components and uses, for some components, an Gradle-based build. The two overarching repositories are omero-build and omero-gradle-plugins. See the README of each repository for more details. OMERO still uses an Ant-based build, for some components, with dependency management provided by Ivy. C++ code is built using Cmake and Python uses the traditional distutils/setuptools tools.

Structure of the build

This is an (abbreviated) snapshot of the structure of the filesystem for OMERO:

OMERO_SOURCE_PREFIX
|
|-- build.xml .......................... Top-level build file
|
|-- build.py ........................... Python wrapper to handle OS-specific configuration
|
|-- omero.class ........................ Self-contained Ant launcher
|
|--etc ................................. Configuration folder
|   |-- grid ........................... Deployment files folder
|   |-- ivysettings.xml ................ Main Ivy configuration file
|   |-- hibernate.properties
|   |-- build.properties
|   |-- logback.xml
|   |-- omero.properties
|   \-- profiles
|
|-- examples ........................... User examples
|
\components
  |
  |--<component-name> .................. Each component has this same basic structure.
  |    |-- build.xml ................... Build file
  |    |-- ivy.xml ..................... Jar dependencies
  |    |-- test.xml .................... Test dependencies
  |    |-- src ......................... Source code
  |    |-- resources ................... Other files of interest
  |    |-- test ........................ Test source code and test resources
  |    \-- target ...................... Build output (deleted on clean)
  |
  |  NOTABLE COMPONENTS
  |
  |--tools ............................. Other server-components with special build needs.
  |    |--build.xml .................... Build scripts
  |    |
  |    \--<tool-name>
  |         |--build.xml ............... Build file
  |         \--ivy.xml ................. Jar dependencies
  |
  \--antlib ............................ Special component which is not built, but referenced by the build
      |
      \--resources ..................... Build resources
          |--global.xml ................ Global build properties
          |--hibernate.xml
          |--lifecycle.xml ............. Ivy-related targets
          \--version.xml ............... Version properties

Note

User examples are explained under Working with OMERO

Unfortunately, just the above snapshot of the code repository omits some of the most important code. Many megabytes of source code is generated both by our own DSLTask as well as by the Ice slice2java, slice2cpp, and slice2py code generators. These take an intermediate representation of the OME-Model and generate our OME-Remote Objects. This code is not available in git, but once built, can be found in all the directories named “generated”.

Build tools

Ant

./build.py is a complete replacement for your local ant install. In many cases, you will be fine running ant. If you have any issues (for example OutOfMemory) , please use ./build.py instead. However, only use one or the other; do not mix calls between the two.

The main build targets are defined in the top-level build.xml file. All available targets can be listed using:

./build.py -p

Ivy

The build system uses Ivy 2.3.0 as the dependency manager. The general Ivy configuration is defined in a settings file located under etc/ivysettings.xml.

In order to determine the transitive closure of all dependencies, Ivy resolves each ivy.xml file and stores the resolved artifacts in a cache to speed up other processes. The OMERO build system defines and uses two kinds of caches:

  1. the local dependencies cache under lib/cache is used by most resolvers

  2. Maven resolvers use the Maven cache under ~/.m2/repository

Note

When the Ivy configuration file or the version number is changed, the cache can become stale. Calling ./build.py clean from the top-level build will delete the content of the local cache.

Resolvers are key to how Ivy functions. Multiple dependency resolvers can be defined fine-grained enough to resolve an individual jar in order to pick up the latest version of any library from a repository, a generic URL or from the local file system. Since OMERO 5.1.3, the remote repository resolvers are set up to resolve transitive dependencies.

The OMERO build system uses by default a chain resolver called omero-resolver which resolves the following locations in order:

  1. target/repository which contains most artifacts published by the build system in the install step of the lifecycle

  2. the local dependency repository under lib/repository

  3. the local Maven cache under ~/.m2/repository

  4. the Maven central repository

  5. the OME artifactory

Bio-Formats dependencies are resolved using a specific chain resolver called ome-resolver which resolves the following locations in order:

  1. the local Maven cache under ~/.m2/repository

  2. the OME artifactory

To define its dependencies, each component uses a top-level Ivy file, ivy.xml, for the build and optionally another Ivy file, test.xml, for the tests.

The OMERO build system defines and uses four types of Ivy configurations:

  1. build: defines dependencies to be used for building

  2. server: defines dependencies to be bundled under lib/server

  3. client: defines dependencies to be bundled under lib/client

  4. test: defines dependencies to be used for running the tests

While building, most Java components follow the same lifecycle define in lifecycle.xml. The default dist target for each component calls each of the following steps in order:

  1. retrieve: retrieve the resolved dependencies and copy them under target/libs

  2. prepare: prepare various resources (property files, lib/logback-build.xml)

  3. generate: copy all resources from the previous step for compilation

  4. compile: compile the source files into the destination repository

  5. package-extra: package the sources and the Javadoc into Jar files for publication

  6. package: package the compiled classes into a Jar file for publication

  7. install: convert the component Ivy file into a pom file using makepom and publish the component artifacts

Individual components can override the content of this default lifecycle via their build.xml.

OmeroTools

The Ant build alone is not enough to describe all the products which get built. Namely, the builds for the non-Java components stored under components/tools are a bit more complex. Each tools component installs its artifacts to the tools/target directory which is copied on top of the dist top-level distribution directory.

Jenkins

The OME project currently uses Jenkins as a continuous integration server available here, so many binary packages can be downloaded without compiling them yourself. See the Continuous Integration documentation for further details.

Server build

The default ant target (build-default) will build the OMERO system and copy the necessary components for a binary distribution to the dist directory. Below is a comparison of what is taken from the build, where it is put, and what role it plays in the distribution.

OMERO_SOURCE_PREFIX

OMERO_SOURCE_PREFIX/dist

Comments

components/tools/OmeroCpp/lib*

lib/

Native shared libraries

lib/repository/<some>

lib/client & lib/server

Libraries needed for the build

etc/

etc/

Configuration

sql/*.sql

sql/

SQL scripts to prepare the database

Note

By default, OMERO C++ language bindings are not built. Use build-all for that.

These files are then zipped to OMERO.server-<version>.zip via release-zip

Coupled development

Since OMERO 5.1.3, Bio-Formats is decoupled from the OMERO build system which consumes Bio-Formats artifacts from the OME Maven repository via Ivy.

While this decoupling matches most of the development use cases, it is sometimes necessary to work on coupled Bio-Formats and OMERO branches especially during breaking changes of the OME Data Model or the Bio-Formats API.

The general rule for coupled branches is to build each component in their dependency order and use the local Maven repository under ~/.m2/repository to share artifacts.

Building Bio-Formats

From the top-level folder of the Bio-Formats repository,

  1. if necessary, adjust the version of Bio-Formats which will be built, installed locally and consumed by OMERO e.g. for 5.2.0-SNAPSHOT:

    $ ./tools/bump_maven_version.py 5.2.0-SNAPSHOT

  2. run the Maven command allowing to build and install the artifacts under the local Maven cache:

    $ mvn clean install

Building OMERO

From the top-level folder of the OMERO repository,

  1. in omero-model, adjust the version of ome:formats-gpl in build.gradle to the version chosen for the Bio-Formats build

  2. publish locally the change using gradle publishToMavenLocal