Build System¶
Overview
The page goes into details about how the build system is configured.
Since 5.5, OMERO decouples many components and uses, for some components, an Gradle-based build. The two overarching repositories are omero-build and omero-gradle-plugins. See the README of each repository for more details. OMERO still uses an Ant-based build, for some components, with dependency management provided by Ivy. C++ code is built using Cmake and Python uses the traditional distutils/setuptools tools.
Structure of the build¶
This is an (abbreviated) snapshot of the structure of the filesystem for OMERO:
OMERO_SOURCE_PREFIX
|
|-- build.xml .......................... Top-level build file
|
|-- build.py ........................... Python wrapper to handle OS-specific configuration
|
|-- omero.class ........................ Self-contained Ant launcher
|
|--etc ................................. Configuration folder
| |-- grid ........................... Deployment files folder
| |-- ivysettings.xml ................ Main Ivy configuration file
| |-- hibernate.properties
| |-- build.properties
| |-- logback.xml
| |-- omero.properties
| \-- profiles
|
|-- examples ........................... User examples
|
\components
|
|--<component-name> .................. Each component has this same basic structure.
| |-- build.xml ................... Build file
| |-- ivy.xml ..................... Jar dependencies
| |-- test.xml .................... Test dependencies
| |-- src ......................... Source code
| |-- resources ................... Other files of interest
| |-- test ........................ Test source code and test resources
| \-- target ...................... Build output (deleted on clean)
|
| NOTABLE COMPONENTS
|
|--tools ............................. Other server-components with special build needs.
| |--build.xml .................... Build scripts
| |
| \--<tool-name>
| |--build.xml ............... Build file
| \--ivy.xml ................. Jar dependencies
|
\--antlib ............................ Special component which is not built, but referenced by the build
|
\--resources ..................... Build resources
|--global.xml ................ Global build properties
|--hibernate.xml
|--lifecycle.xml ............. Ivy-related targets
\--version.xml ............... Version properties
Note
User examples are explained under Working with OMERO
Unfortunately, just the above snapshot of the code repository omits some
of the most important code. Many megabytes of source code is generated both by
our own DSLTask as well as by
the Ice slice2java
, slice2cpp
, and
slice2py
code generators. These take an intermediate representation
of the OME-Model and generate our OME-Remote Objects.
This code is not available in git, but once built, can be found in all the
directories named “generated”.
Build tools¶
Ant¶
./build.py
is a complete replacement for your local ant install. In
many cases, you will be fine running ant. If you have any issues
(for example OutOfMemory
) , please use ./build.py
instead. However,
only use one or the other; do not mix calls between the two.
The main build targets are defined in the top-level build.xml
file.
All available targets can be listed using:
./build.py -p
Ivy¶
The build system uses Ivy 2.3.0 as the dependency manager. The general Ivy configuration is defined in a settings file located under etc/ivysettings.xml.
In order to determine the transitive closure of all dependencies, Ivy resolves
each ivy.xml
file and stores the resolved artifacts in a
cache to speed up other processes. The
OMERO build system defines and uses two kinds of caches:
the local dependencies cache under
lib/cache
is used by most resolversMaven resolvers use the Maven cache under
~/.m2/repository
Note
When the Ivy configuration file or the version number is changed, the
cache can become stale. Calling ./build.py clean
from the top-level
build will delete the content of the local cache.
Resolvers are key to how Ivy functions. Multiple dependency resolvers can be defined fine-grained enough to resolve an individual jar in order to pick up the latest version of any library from a repository, a generic URL or from the local file system. Since OMERO 5.1.3, the remote repository resolvers are set up to resolve transitive dependencies.
The OMERO build system uses by default a
chain resolver called omero-resolver
which
resolves the following locations in order:
target/repository
which contains most artifacts published by the build system in the install step of the lifecyclethe local dependency repository under
lib/repository
the local Maven cache under
~/.m2/repository
the OME artifactory
Bio-Formats dependencies are resolved using a specific
chain resolver called ome-resolver
which
resolves the following locations in order:
the local Maven cache under
~/.m2/repository
the OME artifactory
To define its dependencies, each component uses a top-level
Ivy file, ivy.xml
, for the build and
optionally another Ivy file, test.xml
, for the tests.
The OMERO build system defines and uses four types of Ivy configurations:
build: defines dependencies to be used for building
server: defines dependencies to be bundled under
lib/server
client: defines dependencies to be bundled under
lib/client
test: defines dependencies to be used for running the tests
While building, most Java components follow the same lifecycle define in lifecycle.xml. The default dist target for each component calls each of the following steps in order:
retrieve: retrieve the resolved dependencies and copy them under
target/libs
prepare: prepare various resources (property files, lib/logback-build.xml)
generate: copy all resources from the previous step for compilation
compile: compile the source files into the destination repository
package-extra: package the sources and the Javadoc into Jar files for publication
package: package the compiled classes into a Jar file for publication
install: convert the component Ivy file into a pom file using makepom and publish the component artifacts
Individual components can override the content of this default lifecycle via
their build.xml
.
OmeroTools¶
The Ant build alone is not enough to describe all the products which get
built. Namely, the builds for the non-Java components stored under
components/tools are a bit more complex. Each tools component
installs its artifacts to the tools/target directory which is copied on top of the dist
top-level distribution directory.
Jenkins¶
The OME project currently uses Jenkins as a continuous integration server available here, so many binary packages can be downloaded without compiling them yourself. See the Continuous Integration documentation for further details.
Server build¶
The default ant target (build-default
) will build the OMERO system and
copy the necessary components for a binary distribution to the dist
directory. Below is a comparison of what is taken from the build, where
it is put, and what role it plays in the distribution.
OMERO_SOURCE_PREFIX |
OMERO_SOURCE_PREFIX/dist |
Comments |
---|---|---|
components/tools/OmeroCpp/lib* |
|
Native shared libraries |
lib/repository/<some> |
|
Libraries needed for the build |
etc/ |
|
Configuration |
|
|
SQL scripts to prepare the database |
Note
By default, OMERO C++ language bindings are not built. Use build-all
for that.
These files are then zipped to OMERO.server-<version>.zip via release-zip
Coupled development¶
Since OMERO 5.1.3, Bio-Formats is decoupled from the OMERO build system which consumes Bio-Formats artifacts from the OME Maven repository via Ivy.
While this decoupling matches most of the development use cases, it is sometimes necessary to work on coupled Bio-Formats and OMERO branches especially during breaking changes of the OME Data Model or the Bio-Formats API.
The general rule for coupled branches is to build each component in their
dependency order and use the local Maven repository under ~/.m2/repository
to share artifacts.
Building Bio-Formats¶
From the top-level folder of the Bio-Formats repository,
if necessary, adjust the version of Bio-Formats which will be built, installed locally and consumed by OMERO e.g. for 5.2.0-SNAPSHOT:
$ ./tools/bump_maven_version.py 5.2.0-SNAPSHOT
run the Maven command allowing to build and install the artifacts under the local Maven cache:
$ mvn clean install
Building OMERO¶
From the top-level folder of the OMERO repository,
in omero-model, adjust the version of
ome:formats-gpl
in build.gradle to the version chosen for the Bio-Formats buildpublish locally the change using
gradle publishToMavenLocal