Running and writing tests¶
The following guidelines apply to tests in both the Java and Python test components. However, some of the presented options apply to only one or the other.
The default build target does not compile all the required testing resources. You should run test-compile (or build-dev if you are using Eclipse) first:
./build.py build-default test-compile
You must rebuild the test-compile target if you subsequently modify any of the Java tests.
Note
The OMERO C++ components and tests are under heavy development, and are not compiled or run by the targets mentioned on this page.
Running tests¶
Running unit tests¶
The unit testing framework is fairly simple. Only methods which contain logic written within OMERO are tested. This means that framework functionality like remoting or the Hibernate layer is not tested. This is a part of integration testing (see below).
You can run the unit tests for any component from its directory by entering:
./build.py -f components/<component>/build.xml test
The same can be done for all components using:
./build.py test-unit
Note that for tests written in Python the package pytest must be installed, see Writing Python tests. Also note that some Python tests are excluded by default, see Using markers in OmeroPy tests for more details.
Running integration tests¶
Integration testing is a bit more complex because of the reliance on a database, which is not easily mockable. All Hibernate-related classes are tested in integration mode.
The tests require a fast computer. Running all the integration tests places several restrictions on the environment:
- There must be a running OMERO database.
- An OMERO.server instance must be running.
Integration tests assume that:
ICE_CONFIG
has been properly set. The contents of theetc/ice.config
file should be enough to configure a running server for integration testing. This means that code creating a client connection as outlined in Developing OMERO clients should execute without errors.- An OMERO.server instance is running on the host and port specified in
the
ICE_CONFIG
file.
If any of the tests fail with a user authentication exception (or
omero.client
throws an exception), a new ice.config
file can be
created and pointed to by the ICE_CONFIG
environment variable.
Most likely the first settings that will have to be put there will be
omero.user
and omero.pass
.
Running all tests¶
To run all the integration tests, use
./build.py test-integration
Note that some Python tests are excluded by default, see Using markers in OmeroPy tests for more details.
Component tests¶
Running an integration test suite for an individual component can be done explicitly via:
./build.py -f components/<component>/build.xml integration
Results are placed in components/<component>/target/reports
.
Individual tests¶
Alternatively, you can run individual tests which you may currently be
working on. This can be done using the test
target. For example:
./build.py -f components/tools/OmeroJava/build.xml test -DTEST=integration/chown/PermissionsTest
./build.py -f components/tools/OmeroPy/build.xml test -DTEST=test/integration/test_admin.py
Warning
Some integration tests leak file descriptors. If many tests are run
then they may start to fail after the system’s open files limit is
reached. Depending on your system the limit may be checked or
adjusted using ulimit -n and /etc/login.conf
or
/etc/security/limits.conf
.
Running Java tests¶
Individual test class methods¶
Individual OmeroJava test class methods (or a comma-separated list of
methods) can be run using the -DMETHODS
parameter together with
the test
target. The test method must be provided in the fully
qualified name form (-Dpackage.class.method
).
./build.py -f components/tools/OmeroJava/build.xml test -DMETHODS=integration.chgrp.AnnotationMoveTest.testMoveTaggedImage
Individual test groups¶
To run individual OmeroJava test groups (or comma-separated sets of groups)
of tests, the -DGROUPS
parameter can be used together with the
test
target
./build.py -f components/tools/OmeroJava/build.xml test -DGROUPS=integration
Using Eclipse to run tests¶
To facilitate importing OMERO components into Eclipse, there are
.project
and .classpath-template
files stored in each
component directory (e.g. common’s
.classpath and common’s
.project).
There are also top-level .classpath
and .project
files which
allow for importing all components as a single project, but this approach
requires more memory and does not clearly differentiate the classpaths, and
so can lead to confusion.
Before importing any component as a project into Eclipse, a successful build has to have taken place:
./build.py
This is for two reasons. Firstly, the Eclipse projects are not configured to perform the code generation needed. The build.py command creates the directory:
<component>/target
which will be missing from any Eclipse project you open before building the source.
Secondly, Ivy is used to copy all the jar dependencies from
OMERO_SOURCE_PREFIX/lib/repository
to <component>/target/libs
, which
is then used in the Eclipse .classpath
files.
If Eclipse ever gets out of sync after the first build, ./build.py build-eclipse can be used to quickly synchronize.
A prerequisite of running unit and integration tests in the Eclipse UI is having the TestNG plug-in installed and working (help available on the TestNG site).
Running the unit tests under Eclipse requires no extra settings and is as easy as navigating to the package or class context menu Run As or Debug As, then selecting TestNG.
Integration tests require the ICE_CONFIG
environment variable to
be available for the Eclipse-controlled JVM. This can be done by editing
Debug/Run configurations in Eclipse. After navigating to the Debug (or Run)
Configurations window, the Environment tab needs to be selected.
After clicking New, ICE_CONFIG
can be defined as a
path to the ice.config
file. This setting needs to be defined per
package, class or method.
By using the “debug” target from templates.xml, it is possible to have OMERO listen on port 8787 for a debugging connection.
bin/omero admin stop
bin/omero admin start debug
Then in Eclipse, you can create a new “Debug” configuration by clicking on Remote Java Application, and setting the port to 8787. These values are arbitrary and can be changed locally.
Keep in mind:
- The server will not start up until you have connected with Eclipse. This is due to the “suspend=y” clause in templates.xml. If you would like the server to start without you connecting, use “suspend=n”.
- If you take too much time examining your threads, your calls may throw timeout exceptions.
Running Python tests¶
Using markers in OmeroPy tests¶
Tests under OmeroPy can be included or excluded according to markers defined
in the tests.
This can be done by using the -DMARK
option. For example, to run all
the integration tests marked as broken
:
./build.py -f components/tools/OmeroPy/build.xml integration -DMARK=broken
By default tests marked as broken
are excluded so
the following two builds are equivalent:
./build.py -f components/tools/OmeroPy/build.xml integration
./build.py -f components/tools/OmeroPy/build.xml integration -DMARK="not broken"
In order to run all tests, including broken
,
an empty marker must be used:
./build.py -f components/tools/OmeroPy/build.xml integration -DMARK=
See also
Running tests directly¶
When writing tests it can be more convenient, flexible and powerful to run the tests from components/tools/OmeroPy using setup.py test. Since Python is interpreted, tests can be written and then run without having to rebuild or restart the server. A few basic options are shown below.
-
-t
<test_path>
,
--test-path
<test_path>
¶ This option specifies the test suite to run. For instance to run a single test file:
cd components/tools/OmeroPy ./setup.py test -t test/integration/test_admin.py
Or to run all tests under a given folder:
cd components/tools/OmeroPy ./setup.py test -t test/integration/clitest
-
-k
<string>
¶ This option will run all integration tests containing the given string in their names. For example, to run all the tests under
test/integration
with permissions in their names:./setup.py test -t test/integration -k permissions
This option can also be used to run a named test within a test module:
./setup.py test -t test/integration/test_admin.py -k testGetGroup
-
-m
<marker>
¶ This option will run integration tests depending on the markers they are decorated with. Available markers can be listed using the
setup.py test --markers
option. For example, to run all integration tests excluding those decorated with the marker broken:./setup.py test -t test/integration -m "not broken"
-
--markers
¶
This option lists available markers for decorating tests:
./setup.py test --markers
-
-s
¶
This option allows the standard output to be shown on the console:
./setup.py test -t test/integration/test_admin.py -s
-
-h
,
--help
¶
This option displays the full list of available options:
./setup.py test -h
To make use of the more advanced options available in pytest that are not
accessible using setup.py test, the py.test script can
be used directly. To use this PYTHONPATH
must contain the path to
the OMERO Python libraries, see OMERO Python language bindings as well as the path to the
OMERO Python test library.
Alternatively, the pytest plugin pytest-pythonpath can be used to
add paths to PYTHONPATH
specifically for pytest.
-
--repeat
<number>
¶ This option allows to repeat tests for number occurences:
py.test --repeat 20 test/unit/fstest
-
-h
,
--help
¶
This option displays the full list of options:
py.test --help
and https://pytest.org/latest/usage.html for more help in running tests.
Failing tests¶
The test.with.fail
ant property is set to false
by default,
which prevents test failures from failing the build. However, it can instead
be set to true
to allow test failures to fail the build. For example:
./build.py -Dtest.with.fail=true integration
Some components might provide individual targets for specific tests (e.g.
OmeroJava provides the broken
target for running broken tests).
The build.xml
file is the reference in each component.
Writing tests¶
Writing Java tests¶
For more information on writing tests in general see https://testng.org. For a test to be an “integration” test, place it in the “integration” TestNG group. If a test is temporarily broken, add it to the “broken” group:
@Test(groups = {"integration", "broken"}
public void testMyStuff() {
}
Tests should be of the Acceptance Test form. The ticket number for which a test is being written should be added in the TestNG annotation:
@Test(groups = "ticket:60")
This works at either the method level (see SetsAndLinksTest.java) or the class level (see UniqueResultTest.java).
The tests under components/tools/OmeroJava/test will be the starting point for most Java-client developers coming to OMERO. An example skeleton for an integration test looks similar to
@Test(groups = "integration")
public class MyTest {
omero.client client;
@BeforeClass
protected void setup() throws Exception {
client = new omero.client();
client.createSession();
}
@AfterClass
protected void tearDown() throws Exception {
client.closeSession();
}
@Test
public void testSimple() throws Exception {
client.getSession().getAdminService().getEventContext();
}
}
Writing Python tests¶
To write and run Python tests you first need to install pytest:
pip install pytest
For more information on writing tests in general see https://pytest.org.
Similar to the OmeroJava tests, the tests under
components/tools/OmeroPy/test,
components/tools/OmeroFS/test and
components/tools/OmeroWeb/test will be the starting point
for most Python-client developers coming to OMERO. Integration tests should
be placed under the integration
subfolders. The file names must begin
with test_ for the tests to be found by pytest.
import omero.clients
class TestExample(object)
def setup_method(self, method):
client = new omero.client()
client.createSession()
def teardown_method(self, method):
client.closeSession()
def testSimple():
ec = client.getSession().getAdminService().getEventContext()
assert ec, "No EventContext!"
Marking OmeroPy tests¶
Methods, classes and functions can be decorated with pytest markers to allow for the selection of tests. pytest provides some predefined markers and markers can be simply defined as they are used. However, to centralize the use of custom markers they should be defined in components/tools/pytest.ini.
To view all available markers the setup.py test --markers
option can
be used with setup.py test or py.test as detailed in
Running tests directly.
There is one custom marker defined:
- broken
- Used to mark broken tests. These are tests that fail consistently with no obvious quick fix. Broken tests are excluded from the main integration builds and instead are run in a separate daily build. broken markers should have a reason, an associated Trac ticket number or both. If there are multiple associated tickets then a comma-separated list should be used.
import pytest
class TestExample2(object):
@pytest.mark.broken(reason="Asserting false", ticket="12345,67890")
def testBroken():
assert False, "Bound to fail"
Using the Python test library¶
The OMERO Python test library
defines an abstract ITest
class that implements the connection set up as
well as many methods shared amongst all Python integration tests.
Each concrete instance of the ITest
will initiate a connection to the
server specified by the ICE_CONFIG
environment variable at the
setup_class()
level. The following objects are created by
ITest.setup_class()
and shared by all test methods of this class:
self.root
is a client for the root userself.group
is a new group which permissions are set toITest.DEFAULT_PERMS
by default. OverridingDEFAULTS_PERMS
in a subclass ofITest
means the group will be created with the new permissions.self.user
is a new user and member ofself.group
self.client
is a client for theself.user
created at class setup.
Additionally, for the self.client
object, different shortcuts are available:
self.sf
is the non-root client sessionself.update
is the update service for the non-root client sessionself.query
is the query service for the non-root client sessionself.ctx
is the event context for the non-root client session. Note this corresponds to the context at creation time and should be refreshed if the context is modified.
The example below inherits the ITest
class and would create a read-write
group by default
from omero.testlib import ITest
class TestExample(ITest):
DEFAULT_PERMS = 'rwrw--' # Override default permissions
def test1():
doAction(self.sf)
New user and groups can be instantiated by individual tests using the
ITest.new_user()
and ITest.new_group()
methods:
def testNewGroupOwner():
new_group = self.new_group(perms='rwa---')
new_owner = self.new_use(group=new_group, owner=True)
assert new_owner.id.val, "No EventContext!"
New clients can be instantiated by individual tests using the
ITest.new_client()
or ITest.new_client_and_user()
methods:
def testNewClient():
new_client = self.new_user_and_client()
ec = new_client.getSession().getAdminService().getEventContext()
assert ec, "No EventContext!"
Images can be imported using the ITest.import_fake_file()
method:
def testFileset():
# 2 images sharing a fileset
images = self.import_fake_file(2)
assert len(images) == 2
Writing OMERO.web tests¶
For OMERO.web integration tests, the OMERO.web test library
defines an abstract IWebTest
class that inherits from ITest
and
also implements Django clients at the class setup using the
Django testing tools.
On top of the elements created by ITest.setup_class()
, the IWebTest
class creates:
self.django_root_client
is a Django test client for the root userself.django_client
is a client for the new user created at the class setup.
from omeroweb.testlib import IWebTest
class TestExample(IWebTest):
def testSimple():
self.django_client.post('/login/', {'username': 'john'})
New Django test clients can be instantiated by individual tests using the
IWebTest.new_django_client()
method:
def testNewDjangoClient():
new_user = self.new_user()
omeName = new_user.omeName.val
new_django_client = self.new_django_client(omeName, omeName)
See also
- test_simple.py
- Example test class using the OMERO.web test library methods