Note
This documentation is for OMERO 5.2. This version is now in maintenance mode and will only be updated in the event of critical bugs or security concerns. OMERO 5.3 is expected before the end of 2016.
Troubleshooting OMERO¶
Which user account and password do I use where?¶
Accounts table, including the example usernames and passwords used in the installation guides:
| Account type | Function | Username | Password |
|---|---|---|---|
| System | Administrator/Root | ||
| System | (Database) service account | postgres | |
| System | (OMERO) service account | omero_user | |
| Database | Database administrator | postgres | |
| Database | Database user | db_user | db_password |
| OMERO | OMERO administrator | root | root_password |
| OMERO | OMERO users |
Note
These example usernames and passwords are provided to help you follow the installation guide examples. Do not use root_password or db_password; substitute your own passwords.
There are a total of three types of user accounts: system, database and OMERO accounts.
System accounts¶
These are accounts on your machine or directory server (e.g. LDAP, Active Directory). One account is used for running the OMERO server (either your own or one you created specially for running OMERO, referred to here as “omero_user”). There is also a user called the “administrator-level user” on the Windows installation page and “root-level user” on the UNIX installation page (which includes Mac OS X). A separate “postgres” user is used for running the database server. The “omero_user” account runs the OMERO server, and owns the files uploaded to OMERO. This account must have permission to write to the /OMERO/ binary repository. Some operations in the install scripts require the root-level/administrator-level privileges in order to use another account to perform particular actions e.g. the “postgres” user to create a database. However the OMERO.server should never be run as the root-level/administrator-level user or as the system-level “postgres” user.
Database accounts¶
The PostgreSQL database system contains user and administrative accounts; these are completely separate from the system accounts, above, existing only inside the database. The database administrative user (“postgres”) is the owner of all the database resources, and can create new users internal to the database. A single database account is used at run time by OMERO to talk to your database. Therefore, you must configure the database values during installation:
$ bin/omero config set omero.db.user db_user
$ bin/omero config set omero.db.pass db_password
Note
Do not use db_user or db_password here; substitute your own username and password.
A database user may have the same name as an account on your machine, in which case a password might not be necessary.
OMERO accounts¶
These accounts only exist inside the OMERO system, and are completely separate from both the system and database accounts, above. The first user which you will need to configure is the “root” OMERO user (different from any root-level Unix account). This is done by setting the password in the database script, see Database tools.
Other OMERO users can be created via the OMERO.web admin tool. None of the passwords have to be the same, in fact they should be different unless you are using the LDAP plugin.
Server fails to start¶
- Check that you are able to successfully connect to your PostgreSQL installation as outlined on the PostgreSQL page for your OS (Windows PostgreSQL page or UNIX/Mac PostgreSQL page).
- Check the permissions on your omero.data.dir (/OMERO or C:OMERO by default) as outlined on the OMERO.server installation page for Unix/Mac users, or the OMERO.server binary repository page for Windows users.
- Are you on a laptop? If you see an error message mentioning “node master couldn’t be reached”, you may be suffering from a network address swap. Ice does not like to have its network changed as can happen if the server is running on a laptop on wireless. If you lose connectivity to icegridnode, you may have to kill it manually via kill PID or killall icegridnode (under Unix).
- If you see an error message mentioning “Freeze::DatabaseException” or “could not lock file: var/registry/__Freeze/lock”, your icegrid registry may have become corrupted. This is not a problem, but it will be necessary to stop OMERO and delete the var/master directory (e.g. rm -rf var/master). When restarting OMERO, the registry will be automatically re-created.
- If you see an error message mentioning “Protocol family unavailable”, you will need to set the Ice.IPv6 property with omero config set Ice.IPv6 0.
- If you upgraded from a 5.0.2 server or older and copied the entire content of the etc/grid directory from the old server to the new server, you will need to revert the changes made to templates.xml to generate the new memory settings.
- If OMERO starts up, but fails to respond to connection requests, check netstat -a for port 4064. If nothing is listening on 4064, you may need to specify which network interface to use. omero config set Ice.Default.Host 127.0.0.1, for example, would force OMERO to only listen on localhost. See Proxy and Endpoint Syntax for more information.
Remote clients cannot connect to OMERO installation¶
The Admin section of OMERO.web appears to work properly and you may or may not have created some users, but no matter what you do remote clients will not speak to OMERO. OMERO.insight gives you an error message similar to the following despite giving the correct username and password:
This is often due to firewall misconfiguration on the machine that runs your OMERO server, affecting the ability of remote clients to locate it. Please see the Server security and firewalls page.
Server crashes with…¶
- X11 connection rejected because of wrong authentication
- X connection to localhost:10.0 broken (explicit kill or server shutdown).
OMERO uses image scaling and processing techniques that may be interfered with when used with SSH X11-forwarding. You should disable SSH X11-forwarding in your SSH session by using the -x flag as follows before you restart the OMERO.server:
ssh -x my_server.examples.com
OutOfMemoryError / PermGen space errors in OMERO.server logs¶
Out of memory or permanent generation (PermGen) errors can be caused by many things. You may be asking too much of the server. Or you may require an increase in the maximum Java heap or the permanent generation space. This can be done by modifying the configuration for your OMERO.server. See Memory configuration.
Similarly, if you are finding out of memory errors in one of the other process logs (e.g. Indexer-0.log or PixelData-0.log), you might try optimizing the JVM memory settings.
Furthermore, under certain conditions access of images greater than 4GB can be problematic on 32-bit platforms due to certain bugs within the Java Virtual Machine including Bug ID: 4724038. A 64-bit platform for your OMERO.server is HIGHLY recommended.
Import errors¶
Import error when running bin/omero¶
Traceback (most recent call last):
File "bin/omero", line 67, in ?
import omero.cli
ImportError: No module named omero.cli
If you get any import related errors while running bin/omero, the most likely cause is that your PYTHONPATH is not properly set.
- If you installed Ice globally via your package manager, make sure you included ice-python.
- If you installed Ice manually, e.g. under /opt/Ice-3.5.1 you need to add /opt/Ice-3.5.1/python (or similar) to your PYTHONPATH environment variable. See the Ice installation instructions for more information.
Too many open files¶
This is most often seen as an error during importing and is caused by the number of opened files exceeding the limit imposed by your operating system. It might be due to OMERO leaking file descriptors; if you are not using the latest version, please upgrade, since a number of bugs which could cause this behavior have been fixed. It is also possible for buggy scripts which do not properly release resources to cause this error.
To view the current per-process limit, run
ulimit -Hn
which will show the hard limit for the maximum number of file descriptors (-Sn will show the soft limit). This limit may be increased. On Linux, see /etc/security/limits.conf (global PAM per-user limits configuration); it is also possible to increase the limit in the shell with
ulimit -n newlimit
providing that you are uid 0 (other users can only increase the soft limit up to the hard limit). To view the system limit, run
cat /proc/sys/fs/file-max
We recommend 8K as a minimum number of files limit for production systems, with 16K being reasonable for bigger machines.
On Mac OS X, the standard ulimit will not work properly. There are several different ways of setting the ulimit, depending upon the version of OS X you are using, but the most common are to edit sysctl.conf or launchd.conf to raise the limit. However, note that both of these methods change the defaults for every process on the system, not just for a single user or service.
Increasing the number of available filehandles via ‘ulimit -n’¶
ValueError: filedescriptor out of range in select() - this is a known issue in Python versions prior to 2.7.0. See #6201 and Python Issue #3392 for more details.
Windows import errors¶
Directory exists but is not registered¶
Import errors of type Directory exists but is not registered: CheckedPath(username_id) suggest a server-side issue under the ManagedRepository.
For production servers, this can be caused by a server crash during import or an issue at the file system level (permissions, renaming). If the ManagedRepository/username_id folder is empty, you should try removing it before trying another import.
For development servers, this may be caused by the binary directory not being cleaned after the database has been wiped.
DropBox fails to start: failed to get session¶
If the main server starts but DropBox fails with the following entry in var/log/DropBox.log,
2011-06-07 03:42:56,775 ERROR [ fsclient.DropBox] (MainThread) Failed to get Session:
then it may be that the server is taking a relatively long time to start.
A solution to this is to increase the number of retries and/or the period (seconds) between retries in etc/grid/templates.xml
<property name="omero.fs.maxRetries" value="5"/>
<property name="omero.fs.retryInterval" value="3"/>
Search does not return expected results¶
If searching for a specific term does not return the expected results (e.g. searching for the name of a tag does not return the full list of a user’s images annotated with that tag), there are a few things that may have gone wrong. See Missing search results for more details.
OMERO.web issues¶
OMERO.web and Apache¶
Do not enable mod_python and mod_wsgi in the same apache process. mod_wsgi will deadlock if run in daemon mode while mod_python is enabled
OMERO.web did not start¶
If the Apache error logs contain lines of type Permission denied: access to xxx denied, you need to check the permissions of the folder and make sure it is readable and executable by the Apache user.
OMERO.web running but status says not started¶
If you upgraded OMERO but forgot to stop OMERO.web, processes will still be running. In order to kill stale processes by hand, run:
$ ps aux | grep /home/omero/OMERO.server/var/django.pid
Note
As Gunicorn is based on the pre-fork worker model it is enough to kill the master process, the one with the lowest PID.
OMERO.web not available HTTP 404¶
Consult nginx error.log for more details.
The most common problem appears when the default configuration for location / is loaded prior to omeroweb.conf
2016/01/01 00:00:00 [error] 1234#0: *5 "/usr/share/nginx/html/webclient/login/index.html" is not found (2: No such file or directory), client: 1.2.3.4, server
OMERO.web not responding/timeout issues¶
[CRITICAL] WORKER TIMEOUT (pid:1234)
OMERO.web deployed with Gunicorn relies on the operating system to provide all of the load balancing while handling requests. Adjust the timeout using omero.web.wsgi_timeout and scale the number of omero.web.wsgi_workers starting with (2 x NUM_CORES) + 1 workers. For more details refer to Gunicorn default configuration (Unix/Linux).
Issues with downloading data from OMERO.web¶
An EXPERIMENTAL: Gunicorn advance configuration (Unix/Linux) is available for testing with nginx if you are encountering problems with downloads failing. You can also configure OMERO.web to limit downloads - refer to the OMERO.web deployment documentation and Download restrictions for further details.
OMERO.web piecharts¶
‘Drive space’ does not generate pie chart or ‘My account’ does not show markup picture and crop the picture options.
Error message says: ‘Piechart could not be displayed. Please check log file to solve the problem’. Please check var/log/OMEROweb.log for more details. There are a few known possibilities:
- ‘TclError: no display name and no $DISPLAY environment variable’. Turn off the compilation of TCL support in Matplotlib.
- ‘ImportError: No module named Image’. Install Pillow (packages should be available for your distribution). Also double check if all of the prerequisites were installed from OMERO.web deployment (UNIX instructions or Windows instructions).
Troubleshooting performance issues with the clients¶
If you are having issues with slowdown and timeouts in the clients, there are three things to check:
- your network connection
- if you are running out of memory (processing large datasets can cause problems)
- whether your server’s HOME directory is on a network share
In the final case, two issues arise. Firstly, when performing some operations, the clients make use of temporary file storage and log directories, as described in the Client configuration section of System requirements. If your home directory is stored on a network, possibly NFS mounted (or similar), then these temporary files are being written and read over the network which can slow access down. Secondly, the OMERO.server also accesses the temporary and log folders of the user the server process is running as. As the server makes heavier use of these folders than the clients, if the OMERO.server user directory is stored on a network drive then slow performance can occur.
To resolve this, define the OMERO_TMPDIR environment variable to point at a temporary directory located on the local file system (e.g. /tmp/omero).
Other issues¶
Connection problems and TCP window scaling on Linux systems¶
Later versions of the 2.6 Linux kernel, specifically 2.6.17, have TCP window scaling enabled by default. If you are having initial logins never timeout or problems with connectivity in general you can try turning the feature off as follows:
# echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
Server or clients print “WARNING: Prefs file removed in background...”¶
Nov 12, 2008 3:02:50 PM java.util.prefs.FileSystemPreferences$7 run
WARNING: Prefs file removed in background /root/.java/.userPrefs/prefs.xml
Nov 12, 2008 3:02:50 PM java.util.prefs.FileSystemPreferences$7 run
WARNING: Prefs file removed in background /usr/lib/jvm/java-1.7.0-icedtea-1.7.0.0/jre/.systemPrefs/prefs.xml
These warnings (also sometimes listed as ERRORS) can be safely ignored, and are solely related to how Java is installed on your system. See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4751177 or this ome-users thread on our mailing list for more information.
Data corruption¶
If you are dealing with a data corruption issue, you may find the information on PixelService resolution order for locating binary data for images useful.