Scaling Omero¶
There are several ways that OMERO, or any server system, can scale. Optimizing your system for more than one of these factors is non-trivial, but we try to lay out some guidelines below for what has worked, what almost certainly will not work, and what – under the right circumstances – might be optimal.
Concurrent invocations¶
The bottlenecks for concurrent invocations are:
- database connections
- server threads
- the router
Database connections¶
Database servers, in general, have a maximum number of allowed
connections. In postgres, the default max_connections is 100, though
in many cases this will be significantly lower due to the available
shared memory (SHMMAX). If OMERO were to use direct connections to the
database, after max_connections invocations, all further attempts to
connect to the server would fail with “too many connection” exceptions.
Instead, OMERO uses a connection pool in front of Postgres, which
manages many more simultaneous attempts to connect to the database.
OMERO’s connection pool size as determined by omero.db.poolsize
should be lower than max_connections.
With the default max_connections set to 64,
it is possible to execute 500 queries simultaneously without database
exceptions. Instead, one receives server exceptions.
Server threads¶
In OMERO.blitz, too many (500+ on the default
configuration) simultaneous invocations will result in
ConnectionLost exceptions. We are currently working on ways to
extend the number of single invocations on one server, but a simpler
solution is to start another OMERO.blitz server.
Total throughput¶
The bottlenecks for throughput are:
- maximum message size
- server memory
- IO
- network
See also
- OMERO.server and PostgreSQL
- Instructions about OMERO.server and PostgreSQL under UNIX and UNIX-like platforms.
