OMERO.web Nginx and Gunicorn deployment (Unix/Linux)
====================================================

.. note:: Since OMERO 5.2, the OMERO.web framework no longer bundles
      a copy of the Django package, instead manual installation of
      the Django dependency is required. It is highly recommended to use
      `Django 1.8`_ (LTS) which requires Python 2.7. For more information
      see :ref:`python-requirements` on the
      :doc:`/sysadmins/version-requirements` page.


OMERO.web uses the
`Web Server Gateway Interface (WSGI) <http://wsgi.readthedocs.org/en/latest/learn.html>`_
as a primary deployment platform. The Python standard
`PEP 3333 <https://www.python.org/dev/peps/pep-3333/>`_
that describes how a web server communicates with web applications.



.. _gunicorn_default_configuration:

Gunicorn default configuration (Unix/Linux)
-------------------------------------------

Install `Django 1.8`_ and `Gunicorn >= 19.3 <http://gunicorn.org>`_
using the package requirements file:

::

   $ pip install -r share/web/requirements-py27-nginx.txt

.. note:: For more details refer to
      :djangodoc:`how to install Django 1.8 <topics/install/#install-the-django-code>`
      or :djangodoc:`upgrade Django to 1.8 <topics/install/#remove-any-old-versions-of-django>`.


Additional settings can be configured by changing the following properties:

- :property:`omero.web.application_server.max_requests` to 500

- :property:`omero.web.wsgi_workers` to (2 x NUM_CORES) + 1

  .. note::
      **Do not** scale the number of workers to the number of clients
      you expect to have. OMERO.web should only need 4-12 worker
      processes to handle many requests per second.

- :property:`omero.web.wsgi_args` Additional arguments. For more details
  check `Gunicorn Documentation <http://docs.gunicorn.org/en/stable/settings.html>`_.

.. _nginx_gunicorn_wsgi_configuration:

Nginx configuration (Unix/Linux)
--------------------------------

If you have installed Nginx, OMERO can automatically generate a
configuration file for your web server. The location of the file
will depend on your system, please refer to your web server's manual.
See :ref:`customizing_your_omero_web_installation_unix`
for additional customization options.

Set the following::

    $ bin/omero config set omero.web.application_server "wsgi-tcp"

To create a site configuration file for inclusion in a system-wide nginx
configuration redirect the output of the following command into a file:

::

    $ bin/omero web config nginx

.. literalinclude:: nginx-omero.conf

.. note::
    OMERO.web requires ``body_in_file_only`` adjusted in your default nginx
    config because nginx must buffer incoming data. Make sure you have that
    set to the following config:

    ::

        http {
            ...
            sendfile on;
            send_timeout 60s;
            client_max_body_size 0;
            ...
        }

To configure an HTTPS server follow
`the nginx documentation <http://nginx.org/en/docs/http/configuring_https_servers.html>`_.

Running OMERO.web (Unix/Linux)
------------------------------

Start the Gunicorn worker processes listening by default on 127.0.0.1:4080:

::

    $ bin/omero web start
    ... static files copied to '/home/omero/OMERO.server/lib/python/omeroweb/static'.
    Starting OMERO.web... [OK]

The Gunicorn workers are managed **separately** from other OMERO.server
processes. You can check their status or stop them using the
following commands:

::

    $ bin/omero web status
    OMERO.web status... [RUNNING] (PID 59217)
    $ bin/omero web stop
    Stopping OMERO.web... [OK]
    Django WSGI workers (PID 59217) killed.

.. _download_limitation:

Download limitations
^^^^^^^^^^^^^^^^^^^^

In order to offer users the ability to download data from OMERO.web you have
to deploy using :ref:`gunicorn_async_configuration`.
OMERO.web is able to handle multiple clients on a single worker
thread switching context as necessary while streaming binary data from
OMERO.server. Depending on the traffic and scale of the repository you should
configure connections and speed limits on your server to avoid blocking
resources. We recommend you run benchmark and performance tests.
It is also possible to apply :ref:`download_restrictions` and
offer alternative access to binary data.

.. note::
    Handling streaming request/responses requires proxy buffering
    to be turned off. For more details refer to
    `Gunicorn deployment <http://docs.gunicorn.org/en/stable/deploy.html>`_
    and
    `Nginx configuration <http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering>`_.

.. note::
    :property:`omero.web.application_server.max_requests` should be set to 0


.. _nginx_wsgi_benchmark:

Benchmark
---------

We run example benchmarks on rendering thumbnails and 512x512 pixels planes
for 100 concurrent users making 5000 requests in total::

    $ ab -n 5000 -c 100 https://server.openmicroscopy.org/omero/webclient/render_thumbnail/size/96/1234/
    This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/

    Server Software:        nginx/1.9.9
    Server Hostname:        server.openmicroscopy.org
    Server Port:            80

    Document Path:          /omero/webclient/render_thumbnail/size/96/31851/
    Document Length:        1880 bytes

    Concurrency Level:      100
    Time taken for tests:   224.488 seconds
    Complete requests:      5000
    Failed requests:        0
    Write errors:           0
    Total transferred:      10450000 bytes
    HTML transferred:       9400000 bytes
    Requests per second:    22.27 [#/sec] (mean)
    Time per request:       4489.763 [ms] (mean)
    Time per request:       44.898 [ms] (mean, across all concurrent requests)
    Transfer rate:          45.46 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   0.3      0       3
    Processing:   435 4446 685.2   4363    7644
    Waiting:      432 4446 685.3   4362    7644
    Total:        435 4446 685.1   4363    7644

    Percentage of the requests served within a certain time (ms)
      50%   4363
      66%   4553
      75%   4670
      80%   4750
      90%   5072
      95%   5398
      98%   6795
      99%   6955
     100%   7644 (longest request)


::

    $ ab -n 5000 -c 100 http://server.openmicroscopy.org/omero/webclient/render_image/1234/20/0/
    This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/

    Server Software:        nginx/1.9.9
    Server Hostname:        server.openmicroscopy.org
    Server Port:            80

    Document Path:          /omero/webclient/render_image/1234/20/0/
    Document Length:        24293 bytes

    Concurrency Level:      100
    Time taken for tests:   247.154 seconds
    Complete requests:      5000
    Failed requests:        0
    Write errors:           0
    Total transferred:      122515000 bytes
    HTML transferred:       121465000 bytes
    Requests per second:    20.23 [#/sec] (mean)
    Time per request:       4943.080 [ms] (mean)
    Time per request:       49.431 [ms] (mean, across all concurrent requests)
    Transfer rate:          484.09 [Kbytes/sec] received

    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   0.4      0       4
    Processing:   482 4898 855.3   4737    8303
    Waiting:      476 4898 855.3   4737    8303
    Total:        482 4898 855.2   4737    8303

    Percentage of the requests served within a certain time (ms)
      50%   4737
      66%   5041
      75%   5250
      80%   5397
      90%   5862
      95%   6621
      98%   7301
      99%   8062
     100%   8303 (longest request)


.. _nginx_troubleshooting:

Troubleshooting
---------------

In order to identify why OMERO.web is not available run:

::

    $ bin/omero web status

Then consult nginx :file:`error.log` and :file:`OMERO.server/var/log/OMEROweb.log`

For more details check :ref:`troubleshooting-omeroweb`.


Debugging
^^^^^^^^^

To run the WSGI server in debug mode, enable
`Gunicorn logging <http://docs.gunicorn.org/en/stable/settings.html#logging>`_
using :property:`omero.web.wsgi_args`:

::

    $ bin/omero config set omero.web.wsgi_args -- "--log-level=DEBUG --error-logfile=/home/omero/OMERO.server/var/log/error.log".


.. _gunicorn_advance_configuration:

EXPERIMENTAL: Gunicorn advance configuration (Unix/Linux)
---------------------------------------------------------

.. note:: Experimental configurations are not ready for production use.
    Configuration may change. Features may not work properly.


OMERO.web deployment can be configured with sync and async workers.
Sync workers are faster and recommended for a data repository with
:ref:`download_restrictions`. If you wish to offer users the ability
to download data then you have to use async workers; read more about
:ref:`download_limitation`.

For more details refer to
`Gunicorn design <http://docs.gunicorn.org/en/stable/design.html>`_.


.. _gunicorn_sync_configuration:

EXPERIMENTAL: Sync workers
^^^^^^^^^^^^^^^^^^^^^^^^^^

.. note:: Experimental configurations are not ready for production use.
    Configuration may change. Features may not work properly.


- Install `futures <https://pypi.python.org/pypi/futures>`_ ::

      $ pip install futures

Additional settings can be configured by changing the following properties:

- The number of worker threads for handling requests, see
  `Gunicorn threads <http://docs.gunicorn.org/en/stable/settings.html#threads>`_ ::

      $ bin/omero config set omero.web.wsgi_worker_class
      $ bin/omero config set omero.web.wsgi_threads $(2-4 x NUM_CORES)


.. _gunicorn_async_configuration:

EXPERIMENTAL: Async workers
^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. note:: Experimental configurations are not ready for production use.
    Configuration may change. Features may not work properly.


- Install `Gevent >= 0.13 <http://www.gevent.org/>`_ ::

      $ pip install "gevent>=0.13"

Additional settings can be configured by changing the following properties:

- The maximum number of simultaneous clients, see
  `Gunicorn worker-connections <http://docs.gunicorn.org/en/stable/settings.html#worker-connections>`_ ::

      $ bin/omero config set omero.web.wsgi_worker_class gevent
      $ bin/omero config set omero.web.wsgi_worker_connections 1000
      $ bin/omero config set omero.web.application_server.max_requests 0