Note
This documentation is for OMERO 5.2. This version is now in maintenance mode and will only be updated in the event of critical bugs or security concerns. OMERO 5.3 is expected in the first quarter of 2017.
DropBox was originally designed as the first stage of the file system changes referred to as OMERO.fs. It utilizes a file system monitor to find newly uploaded files and run a fully automatic import on those files if possible. This release of OMERO.dropbox runs on the same machine as the OMERO.server and watches designated areas of the local filesystem for new or modified files. If those files are importable, then an automatic import is initiated. OMERO.dropbox is started automatically when the OMERO.server starts and it will run if the prerequisites below are met.
In addition to the general System requirements OMERO.dropbox has the following more specific requirements:
Note
It is likely, but not guaranteed, that DropBox will run on later versions of Windows, however only tested versions are supported by default. Setting the property omero.fs.platformCheck to False in etc/grid/templates.xml or using omero config set and restarting the server will allow DropBox to be started on an untested version of Windows.
If you do successfully run DropBox on an as yet untested version of Windows please do let us know via the forums and mailing lists.
In its default configuration the monitored area of the file system is a DropBox subdirectory of the OmeroBinaryRepository directory. The system administrator should create DropBox and then under that a directory for each user, using their omero username. The ownership and permissions should be set so that a user can copy files into their DropBox directory:
/OMERO/DropBox/amy
/emily
/edgar
/root
/zak
Experimenters can add subdirectories under their named directory for convenience. Copying or moving a file of an importable file type into a named directory or nested subdirectory will initiate an automatic import of that file for that user. Multi-file formats will be imported after the last required file of a set is copied into the directory. Images and plates will be imported into the group the user was last logged into, with images placed into Orphaned images.
Acquisition systems can then be configured to drop a user’s images into a given DropBox.
Note
Changing the permissions of a directory within DropBox may result in duplicate imports as a newly readable directory appears identical to a new directory. If directories need to be modified it is recommended that the DropBox system is stopped and then restarted around any changes, as below.
$ bin/omero admin ice server disable DropBox
$ bin/omero admin ice server stop DropBox
$ bin/omero admin ice server disable MonitorServer
$ bin/omero admin ice server stop MonitorServer
# make any directory changes
$ bin/omero admin ice server enable MonitorServer
$ bin/omero admin ice server enable DropBox
Note
Any new files copied into DropBox during this disabled period will not be detected and thus not imported.
The log files var/log/FileServer.log, var/log/MonitorServer.log and var/log/DropBox.log will indicate success or otherwise of start-up of the two components. Once running, var/log/MonitorServer.log will log file events seen within designated file areas and var/log/DropBox.log will log the progress of any file imports.
If file or path names contain Unicode characters this can cause DropBox to fail. This can be remedied by the use of a sitecustomize.py or usercustomize.py file containing the following:
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
For more details on using customization files in Python see: site — Site-specific configuration hook. For more discussion on this issue within OMERO see the forum post: Dropbox halts on certain unicode characters.
Note
If a customization file is used and the OMERO server is upgraded please ensure the file is still available to DropBox after the upgrade.
OMERO.dropbox can be configured in several ways through etc/grid/templates.xml. In its default configuration, as detailed above, it monitors the subdirectory DropBox of the OMERO data directory for all users.
A number of the properties in templates.xml accept a semi-colon separated list of values. This extended configuration allows a site to watch multiple directories, and configure each for a different user, a different type of file, etc. Any value missing from the configuration (e.g. value="1;;2") will be replaced by the default value.
One example alternative configuration would be to watch specific directories for specific users. In the example below two directories are monitored, one for user amy and one for zak:
<property name="omero.fs.importUsers" value="amy;zak"/>
<property name="omero.fs.watchDir" value="/home/amy/myData;/home/zak/work/data"/>
The remaining properties have been left at their default values for both users.
To limit DropBox to import only files belonging to specific image types the following property can be set,
<property name="omero.fs.readers" value="/home/amy/my_readers.txt;"/>
Here only the image types listed in my_readers.txt will be imported for the user amy while the system-wide readers.txt will be used for zak.
For a full description of the properties see below.
Each property takes the form of a single item or a semi-colon separated list of items. Where the item is a list, values within that list should be comma separated.
importUsers
The importUsers is either default or a list of OMERO user names. In the case of the value being default, the same configuration is applied to all users and each subsequent configuration setting should be a single value. In the case of this value being a list of users, each subsequent value should be a list of the same length as the number of users. The default value is default.
<property name="omero.fs.importUsers" value="default"/>
watchDir
The absolute directory path of interest for each user. The default is empty.
<property name="omero.fs.watchDir" value=""/>
eventTypes
For automatic import Creation and Modification events are monitored. It is also possible to monitor Deletion events though these are not used by DropBox. The default is Creation,Modification.
<property name="omero.fs.eventTypes" value="Creation,Modification"/>
pathMode
By default existing and newly created subdirectories are monitored. It is possible to restrict monitoring to a single directory (“Flat”), only existing subdirectories (“Recurse”), or all subdirectories (“Follow”). For DropBox to function correctly the mode should be Follow. The default is Follow.
<property name="omero.fs.pathMode" value="Follow"/>
whitelist
A list of file extensions of interest. An empty list implies all file extensions are monitored. The default is an empty list.
<property name="omero.fs.whitelist" value=""/>
blacklist
A list of subdirectories to ignore. Not currently supported.
<property name="omero.fs.blacklist" value=""/>
timeout
This timeout in seconds is used by one-shot monitors. This property is not used by DropBox.
property name="omero.fs.timeout" value="0.0"/>
blockSize
The number of events that should be propagated to DropBox in one go. Zero implies all events possible. The default is zero.
<property name="omero.fs.blockSize" value="0"/>
ignoreSysFiles
If this is True events concerning system files, such as filenames beginning with a dot or default new folder names, are ignored. The exact events ignored will be OS-dependent. The default is True.
<property name="omero.fs.ignoreSysFiles" value="True"/>
ignoreDirEvents
If this is True then the creation and modification of subdirectories is not reported to DropBox. The default is True.
<property name="omero.fs.ignoreDirEvents" value="True"/>
dirImportWait
The time in seconds that DropBox should wait after being notified of a file before starting an import on that file. This allows for companion files or filesets to be copied. If a new file is added to a fileset during this wait period DropBox begins waiting again. The default is 60 seconds.
<property name="omero.fs.dirImportWait" value="60"/>
fileBatch
The number of files that can be copied in before processing the batch. In cases where there are large numbers of files in a typical file set it may be more efficient to set this value higher. The default is 10.
<property name="omero.fs.fileBatch" value="10"/>
throttleImport
The time in seconds that DropBox should wait after initiating an import before initiating a second import. If imports are started too close together connection issues can arise. The default is 10 seconds.
<property name="omero.fs.throttleImport" value="10"/>
readers
A file of readers. If this is a valid file then it is used to filter those events that are of interest. Only files corresponding to a reader in the file will be imported. The default is empty.
<property name="omero.fs.readers" value=""/>
importArgs
A string of extra arguments supplied to the importer. This could include, for example, an email address to report failed imports to: --report --email test@example.com. The default is empty. For details on available extra arguments see Import images.
<property name="omero.fs.importArgs" value=""/>
Here’s a full example of a configuration for two users:
<property name="omero.fs.importUsers" value="amy;zak"/>
<property name="omero.fs.watchDir" value="/home/amy/myData;/home/zak/work/data"/>
<property name="omero.fs.eventTypes" value="Creation,Modification;Creation,Modification"/>
<property name="omero.fs.pathMode" value="Follow;Follow"/>
<property name="omero.fs.whitelist" value=";"/>
<property name="omero.fs.blacklist" value=";"/>
<property name="omero.fs.timeout" value="0.0;0.0"/>
<property name="omero.fs.blockSize" value="0;0"/>
<property name="omero.fs.ignoreSysFiles" value="True;True"/>
<property name="omero.fs.ignoreDirEvents" value="True;True"/>
<property name="omero.fs.dirImportWait" value="60;60"/>
<property name="omero.fs.fileBatch" value="10;10"/>
<property name="omero.fs.throttleImport" value="10;10"/>
<property name="omero.fs.readers" value="/home/amy/my_readers.txt;"/>
<property name="omero.fs.importArgs" value="--report;--report --email zak@example.com"/>