In-place import

In-place import is a feature added to OMERO 5.0.0 to allow files which are already present on the server machine to be imported into OMERO without the need to copy them. This requires users to have shell (SSH, etc.) access to the server machine, and so there are a number of limitations to this implementation. Development of this feature is on-going, with improvements planned to enable a more user-friendly experience. This CLI-based stop-gap is being made available at this stage because for some users, in-place import is essential for their use of OMERO.

This feature is designed to allow imaging facilities to import large datasets into OMERO while keeping them safely stored in a secure repository which is read-only for users. Leaving the data in a user’s file system is very dangerous as they may forget they need to keep it or move to a different institution. Under no circumstances should in-place import be used with temporary storage.

Warning

The instructions below should help you get started but it is critical that you understand the implications of using this feature. Please do not just copy commands and hope for the best.

Responsibilities

As a data management platform, OMERO assumes that it is in control of your data in order to help prevent data loss. It assumes that data was copied into the server and only a server administrator or authorized OMERO user would have the rights to do anything destructive to that data.

With in-place import, the data either resides completely outside of OMERO or is shared with other users. This means that the critical, possibly sole, copy of your data must be protected outside of OMERO. This is your responsibility for the lifetime of the data.

Limitations

In-place import is only available on the OMERO server system itself. In other words, using SSH or similar, you will need to shell into the server and run the command-line importer directly. If you are uncomfortable with this, you should let someone else handle in-place importing.

Someone wanting to perform an in-place import MUST have:

• a regular OMERO account
• an OS-account with access to omero bin/omero
• read access to the location of the data
• write access to the ManagedRepository or one of its subdirectories

The above means that it may be useful to create a single OS account (e.g. “import_user”) which multiple users can log into, and then use their own OMERO accounts to import data. Alternatively, each OMERO user can be given an OS account with access rights to the data storage as well as the managed repository.

At the moment in-place import is available only under Posix systems (Unix, Linux, OS X). Support for Windows is forthcoming.

For soft linking with --transfer=ln_s it has been noticed that some plate imports run rather more slowly than usual. Other operations may also be affected. In determining if or how to use in-place import at your high-content screening facility, we thus recommend time profiling with representative data, and alerting us to any significant disappointments.

Also, there is still some data duplication when pyramids are generated. We are hoping to find a work-around for this in the future.

Safety tips

Whether you chose to use the hard- or soft-linking option below, you should take steps to secure files which are in-place imported to OMERO. The best option is making them read-only for both the OMERO user and also for the owner of the data. This means the server cannot accidentally modify the files (e.g. if a client mixes up the file IDs and tries to write to the wrong location) and that the files cannot be removed from the system while OMERO is still using them. Files may not be renamed or otherwise altered such that the OMERO server user cannot find them at the expected location.

If possible, all the files should be added to your regular backup process.

Getting started

The 5.0.0 command-line import client has a new (hidden) help menu which explains the available options:

$ bin/omero import -- --advanced-help

ADVANCED OPTIONS:

  These options are not intended for general use. Make sure you have read the
  documentation regarding them. They may change in future releases.

  In-place imports:
  -----------------

    --transfer=ARG              File transfer method

        General options:
          upload              # Default
          some.class.Name     # Use a class on the CLASSPATH.

        Server-side options:
          ln                  # Use hard-link.
          ln_s                # Use soft-link.
          ln_rm               # Caution! Hard-link followed by source deletion.


  ex. $ bin/omero import -- --transfer=ln_s foo.tiff
      $ ./importer-cli --transfer=ln bar.tiff
      $ CLASSPATH=mycode.jar ./importer-cli --transfer=com.example.MyTransfer baz.tiff

  Background imports:
  -------------------

    --minutes_wait=ARG          Choose how long the importer will wait on server-side processing.
                                ARG > 0 implies the number of minutes to wait.
                                ARG = 0 exits immediately. Use a *_completed option to clean up.
                                ARG < 0 waits indefinitely. This is the default.

    --close_completed           Close completed imports.

    --wait_completed            Wait for all background imports to complete.


  ex. $ bin/omero import -- --minutes_wait=0 file1.tiff file2.tiff file3.tiff
      $ ./importer-cli --minutes_wait=0 some_directory/
      $ ./importer-cli --wait_completed # Waits on all 3 imports.

  Import speed:
  -------------

    --checksum_algorithm=ARG    Choose a possibly faster algorithm for detecting file corruption,
                                e.g. Adler-32 (fast), CRC-32 (fast), MD5-128,
                                     Murmur3-32, Murmur3-128, SHA1-160 (slow, default)

  ex. $ bin/omero import --checksum_algorithm=CRC-32 foo.tiff
      $ ./importer-cli --checksum_algorithm=Murmur3-128 bar.tiff

Report bugs to <ome-users@lists.openmicroscopy.org.uk>

The option for performing an in-place transfer is –transfer. A new extension point, file transfers allow a choice of which mechanism is used to get a file into OMERO.

$ bin/omero import -- --transfer=ln_s my_file.dv

Using session bba923bb-cf0c-4cf0-80c5-a309be523ad8 (root@localhost:4064). Idle timeout: 10.0 min. Current group: system
...[      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.0.0-rc1-DEV-ice35
...[      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.0.0-rc1-DEV-ice35 revision: 101008f date: 31 January 2014
...[      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln_s
...[      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
...[      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
...[      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 98ms. (100ms total) [0 unknowns]
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.0.0
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.0.0-rc1-DEV-ice35
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.7.0_51
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.8.0-27-generic
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
...[      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.0.0-rc1-DEV-ice35
...[      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.0.0-rc1-DEV-ice35 revision: 101008f date: 31 January 2014
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
...[      main] INFO   s.importer.transfers.SymlinkFileTransfer - Transferring /tmp/a.fake...
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /tmp/a.fake
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_COMPLETE: /tmp/a.fake
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_END
...[l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_IMPORTED Step: 1 of 5  Logfile: 24605
...[l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 24605
...[l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - THUMBNAILS_GENERATED Step: 3 of 5  Logfile: 24605
...[l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - METADATA_PROCESSED Step: 4 of 5  Logfile: 24605
...[l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor - OBJECTS_RETURNED Step: 5 of 5  Logfile: 24605
...[l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /tmp/a.fake
Imported pixels:
5001
Other imported objects:
Fileset:4102
Image:5001
...[l.Client-0] INFO      ome.formats.importer.cli.ErrorHandler - Number of errors: 0

The only visible difference here is the line:

...formats.importer.cli.CommandLineImporter - Setting transfer to ln_s

Rather than uploading via the OMERO API, the command-line importer makes a call to the system ln command.

Transfer options

Previously, OMERO only offered the option of uploading via the API. Files were written in blocks via the RawFileStore interface. With in-place import, several options are provided out of the box as well as the ability to use your own.

“ln_s” - soft-linking

The most flexible option is soft-linking. For each file, it executes ln -s source target on the local file system. This works across file system boundaries and leaves a clear record of what file was imported:

/OMERO/ManagedRepository/root_0/2014-01/24/10-11-14.947$ ls -ltra
total 8
lrwxrwxrwx 1 omero omero   11 Jan 24 10:11 my-file.dv -> /home/demo/my-file.dv

Here you can see in the imported file set, a soft-link which belongs to the omero user, but which points to a file in the /home/demo directory.

Deleting the imported images in OMERO will delete the soft link but not the original file under /home. This could come as a surprise to users, since the deletion will effectively free no space.

Warning

The deletion of the original files under /home (or equivalent) will lead to a complete loss of the data since no copy is held in OMERO. Therefore, this method should only be used in conjunction with a properly managed and backed-up data repository. If the files are deleted, there is no way to use OMERO to retrieve them.

“ln” - hard-linking

The safest option is hard-linking, though it cannot be used across file systems. For each file, it executes ln source target. Attempting to hard link across file system boundaries will lead to an error:

...[      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.0.0-rc1-DEV-ice35
...[      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.0.0-rc1-DEV-ice35 revision: 101008f date: 31 January 2014
...[      main] INFO   formats.importer.cli.CommandLineImporter - Setting transfer to ln
...[      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
...[      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
...[      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 1 group(s) with 1 call(s) to setId in 96ms. (98ms total) [0 unknowns]
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to localhost:4064
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.0.0
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.0.0-rc1-DEV-ice35
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 1.7.0_51
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
...[      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 3.8.0-27-generic
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_PREPARATION
...[      main] INFO          ome.formats.importer.ImportConfig - OMERO Version: 5.0.0-rc1-DEV-ice35
...[      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 5.0.0-rc1-DEV-ice35 revision: 101008f date: 31 January 2014
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILESET_UPLOAD_START
...[      main] INFO   .importer.transfers.HardlinkFileTransfer - Transferring /tmp/a.fake...
...[      main] INFO   ormats.importer.cli.LoggingImportMonitor - FILE_UPLOAD_STARTED: /tmp/a.fake
...[      main] ERROR  .importer.transfers.HardlinkFileTransfer - transfer process returned 1
...[      main] ERROR  .importer.transfers.HardlinkFileTransfer - error in closing raw file store
omero.ResourceError: null
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_51]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_51]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_51]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_51]
    at java.lang.Class.newInstance(Class.java:374) ~[na:1.7.0_51]
    at IceInternal.BasicStream.createUserException(BasicStream.java:2615) ~[ice.jar:na]
    at IceInternal.BasicStream.access$300(BasicStream.java:12) ~[ice.jar:na]
    at IceInternal.BasicStream$EncapsDecoder10.throwException(BasicStream.java:3099) ~[ice.jar:na]
    at IceInternal.BasicStream.throwException(BasicStream.java:2077) ~[ice.jar:na]
    at IceInternal.Outgoing.throwUserException(Outgoing.java:538) ~[ice.jar:na]
    at omero.api._RawFileStoreDelM.close(_RawFileStoreDelM.java:466) ~[blitz.jar:na]
    at omero.api.RawFileStorePrxHelper.close(RawFileStorePrxHelper.java:1739) ~[blitz.jar:na]
    at omero.api.RawFileStorePrxHelper.close(RawFileStorePrxHelper.java:1701) ~[blitz.jar:na]
    at ome.formats.importer.transfers.AbstractFileTransfer.cleanupUpload(AbstractFileTransfer.java:123) ~[blitz.jar:na]
    at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:63) [blitz.jar:na]
    at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:410) [blitz.jar:na]
    at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:465) [blitz.jar:na]
    at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:274) [blitz.jar:na]
    at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:218) [blitz.jar:na]
    at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:658) [blitz.jar:na]
2014-01-31 12:59:20,338 3152       [      main] ERROR        ome.formats.importer.ImportLibrary - Error on import
java.lang.RuntimeException: transfer process returned 1
    at ome.formats.importer.transfers.AbstractExecFileTransfer.exec(AbstractExecFileTransfer.java:137) ~[blitz.jar:na]
    at ome.formats.importer.transfers.AbstractExecFileTransfer.transfer(AbstractExecFileTransfer.java:57) ~[blitz.jar:na]
    at ome.formats.importer.ImportLibrary.uploadFile(ImportLibrary.java:410) ~[blitz.jar:na]
    at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:465) ~[blitz.jar:na]
    at ome.formats.importer.ImportLibrary.importCandidates(ImportLibrary.java:274) ~[blitz.jar:na]
    at ome.formats.importer.cli.CommandLineImporter.start(CommandLineImporter.java:218) [blitz.jar:na]
    at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:658) [blitz.jar:na]
2014-01-31 12:59:20,338 3152       [      main] INFO         ome.formats.importer.ImportLibrary - Exiting on error

The safeness of this method comes from the fact that OMERO also has a pointer to the data. Deletion of the original file under /home would leave data in OMERO in place. Again, this could cause a surprise as the space would not be properly freed, but at least there cannot be an accidental loss.

Warning

The primary concern with this method is modification of files. If the original data is written by a user, unexpected results could follow in OMERO. See the Safety tips section above for ways around this.

If you are unclear about how hard-linking works, please see the Hard link article on Wikipedia.

“ln_rm” - moving

Finally, the least favored option is ln_rm. It first performs a hard-link like ln, but once the import is complete it attempts to delete the original file. This is currently in testing as an option for DropBox but is unlikely to be of use to general users.

Your own file transfer

If none of the above options work for you, it is also possible to write your own implementation of the ome.formats.importer.transfers.FileTransfer class, likely subclassing ome.formats.importer.transfers.AbstractFileTransfer or ome.formats.importer.transfers.AbstractExecFileTransfer. If you do so, please let us know how we might improve either the interface or the implementations that we provide.

Once your implementation has been compiled into a jar and placed in the lib/clients directory, you can invoke it using:

$ bin/omero import -- --transfer=example.package.ClassName ...

Related advanced options

In addition to the –transfer option in 5.0.0, a number of other advanced options have been added which may be useful for either tweaking import performance or dealing with complicated situations. However, like –transfer, these options should be considered experimental and may change in subsequent releases. Comments and suggestions are very welcome.

Checksums

If you think that calculating the checksums for your large files is consuming too much time, you might want to configure the checksum algorithm used. This can be done with the –checksum_algorithm property. Available options are printed with the –advanced-help option and include Adler-32, CRC-32, MD5-128, Murmur3-32, Murmur3-128, and the default SHA1-160.

DropBox

As described in the scenario “In-place DropBox import (automatic delete)”, DropBox can be configured to use any of the options described above. The configuration property to modify is omero.fs.importArgs:

$ bin/omero config set -- omero.fs.importArgs "--transfer=ln_rm"

This will move files into OMERO rather than leaving a copy in the DropBox directory. For this to work, the two directories will need to be on the same file system. Please read and understand the implications of the ln_rm option.

Warning

Use at your own risk!