OMERO Quay source code components

The Manifest class

Assay

Bases: Collection

An assay is a collection of images and is the lower organisation level. Companion files will be added as file annotation in omero

Collection

Bases: NamedThing

A collection is equivalent to a directory in a unix file system. It can have sub-collections

Error

Bases: ConfiguredBaseModel

If the state is errored, provides context of the error In python, message corresponds to exc_value, and details to the traceback

Investigation

Bases: Collection

The top level collection in the hierarchy, equivalent to a Group in iRODS and OMERO

Manifest

Bases: NamedThing

The root data entity

MapAnnotation

Bases: QuayAnnotation

In iRODS, there is an optional Unit, so key value pairs are really triples. For now this is not supported

NamedThing

Bases: Thing

A generic grouping for an entity with foreign identities and a description not for thin objects as states

Role

Bases: str, Enum

Roles for Investigation users or Provenance persons

author = 'author' class-attribute instance-attribute

An author, defined at import time, is either the person who acquired the data or produced secondary data (segmentation results, ROIs). More than one member of an investigation can be author at the dataset level, author declared here will be appended to the assay authors / dataset owners. Shall this granularity not be sufficient, it can be corrected in OMERO by the data steward.

data_steward = 'data_steward' class-attribute instance-attribute

In OMERO, the data steward is Group owner and responsible for group membership for the Investigation. OMERO default group is read write annotate They are responsible for data and metadata management They are the contact person for the investigation's storage resource policies, including quota management. They should have followed a training session of some sort to be familiar with the system

member = 'member' class-attribute instance-attribute

Any other person linked to the project. Read only data access is only quaranteed for regitstered members but any Person can have member Role

sender = 'sender' class-attribute instance-attribute

A collection sender has read_write privileges on the collection Will have 'delete_object' ACls in iRODS omero : group owner

Scheme

Bases: str, Enum

A scheme refers to a data service type (iRODS, omero, filesystem or S3 server)

omero = 'omero' class-attribute instance-attribute

Used to manipulate data or users within OMERO

Status

Bases: str, Enum

Used to indicate up-to-dateness of a service with respect to a manifest.

started: The manifest is new checked: The manifest is conform here and in the other services changed: The manifest is conform here and must be pushed to peers expired: The manifest needs to be processed here errored: Well ...

Study

Bases: Collection

A Study is a (potentially growing) collection of assays

Thing

Bases: ConfiguredBaseModel

A generic grouping for any identifiable entity

User

Bases: NamedThing

Represents a user in the system - this user can log into iRODS and omero and samnba

Core code

Configuration and Utils

get_conf()

Returns a configuration dictionary from the file pointed by the environment variable QUAY_CONF if this is not defined, looks in /etc/quay/quay.yml

recurse_update(old_nested_dict, new_nested_dict)

Update recursively the content of nested dictionary plain files.

expand_ids(id_seq, full_seq)

Replace ids in id_seq with the corresponding object in full_seq

find_by_id(id_, seq)

Finds an object in a sequence such that the object id attribute is equal to id

find_by_name(name, seq)

Finds an object in a sequence such that the object name attribute is equal to name

get_depth(isaobject)

Returns 2 for Investigation, 1 for Study and 0 for Assay

get_identifiers(isaobject)

Check for attributes of the isaobject whose slot_uri in the manifest schema is 'schema:identifier', with the exclusion of the id slot,

returns a dict {atrr: isaobject.attr}

get_path(isaobject, scheme)

Returns the last url with scheme 'scheme' found in isaobject.urls, or None

slugify(value, allow_unicode=True)

Taken from https://github.com/django/django/blob/master/django/utils/text.py Convert to ASCII if 'allow_unicode' is False. Convert spaces or repeated dashes to single dashes. Remove characters that aren't alphanumerics, underscores, or hyphens. Also strip leading and trailing whitespace, dashes, and underscores.

Workflow orchestration

clerk_coroutine(clerk_class, conf, service)

TODO

Base Clerk classes

Clerk

Bases: Interface

Abstract sender

Clerks parse a manifest and perform actions on a spectific system or between systems

__init__(conf, manifest, scheme, host=None)

required conf entries:

crud()

Updates the local hierarchy from the created, updated and deleted isaobjects

make_tree_view()

Creates the HTML tree webpage content. Uses manifest as source.

parse()

resets cached dicts and mappings, parses the manifest to populate created, updated and deleted dictionaries.

populates the mapping between the local object and the manifest's hierarchy.

DataClerk

Bases: Clerk

Abstract sender

Managers parse a manifest and perform actions on a spectific system or between systems

__init__(conf, manifest, scheme, host=None)

required conf entries:

Interface

Base class to interact with a manifest object

set_state(status, progress, task)

Set the state of the interface. Permissible values values for status:

  • started
  • changed
  • checked
  • expired
  • errored
Parameters:
  • status (str) –

    The status of the item.

  • task –

    str: a string describing the current task

UserClerk

Bases: Clerk

register()

do nothing to avoid re-implementing parse

transfer()

do nothing

Protocol / format specific interfaces

Excel

excel_to_manifest_post_request(xlsx, conf=None, post_url=None, sender=None) async

check_if_assay_path_is_subpath_of_other_assay(xlsx_path)

Check path validity of assays:

Conditions
  • Assay:
    • Path must exist as a subdir of the investigation in the buffer machine
    • No assay path cannot be a subpath of another assay (see pathlib.Path)

    • This is invalid:

      path name
      Dir1 assay1
      Dir1/SubDir1 assay2

check_if_user_in_group(user_login, group_name)

Check if OMERO user is in OMERO group. Raise error if not.

validate_user_existence(user_login)

Validate user existence in OMERO database. Raise error if not in database.

validate_xlsx_structure(xlsx_path)

Validates the structure of the XLSX file.

Conditions
  • Global:
    • sheet "Investigation", "Study", "Assay" are present
  • Investigation
    • Must have a name
    • No space in investigation name
    • sender must be a valid user
    • authors: at least one valid user
    • collaborators: zero or more person (no user accounts necessary)
  • Study
    • Must have a name
    • Must have valid parent investigation (name present in the "investigation" sheet)
  • Assay
    • All the paths must be different
    • Must have a valid study name as a parent

Filesystem

gen_manifest(srce_path, hierarchy=None, owner_name=None, scheme='file', route='local.default')

depth 0: assay depth 1: study depth 2: investigation

FSData

Bases: DataClerk

Parses manifest in a file system

example:

.. code-block:: [python]

dry_run = False
...
with FSData(manifest) as fsmngr:
    fsmngr.parse()
    # check every thing is ok
    print(fsmngr)
    fsmngr.transfer()

__init__(conf, manifest, host=None)

required conf entries:

transfer()

Creates the ISA hierarchy for self.new

Moves data_objects from source to destination as mapped in self.destinations

OMERO

OMERO operations

OmeroData

Bases: DataClerk

transfer()

Uses the (augmented) omes created by the parse method to import data in omero

iRODS

OMERO operations

OmeroData

Bases: DataClerk

transfer()

Uses the (augmented) omes created by the parse method to import data in omero