OME Team Developer Practices¶
Overview
A guide to how OME developers work together for current and new project members
Team collaboration¶
For anyone completely new to the project, it is most important to know how to get plugged in. There is a fairly extensive amount of communication flying around related to the project, and being able to find and track it may take some time.
Jabber: Instant messaging¶
On a day-to-day level, the team meets in a Jabber chatroom. (See the list of addresses for the specifics) You are welcome to join via your own Jabber account, but one will likely be provided to you, in which case that account should be preferred. To get started using Adium:
- File ‣ Join Group Chat (Select your GTalk or Jabber Account in the drop-down menu)
- Use either your GTalk or LifeSci LDAP Account Credentials
- Enter the “Chat Room Name” and “Server”
Slightly less frequently, the team meets in teamspeak and/or on skype for having voice discussions. These subgroup, or “minigroup”, meetings are organized as needed, but should provide feedback (tickets, notes, etc) as outlined below. Clients for teamspeak can be downloaded from http://www.teamspeak.com/ and the teamspeak server used by OME is teamspeak.openmicroscopy.org.uk.
More information is available under conference-calls.
As for Skype, being online is not required but it is often beneficial to have it on during the day. Regardless, everyone should have a skype account. Register and download software from http://www.skype.com.
Note
your Skype account name along with all of your contact details should be added to DevContactList on Google Docs. See below.
The three major development tools that the OME project runs are Trac, the developer documentation and Jenkins.
Trac¶
The Trac server is available under https://trac.openmicroscopy.org.uk/ome and uses your LDAP account for authentication. Trac is used to record all tickets (i.e. “bugs” or “issues”) as well as hierarchical groups of “tasks” in “requirements” and “stories”. This functionality is provided by a plugin for Trac named “Agilo” (http://agile42.com/en/) which you may want to read more about. Most importantly it provides a whiteboard where tickets are arranged per story into 3 columns: new, accepted, and closed.
Developer documentation¶
The developer documentation is maintained under version control, generated using Sphinx and hosted on the OME website.
Each section of the code base (OMERO, Bio-Formats) has a landing page that will direct you to all the Developer documentation that you might need. For example, the Developer page for OMERO is here.
Jenkins: Continuous integration¶
Our Jenkins server is available here and also uses LDAP authentication. The name “Hudson” comes from before the Jenkins fork after Sun’s acquisition by Oracle and that’s what the team refers to it as. Jenkins provides a mechanism to run arbitrary tasks (“jobs”) on one or more platforms after particular events (time of day, git push, etc.) These jobs build all of the binaries released by the team, and also run automated testing.
Git and Github: Source code¶
Commits take place primarily on github currently. To be aware of what is really going on, your best option is to become familiar with git, github, and the repositories of all the team members. Information on doing that is available under Checking out the source code and Using Git.
Forums and mailing list¶
Feedback from the OME community happens primarily on 2 public mailing lists that are further described under ome-users and ome-devel as well as on the PhpBB forums, an alternative to the mailing lists since some users prefer the forum interface to the mailing list one, and vice versa.
You should add yourself to all three and be aware of and scan all threads on a fairly regular basis. The general rule is that requests from the community will be responded to by the next working day, where to the best of our ability we keep the ‘working days’ and time zones of the community in mind. If you miss any messages or want to review previous discussions see the archive lists available on the “mailing-lists” page:
Where possible the task of monitoring feedback is spread across the team. Mailing list and forum questions are generally listed at the morning meeting and can be checked off in the accompanying notes when dealt with to ensure nothing is ignored or forgotten.
Anyone on the team should feel free to speak up to answer questions, but do try to verify the correctness of answers, code samples, etc. before posting.
As much information about our activities and decision processes should be made public as possible. For many items, there is no reason to hide our process, but we do not go out of our way to make them public. For example, internally the team often uses OmniGraffle documents to illustrate concepts, but these are kept privately to prevent any confusion.
Internal mailing lists¶
In addition to the two public mailing lists mentioned above, there are also:
- ome-nitpick@lists.openmicroscopy.org, used for team-wide, developer communication that isn’t appropriate for the wider OME community such as organizing mini-group meetings, scheduling vacation, etc.; and
- a number of mail-aliases reserved for automated messages from various pieces of development machinery so do not send mail directly to these addresses, instead use ome-nitpick.
Internal servers¶
There are a number of servers and services inside of the University of Dundee system that are used by the entire team. You may not need access to all of them immediately, but it’s good to know what’s available in case you do.
- vpn.lifesci.dundee.ac.uk (LDAP-based) is necessary for securely accessing some of the following resources (e.g. squig, jenkins)
- squig.openmicroscopy.org is the shared, team-wide repository for data which can be mounted if you are on VPN or within the UoD system. It contains test data for various file formats.
- The official OME website is run using Plone (https://www.openmicroscopy.org/site) (LDAP-based)
- The university provides an Alfresco instance (https://alfresco.lifesci.dundee.ac.uk) (LDAP-based) for collaborating on documents.
- The OME QA system is an in-house system for collecting feedback from users, including failing files, stack traces, etc. Like our community feedback, QA feedback should be turned into a ticket in a timely manner.
- Home directory / data repository on necromancer (SSH-based)
Note
For anyone who has been hired to work at the University of Dundee, you will be provided with another list which itemizes all the things that need to be done to get you setup in RL (building access, a chair, etc.)* See: new start tasklist.
Google Docs¶
In addition to the services hosted in Dundee, the team also makes use of several Google resources due to the improved real-time collaboration that they provide. A single Google collection “OME Docs” is made available to all team members. Anything placed in the collection is automatically editable by everyone.
For example, the primary contact information for all team members is available in the DevContactList spreadsheet.
You can enable notifications on the spreadsheet so that you receive an email if any changes are made.
Meetings¶
Weekly meetings are held online with all members of the team. Agendas are posted on the appropriate page under conference-calls before hand. Notes are taken collaboratively in a Google doc in the “OME Docs > Notes” collection. Once finished, they are added to the page on Plone, and anyone who missed the meeting is expected to review the notes and raise any issues during the next meeting. You should also send an email to ome-nitpick if you will not be attending the meeting since it may change what others can discuss for that week.
Periodically, a technical presentation is held during the weekly meeting. This can be used to either introduce an external tool for suggested use by the team or as a peer review of in-progress work. See meetings for more information.
Minigroup meetings can either be regularly scheduled (e.g. weekly) or on an as-needed basis. Notes from such meetings, however, should be posted centrally to minigroup for review by the team. Either an email should be sent to ome-nitpick with a link to the minigroup notes, or it should be brought up during the weekly meeting.
Process summary¶
Ticket types (day-to-day)¶
Any activities on a day-to-day level will be most accurately and actively trackable in the tickets on Trac. Essentially, if no tickets are marked “active” on the whiteboard for a developer, then it’s assumed that s/he is off or working on a non-OME related activity.
Tickets in OME are divided into 3 types:
- Requirements
- Stories
- Tasks
Requirements¶
Requirements are large, overarching features which will take months (if not longer) to deliver. For a particular release , or “milestone” in Trac terminology, a small number of requirements will be chosen. For patch releases, it’s possible that no requirements will be in-progress, but that only bug fixing will take place.
Stories¶
Requirements are made up of stories, which should take days to weeks to complete. A large number of stories will be put into any one sprint , the two-week period that is visible on the Trac whiteboard at a given time. This is the standard unit of work for the team. After a sprint, the stories that were chosen for the sprint, should be closed if possible, and there should be some evidence of the work (screenshots, screencasts, etc) available from the roadmap page.
Tasks¶
Tasks make up stories and are the most basic building block. They should be on the order of 0.25 to 1 day of work, 2 at the most but no lower than 0.1 days. In fact, they are the only type of ticket that contains a field for recording estimated time, and these should be considered required. Sums of such times are then available in the stories and requirements.
The unit of time used by the OME team is “ideal days”. (Note: in some locations in Trac/Agilo an “h” for hour is shown. Regardless, the time unit is ideal days). An ideal day can be thought of as a day on which a single developer can work without interruption for 6 hours, whether this be coding, testing, designing or documenting. Obviously this almost never happens, but it’s simpler to estimate times if a one ignores meetings, mails, and other annoyances.
Beyond the types used in Trac/Agilo, there are several other ways of identifying or specially marking tickets.
Bugs¶
The Agilo plugin provides another potential level to the hierarchy, “Bug,” which could appear in the whiteboard like a story. We have chosen not to use this feature, since it unduly complicates the workflow (e.g. they are treated as container and one must create a bug inside the bug to do anything with it).
Instead, “Bug:” is pre-pended to the ticket summary to indicate a bug. A list of all current such bugs can be viewed via the BUGS! EEK!! report in the left hand panel:
Where possible, we try not to push bugs out of the current milestone, and instead, we aim to handle them as quickly as possible. If a bug is too large to handle during the current milestone, it should be turned into a story and appropriately scheduled.
RFE¶
“RFE” stands for “Request for Enhancement” and is a fledgling idea for some new feature. They frequently occur during internal testing. While testing a client, for example, a tester will often have the feeling that it’d be nice to be able to do “X”. A kick ticket with “RFE: add support for X” lets the client developer(s) know that such a feature would be useful. The ticket does not contain the necessary technical details, however, to be a story, though it can be turned into one.
Sprint process (week-to-week)¶
Sprints begin at one weekly meeting and terminate two weeks later. They consist of tasks that have been marked for the current sprint, all of which then show up on the whiteboard, most of which are grouped together into stories. A story can have some tickets in the current sprint, while others – though in the same milestone – will be handled in a later sprint. For this reason, a sprint should never be set for a story.
In general, the whiteboard at any given time should clearly reflect the team effort. If a requirement is in another milestone but you are working on it ahead of time, then move the current tasks into the current milestone and sprint so they appear on the whiteboard. (This is a limitation of Trac/Agilo that we are learning to deal with). At certain times, we may have multiple sprints active in which case it’s necessary to be aware of which sprint you are looking at:
Definition of done¶
For stories to be considered “done”, they should include tests, screenshots/casts, and the definition of any “Testing Scenarios” that may be necessary. If it’s easier for you to remember this, then feel free to add individual tasks inside of the story for the tests, screenshots, etc. Others may prefer to write less granular stories and tickets. The key is that someone who is to review the stories and tasks can clearly decide what has changed and what needs to be reviewed and tested. This often means that each story ticket should contain a long-text description and a “usage” statement (“getting started”) along with the related task tickets, and that before it is scheduled into a sprint!
Choosing tasks¶
Once tasks are placed in a sprint choosing between them is more or less arbitrary. Where possible you should prefer to work on:
- Bugs, since they should be considered top priority
- Risky/unclear changes, since they may have extended impact,
- Tasks that are blocking other developers for obvious reasons, and
- Near the end of the sprint if you have completed your tasks, you should help others complete tasks that they may not be able to complete.
Branch process (month-to-month)¶
The output of your daily and weekly work will almost always be the production of a branch. That process is described in (perhaps too much) detail under Using Git
Particularly of importance for this document is the section on “Branch management”. The general idea is that branches also have a lifecycle on the project. They begin as Investigations, progress to Works-in-Progress (WIP), and eventually become Deliverables. Once they are merged into the mainline, they should be removed from github to keep the list of active branches clearly discernible. The Pull Requests that are opened by developers are on-going review conversations, which you are encouraged to get involved in.
Release process¶
A release is primarily defined on the Trac roadmap page.
All tickets (requirements, stories, and tasks) are reachable via the various green reporting bars. The description highlights what the OME team thinks the most important features and goals for the release are. Each of these large ticket items should also contain a link to the requirements or story ticket.
Major releases consist of some number (10+) of the 2-week sprints described above, and should always end with a test cycle of at least 3 weeks. Smaller bug fix releases may be much shorter and won’t need as extensive testing.
Scenarios¶
Testing is largely performed through a number of “scenarios”. Such scenarios should be defined as you develop new features, and the team will periodically review and test the scenarios even before release.
Checklist¶
A checklist of all the things that should be done as the release date approaches is available under Release process
Yearly meetings¶
Typically just before, during, or after a major release, the entire OME team will try to meet to determine the goals and features for the next major release. Being co-located for the release is often also useful in itself, but having time to work through the many different directions is critical. This often happens at the annual users’ meeting in Paris (May/June).
Previews¶
There has been some experimentation on the team with “previews”, which are created and then provided to certain sites for early testing. Previews may be created in the middle of a release or with a final release for testing less stable work.
Specific external groups interested in such previews should be integrated into the process where possible. As soon as previews are ready they should be sent out to interested external parties for testing/review. However, consideration should be taken when choosing such groups since there is a certain amount of indebtedness, i.e. asking a group to test a preview too often could become a burden.
Subgroup processes¶
To see how a specific group works together you might take a look at the web process which is defined under #4772. While numerous new “sub-groups” are getting up-to-speed, we will obviously need to find ways to keep communication and collaboration simple for everyone.
See also
- http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
- Best practices for git commit message formatting
- http://en.wikipedia.org/wiki/Technical_debt
- Wikipedia article on Technical debt
Code contributions¶
In order to expedite the contribution of code to the OME project, whether individual files or entire modules such as a service or web application, we’ve put together the following guidelines. If you have issues with any of the below, please let us know.
File headers¶
The official header templates for each file type (Java, Python, HTML, etc.) can be found in the docs/headers.txt file of the source repository. The correct template should be applied at the top of all newly created files. The header of existing files should not be modified without previous discussion except with regard to keeping the year line up to date, for example changing “2008-2011” to “2008-2013”.
Copyrights¶
The copyright line for a newly created file is based on the institution of the creator of the file and will remain unchanged even if copied or moved. Before redistribution of code can take place, an agreement must be reached between the OME team and the copyright holder.
Licenses¶
The licenses of any files intended for redistribution with OME must be compatible with the GPL and more restrictively for the web components with the AGPL. Some files in the code-base (the schema, etc.) are released under more liberal licenses but are still compatible with the GPL.
Distribution¶
For a block of work to be considered for redistribution with OME, the code must further be made available in one of the following formats.
Patches/Pull requests¶
Smaller changes to the existing code base can be submitted to the team either as patches, or preferably as pull requests on github. You can read the more about pull requests under Branch process (month-to-month). The idea is that such smaller changes are reviewed line-by-line and then maintained by the core team.
Submodules¶
Larger submissions, which cannot be effectively reviewed so intensively, should be submitted as git submodules. Such submodules provide a unique way to describe to a component version, which becomes linked into the main codebase. During checkout, all submodules are downloaded into the OME directory; and during the build process, submodules are compiled into the official distribution.
The OME team cannot maintain or ship code which is only available as a long-living branch (a fork) of the code base, and we’d encourage submitters to use one of the above methods.
Examples of contribution templates¶
There are any number of other projects which have setup similar practices for code contributions. If you would like to read more on the rationale, please see: