Continuous integration (scc) scripts ==================================== https://github.com/openmicroscopy/snoopycrimecop hosts a series of utility scripts used in the continuous integration. If you find a bug or if you want an additional feature to be implemented, please `open an issue `_. Installation ------------ The scc tools are a set of Python_ based utility programs. The tools suite can be installed using ``pip``:: $ pip install -U scc If your Python is old then this may also install the **PyGithub** and **argparse** packages, but since Python 2.7 they are already included with the standard library. Github connection ----------------- Most of the scc commands instantiate a Github connection using the PyGithub package. Unless the :option:`--token` option is passed to the scc command, the command first looks for the `github.token` specified in the git config file and, if found, uses this token to connect to Github:: $ ./scc.py merge master --info -v 2013-01-16 22:03:49,633 [ scc.config] DEBUG Found github.token ... If no token is found, the command looks for a `github.user` in the git config file and, if found, uses this username to connect to Github:: $ ./scc.py merge master --info -v 2013-01-16 22:06:00,256 [ scc.config] DEBUG Found github.user Enter password for http://github.com/sbesson: .. note:: The password to be entered here is the Github password. Finally, if no token or user is found, both the Github username and password are queried at the prompt:: $ ./scc.py merge master --info -v # github.token and github.user not found. # See `./scc.py token` for simpifying use. Username or token: sbesson Enter password for http://github.com/sbesson: .. _scc merge: scc merge --------- .. program:: scc merge Merge all the open PRs based on specified branch, including all submodules. The first argument is the name of the base branch of origin, e.g.:: $ scc merge develop .. option:: --comment Add a comment to the PR if there is a conflict while merging the PR :: $ scc merge develop --comment .. option:: --default , -D Specify the default set of PRs to be included in the merge Four options are currently implemented: `none`, `mine`, `org` and `all`. By default (`org`), only PRs opened by public members of the repository organization are included so the following are equivalent:: $ scc merge develop $ scc merge develop --default org $ scc merge develop -D org .. option:: --exclude , -E Exclude PR by label, user or number Label filters can be specified using `label:my_label`:: $ scc merge develop -E label:exclude_label User filters can be specified using `user:username`:: $ scc merge develop -E user:my_user PR filters can be specified using `pr:number` or `#number`:: $ scc merge develop -E pr:45 -E #46 Submodule PR filters can be specified using `submodule_name#number`:: $ scc merge develop -E bioformats#40 If no key is specified, it is assumed the filter is a label filter. By default, only PRs labeled as `exclude` are excluded so the following are equivalent:: $ scc merge develop $ scc merge develop -E label:exclude $ scc merge develop -E exclude .. option:: --include , -I Include PR by label, user or number Label filters can be specified using `label:my_label`:: $ scc merge develop -I label:exclude_label User filters can be specified using `user:username`:: $ scc merge develop -I user:my_user PR filters can be specified using `pr:number` or `#number`:: $ scc merge develop -I pr:45 -I #46 Submodule PR filters can be specified using `submodule_name#number`:: $ scc merge develop -I bioformats#40 If no key is specified, it is assumed the filter is a label filter. By default, only PRs labeled as `include` are included so the following are equivalent:: $ scc merge develop $ scc merge develop -I label:include $ scc merge develop -I include .. option:: --info Display the candidate PRs to merge but do not merge them :: $ scc merge develop --info .. option:: --push Push the locally merged branch to Github :: $ scc merge develop --push my-merged-branch .. option:: --reset Recursively reset each repository to the HEAD of the base branch :: $ scc merge develop --reset .. option:: --shallow Merge the PRs for the top-level directory only, excluding submodules:: $ scc merge develop --shallow .. option:: --remote Specify the name of the remote to use as the origin. Default: origin:: $ scc merge develop --remote gh As a concrete example, the first step of the :term:`OMERO-merge-develop` job is calling the following merge command: :: $ scc merge develop --no-ask --reset --comment --push merge/develop/latest .. versionchanged:: 0.3.0 Added default values for :option:`--include` and :option:`--exclude` options. .. versionchanged:: 0.3.8 Added :option:`--shallow` and :option:`--remote` options. scc travis-merge ---------------- .. program:: scc travis-merge Merge PRs in a Travis environment, using the PR comments to generate the merge filters. :: $ scc travis-merge This command internally defines all the filter options exposed in :program:`scc merge`. The target branch is read from the base of the PR, the :option:`--default` option is set to `none` meaning no PR is merged by default and no default :option:`--exclude` option is defined. The :option:`--include` filter is determined by parsing all the PR comments lines starting with ``--depends-on``. To include PR 67 in the merge, add a comment line starting with ``--depends-on #67`` to the PR. To include PR 60 of the bioformats submodules, add a comment line starting with ``--depends-on bioformats#60`` to the PR. scc update-submodules --------------------- .. program:: scc update-submodules Update the pointer of all submodules based on specified branch. The first argument is the name of the base branch of origin, e.g.:: $ scc update-submodules develop .. option:: --push Push the locally merged branch to Github and open a PR against the base branch:: $ scc merge develop --push submodules_branch .. option:: --no-pr Combined with :option:`--push` option, push the locally merged branch to Github but skip PR opening:: $ scc merge develop --push submodules_branch --no-pr .. option:: --remote Specify the name of the remote to use as the origin (default: origin):: $ scc update-submodules develop --remote gh scc rebase ---------- .. program:: scc rebase Rebase a PR (open or closed) onto another branch and open a new PR. The first argument is the number of the PR to rebase and the second argument is the name of the branch onto which the PR should be rebased:: $ scc rebase 142 develop Assuming the head branch used to open the PR 142 was called `branch_142`, this command will rebase the tip of `branch_142` onto origin/develop, create a new local branch called `rebased/develop/branch_142`, push this branch to Github and open a new PR. Assuming the command opens PR 150, to facilitate the integration with :program:`scc unrebased-prs`, a ``--rebased-to #150`` comment is added to PR 142 and a ``--rebased-from #142`` comment is added to the PR 150. Finally, the command will switch back to the original branch prior to rebasing and delete the local `rebased/develop/branch_142`. .. note:: By default, :program:`scc rebase` uses the branches of the `origin` remote to rebase the PR. To specify another remote, use the :option:`--remote` option. .. option:: --no-pr Skip the opening of the PR :: $ scc rebase 142 develop --no-pr .. option:: --no-delete Do not delete the local rebased branch :: $ scc rebase 142 develop --no-delete .. option:: --remote Specify the name of the remote to use for the rebase (default: origin) :: $ scc rebase 142 develop --remote snoopycrimecop .. option:: --continue Re-run the command after manually fixing conflicts If :program:`scc rebase` fails due to conflict during the rebase, you will end up in a detached HEAD state. If you want to continue the rebase operation, you will need to manually fix the conflicts:: # fix files locally $ git add conflicting_files # add conflicting files $ git rebase --continue This conflict solving operation may need to be repeated multiple times until the branch is fully rebased. Once all the conflicts are resolved, call :program:`scc rebase` with the :option:`--continue` option:: $ scc rebase --continue 142 develop Depending on the input options, this command will perform all the steps of the rebase command (Github pushing, PR opening) skipping the rebase part. Alternately, you can abort the rebase and switch to your previous branch:: $ git rebase --abort $ git checkout old_branch .. versionchanged:: 0.3.10 Automatically add ``--rebased-to`` and ``--rebased-from`` comments to the source and target PRs. scc unrebased-prs ----------------- .. program:: scc unrebased-prs Compare two development branches and check that PRs merged in one branch have been merged to the other. The basic workflow of the :program:`scc unrebased-prs` command is the following: - list all first-parent merge commits for each branch including git notes referenced as `see_also/other_branch` where other_branch is the name of the branch to check against. - exclude all merge commits with a note containing either "See gh-" or "n/a" - for each remaining merge commit, parse the PR number and look into the PR body/comments for lines starting with ``--rebased-to``, ``--rebased-from`` or ``--no-rebase``. Additionally, for each line of each PR starting with ``--rebased-to`` or ``--rebased-from``, the existence of a matching line is checked in the corresponding source/target PR. For instance, if PR 70 has a ``--rebased-from #67`` line and a ``--rebased-from #66`` line, then both PRs 66 and 67 should have a ``--rebased-to #70`` line. This command requires two positional arguments corresponding to the name of the branch of origin to compare:: $ scc unrebased-prs dev_4_4 develop .. option:: --remote Specify the name of the remote to use as the origin (default: origin):: $ scc unrebased-prs dev_4_4 develop --remote gh .. option:: --no-check Do not check links between rebased comments:: $ scc unrebased-prs dev_4_4 develop --no-check .. versionadded:: 0.3.10 Added support for body/comment parsing and ``--rebased-to/from`` linkcheck scc version ----------- .. program:: scc version Return the version of the scc tools:: $ scc version 0.3.0 .. versionadded:: 0.2.0 .. _scc deploy: scc deploy ---------- .. program:: scc deploy Deploy a website update using `file symlink replacement `_:: $ scc deploy folder The goal of this command is to enable overwriting of deployed doc content and allow for “hot-swapping” content served by Apache without downtime and HTTP 404s. .. option:: --init Prepare folder for symlink replacement. Should only be run once :: $ scc deploy folder --init .. versionadded:: 0.3.1 The hudson jobs :term:`BIOFORMATS-docs-merge-stable`, :term:`BIOFORMATS-docs-merge-develop`, :term:`OMERO-docs-merge-stable`, :term:`OMERO-docs-merge-develop` and OMERO-docs-internal deploy the documentation artifacts to necromancer. The target directory (sphinx-docs) is controlled by the hudson:hudson user, so all file system operations are allowed. Each job has the target directory configured in the SSH publisher target directory property. After deployment has happened to a temporary directory, a series of symlink moves happens making sure that the symlink points to the updated content.