diff --git a/distribution/asf-release-process-guide.md b/distribution/asf-release-process-guide.md index 7799f9a1de5f..7d3731885c3c 100644 --- a/distribution/asf-release-process-guide.md +++ b/distribution/asf-release-process-guide.md @@ -17,13 +17,116 @@ ~ under the License. --> -## Getting Started +## 🚀 Release Checklist + +- [ ] [Prior to starting, initial setup for release access](#prior-to-starting-initial-setup-for-release-access) + - [ ] [SVN access](#SVN-access) + - [ ] [GPG key](#GPG-key) + - [ ] [Key propagation](#Key-propagation) + - [ ] [Maven credentials](#Maven-credentials) +- [ ] [Announce intention to release](#Announce-intention-to-release) +- [ ] [Create a release branch](#Create-a-release-branch) + - [ ] [Preparing the release branch](#Preparing-the-release-branch) + - [ ] [Preparing the master branch for the next version after branching](#Preparing-the-master-branch-for-the-next-version-after-branching) + - [ ] [Backport PRs tagged with release milestone](#Backport-PRs-tagged-with-release-milestone) + - [ ] [Release branch hygiene](#Release-branch-hygiene) +- [ ] [LICENSE and NOTICE handling](#LICENSE-and-NOTICE-handling) +- [ ] [Release notes](#Release-notes) +- [ ] [Building a release candidate](#Building-a-release-candidate) + - [ ] [Update the release branch to have all commits needed](#Update-the-release-branch-to-have-all-commits-needed) + - [ ] [Set version and make a tag](#Set-version-and-make-a-tag) + - [ ] [Do a clean clone (so the source distribution does not pick up extra files)](#Do-a-clean-clone-so-the-source-distribution-does-not-pick-up-extra-files) + - [ ] [Switch to tag](#Switch-to-tag) + - [ ] [Build Artifacts](#Build-Artifacts) + - [ ] [Verify checksums](#Verify-checksums) + - [ ] [Verify GPG signatures](#Verify-GPG-signatures) + - [ ] [Commit artifacts to SVN repo](#Commit-artifacts-to-SVN-repo) + - [ ] [Update druid.staged.apache.org](#Update-druidstagedapacheorg) + - [ ] [Create staged Maven repo](#Create-staged-Maven-repo) + - [ ] [Release candidates and voting](#Release-candidates-and-voting) + - [ ] [Druid PMC vote result](#Druid-PMC-vote-result) +- [Final Release](#Final-Release) + - [ ] [Create git tag](#Create-git-tag) + - [ ] [Publish release artifacts to SVN](#Publish-release-artifacts-to-SVN) + - [ ] [Publish the staged Maven repo](#Publish-the-staged-Maven-repo) + - [ ] [Wait 24 hours then update druid.apache.org](#Wait-24-hours-then-update-druidapacheorg) + - [ ] [Draft a release on github](#draft-a-release-on-github) + - [ ] [Announce the release](#Announce-the-release) + - [ ] [Update Wikipedia](#Update-Wikipedia) + - [ ] [Remove old releases which are not 'active'](#remove-old-releases-which-are-not-active) + + +## Prior to starting, initial setup for release access -### Announce intention to release +### SVN access + +Make sure that you have Apache SVN access set up, as several steps in the release process will require you to make SVN commits. + +### GPG key + +First, make sure you've set up a GPG key according to ASF instructions: https://www.apache.org/dev/openpgp.html + +### Key propagation + +The Apache guide suggests using the MIT keyserver, but if there are availability issues consider using https://sks-keyservers.net/ or http://pgp.surfnet.nl/ + +After your key has been propagated to public key servers, add your key fingerprint as your `OpenPGP Public Key Primary Fingerprint` at https://id.apache.org. + +The key fingerprint can be seen with `gpg --list-keys`, e.g.: + +```plaintext +... +pub rsa4096 2019-02-28 [SC] + 0495E031C35D132479C241EC9283C4921AC4483E +uid [ultimate] Jonathan Wei (CODE SIGNING KEY) +sub rsa4096 2019-02-28 [E] +... +``` + +`0495E031C35D132479C241EC9283C4921AC4483E` is the fingerprint in the example above. + +If all goes well, your Apache account will be associated with that key, which you can check at https://people.apache.org/committer-index.html + +You will also need to add your key to our project KEYS file: https://dist.apache.org/repos/dist/release/druid/KEYS, this repo can be pulled with: + +```bash +$ svn co https://dist.apache.org/repos/dist/release/druid +``` + +Add your entry to the file and commit: + +```bash +$ svn commit -m 'add key for jonwei@apache.org' +``` + +### Maven credentials + +You'll need to configure Maven with your Apache credentials by adding the following to `~/.m2/settings.xml`. Encrypt the password you store here, see https://maven.apache.org/guides/mini/guide-encryption.html for details. + +```xml + + + + + apache.snapshots.https + your-apache-username + {your-encrypted-password} + + + + apache.releases.https + your-apache-username + {your-encrypted-password} + + + +``` + +## Announce intention to release First up in performing an official release of Apache Druid is to announce in the dev mailing list, dev@druid.apache.org, that it is about time for the next (approximately) quarterly release, or, that there is a critical bug that warrants doing a bug fix release, whatever the reason happens to be. Check for any critical bugs that are still open, or issues or PRs tagged with the release milestone, and give the community a bit of heads up to try and wrap up anything that _needs_ to be in the next release. -### Create a release branch +## Create a release branch Next up is creating the release branch off of master (or the previous release branch for a bug fix release). Be sure to do this from the apache git repo, _not your fork_. Name the branch for the release version that is being created, omitting the `druid` prefix that will appear on the release candidate and release tags: @@ -39,7 +142,7 @@ $ git checkout origin/master $ git checkout -b 0.17.0 ``` -#### Preparing the release branch +### Preparing the release branch Ensure that the web console and docker-compose file are in the correct state in the release branch. [package.json](../web-console/package.json) and [package-lock.json](../web-console/package-lock.json) should match the release version. If they do not, run: @@ -85,7 +188,7 @@ The sample [`docker-compose.yml`](https://github.com/apache/druid/blob/master/di Once everything is ready, then push the branch to `origin`. -#### Preparing the master branch for the next version after branching +### Preparing the master branch for the next version after branching If doing a quarterly release, it will also be necessary to prepare master for the release _after_ the release you are working on, by setting the version to the next release snapshot: ```bash @@ -120,10 +223,10 @@ The sample [`docker-compose.yml`](https://github.com/apache/druid/blob/master/di Once this is completed, open a PR to the master branch. Also, be sure to confirm that these versions are all correct in the release branch, otherwise fix them and open a backport PR to the release branch. -#### Updating redirect links in the docs - -For docs, please make sure to add any relevant redirects in `website/redirects.json`. This has to be done before building the new website. - +### Backport PRs tagged with release milestone +If a PR is merged into the master branch after the release branch is created and tagged with the release version, then the commit must be backported to the release branch. Following tools can be helpful: +- [tag-missing-milestones](bin/tag-missing-milestones.py) +- [find-missing-backports](bin/find-missing-backports.py) ### Release branch hygiene @@ -134,85 +237,19 @@ To check the CI status on a release branch, you can go to the commits page e.g. a green ✔ in the commit description. If the commit has a failed build, please click on red ✕ icon in the commit description to go to travis build job and investigate. You can restart a failed build via travis if it is flaky. -The release manager should also keep an eye-out for `Cron Job Unit Tests` failures: https://github.com/apache/druid/actions/workflows/cron-job-unit-tests.yml. If there are failures, we need to fix them before creating an RC. +The release manager should also keep an eye-out for `Cron Job ITS` failures: https://github.com/apache/druid/actions/workflows/cron-job-its.yml. If there are failures, we need to fix them before creating an RC. Once all issues and PRs that are still tagged with the release milestone have been merged, closed, or removed from the milestone and CI on branch is green, the next step is to put together a release candidate. -## Initial setup to create a release candidate - -### SVN access - -Make sure that you have Apache SVN access set up, as several steps in the release process will require you to make SVN commits. - -### GPG key - -First, make sure you've set up a GPG key according to ASF instructions: https://www.apache.org/dev/openpgp.html - -### Key propagation - -The Apache guide suggests using the MIT keyserver, but if there are availability issues consider using https://sks-keyservers.net/ or http://pgp.surfnet.nl/ - -After your key has been propagated to public key servers, add your key fingerprint as your `OpenPGP Public Key Primary Fingerprint` at https://id.apache.org. - -The key fingerprint can be seen with `gpg --list-keys`, e.g.: - -```plaintext -... -pub rsa4096 2019-02-28 [SC] - 0495E031C35D132479C241EC9283C4921AC4483E -uid [ultimate] Jonathan Wei (CODE SIGNING KEY) -sub rsa4096 2019-02-28 [E] -... -``` - -`0495E031C35D132479C241EC9283C4921AC4483E` is the fingerprint in the example above. - -If all goes well, your Apache account will be associated with that key, which you can check at https://people.apache.org/committer-index.html - -You will also need to add your key to our project KEYS file: https://dist.apache.org/repos/dist/release/druid/KEYS, this repo can be pulled with: - -```bash -$ svn co https://dist.apache.org/repos/dist/release/druid -``` - -Add your entry to the file and commit: - -```bash -$ svn commit -m 'add key for jonwei@apache.org' -``` - -### Maven credentials - -You'll need to configure Maven with your Apache credentials by adding the following to `~/.m2/settings.xml`. Encrypt the password you store here, see https://maven.apache.org/guides/mini/guide-encryption.html for details. - -```xml - - - - - apache.snapshots.https - your-apache-username - {your-encrypted-password} - - - - apache.releases.https - your-apache-username - {your-encrypted-password} - - - -``` - ## LICENSE and NOTICE handling -Before cutting a release candidate, the release manager should ensure that the contents of our `LICENSE` and `NOTICE` files are up-to-date. You should specifically check that copyright YEAR is updated in the `NOTICE` file. +The license check has been handle by CI. This section is for informational purposes only in most cases. However, if this is the first release of the year, you should ensure that copyright `YEAR` in the `NOTICE` file has been updated. The following links are helpful for understanding Apache's third-party licensing policies: -http://www.apache.org/dev/licensing-howto.html -http://www.apache.org/legal/src-headers.html -http://www.apache.org/legal/resolved.html +- http://www.apache.org/dev/licensing-howto.html +- http://www.apache.org/legal/src-headers.html +- http://www.apache.org/legal/resolved.html There are in effect 2 versions of the `LICENSE` and `NOTICE` file needed to perform a release, one set for the official ASF source release which are the actual `LICENSE` and `NOTICE` file in the root directory, and second for the convenience binary release which includes all of Druid's dependencies which we synthesize using some tools we have developed over the initial set of releases. @@ -282,10 +319,10 @@ The [make-linkable-release-notes](bin/make-linkable-release-notes.py) script can The release branch should have all commits merged before you create a tag. A commit must be in the release branch if 1) it is merged into the master branch before the release branch is created. In this case, the PR corresponding -to the commit might not have the milestone tagged. The `tag-missing-milestones` script can be useful to find such PRs and +to the commit might not have the milestone tagged. The [tag-missing-milestones](bin/tag-missing-milestones.py) script can be useful to find such PRs and tag them properly. See the above [Release notes](#release-notes) section for more details about the script. 2) it is merged into the master branch after the release branch is created and tagged with the release version. -In this case, the commit must be backported to the release branch. The `find-missing-backports` script can be used to +In this case, the commit must be backported to the release branch. The [find-missing-backports](bin/find-missing-backports.py) script can be used to find such commits that have not been backported. Note that this script relies on the milestone tagged in the PR, so PRs must be tagged properly to make this script working. See the above [Release notes](#release-notes) section for more details about the script. @@ -380,7 +417,7 @@ $ svn commit -m 'add 0.17.0-rc3 artifacts' > Before you start, you need the following: Python 3.11 (or later) and Node 16.14 (or later). -This repo (`druid`) is the source of truth for the Markdown files. The Markdown files get copied to `druid-website-src` and built there as part of the release process. It's all handled by a script in that repo called `do_all_things`. +This repo (`druid-release`) is the source of truth for the Markdown files. The Markdown files get copied to `druid-website-src` and built there as part of the release process. It's all handled by a script in that repo called `do_all_things`. For more thorough instructions and a description of what the `do_all_things` script does, see the [`druid-website-src` README](https://github.com/apache/druid-website-src) @@ -402,15 +439,17 @@ For more thorough instructions and a description of what the `do_all_things` scr 4. Make a PR to the src repo (https://github.com/apache/druid-website-src) for the release branch. In the changed files, you should see the following: - - In `published_versions` directory: HTML files for `docs/VERSION` , `docs/latest`, and assorted HTML and non-HTML files + - In `build` directory: HTML files for `docs/VERSION` , `docs/latest`, and assorted HTML and non-HTML files - In the `docs` directory at the root of the repo, the new Markdown files. All these files should be part of your PR to `druid-website-src`. - Verify the site looks fine and that the versions on the homepage and Downloads page look correct. You can run `http-server` or something similar in `published_versions`. + Verify the site looks fine and that the versions on the homepage and Downloads page look correct. You can run `http-server` or something similar in `build`. - The PR to `druid-website-src` should not be merged. Leave it in draft for reference. It can be closed when you're ready to push to production. -5. Make a PR to the website repo (https://github.com/apache/druid-website) for the `asf-staging` branch using the contents of `published_versions` in `druid-website-src`. Once the website PR is pushed to `asf-staging`, https://druid.staged.apache.org/ will be updated near immediately with the new docs. +5. Make a PR to the website repo (https://github.com/apache/druid-website) for the `asf-staging` branch using the contents of `build` in `druid-website-src`. + - From `druid-website`, run `rsync -av ../druid-website-src/build/* .`, this overwrites assets/docs and doesn't delete docs from previous versions. + - Once the website PR is pushed to `asf-staging`, https://druid.staged.apache.org/ will be updated near immediately with the new docs. ### Create staged Maven repo @@ -626,15 +665,12 @@ Returning to the staged repo you created for the Druid PMC vote ( https://reposi https://central.sonatype.org/pages/releasing-the-deployment.html#close-and-drop-or-release-your-staging-repository -### Wait 24 hours +### Wait 24 hours then update druid.apache.org Apache policy requires projects to wait at least 24 hours after uploading artifacts before announcing a release, to allow time for the release artifacts to propagate across mirrors. http://www.apache.org/legal/release-policy.html#release-announcements - -### Update druid.apache.org - > Before you start, you need the following: Python 3.11 (or later) and Node 16.14 (or later). This repo (`druid`) is the source of truth for the Markdown files. The Markdown files get copied to `druid-website-src` and built there as part of the release process. It's all handled by a script in that repo called `do_all_things`. @@ -662,15 +698,17 @@ For more thorough instructions and a description of what the `do_all_things` scr 5. Add the files to a PR to the src repo (https://github.com/apache/druid-website-src) for the release branch you just created. In the changed files, you should see the following: - - In `published_versions` directory: HTML files for `docs/VERSION` , `docs/latest`, and assorted HTML and non-HTML files. + - In `build` directory: HTML files for `docs/VERSION` , `docs/latest`, and assorted HTML and non-HTML files. - In the `docs` directory at the root of the repo, the new Markdown files. All these files should be part of your PR to `druid-website-src`. - Verify the site looks fine and that the versions on the homepage and Downloads page look correct. You can run `http-server` or something similar in `published_versions`. + Verify the site looks fine and that the versions on the homepage and Downloads page look correct. You can run `http-server` or something similar in `build`. -6. Make a PR to the website repo (https://github.com/apache/druid-website) for the `asf-site` branch using the contents of `published_versions` in `druid-website-src`. Once the website PR is pushed to `asf-site`, https://druid.apache.org/ will be updated near immediately with the new docs. +6. Make a PR to the website repo (https://github.com/apache/druid-website) for the `asf-site` branch using the contents of `build` in `druid-website-src`. + - From `druid-website`, run `rsync -av ../druid-website-src/build/* .`, this overwrites assets/docs and doesn't delete docs from previous versions. + - Once the website PR is pushed to `asf-site`, https://druid.apache.org/ will be updated near immediately with the new docs. -7. When the site is published, the release PR to `druid-website-src` can also be merged. `asf-site` in `druid-website` and `published_versions` on the `master` branch in `druid-website-src` should align. +7. When the site is published, the release PR to `druid-website-src` can also be merged. `asf-site` in `druid-website` and `build` on the `master` branch in `druid-website-src` should align. ### Draft a release on github diff --git a/distribution/bin/find-missing-backports.py b/distribution/bin/find-missing-backports.py index e412527d00f9..9f3400c0118e 100755 --- a/distribution/bin/find-missing-backports.py +++ b/distribution/bin/find-missing-backports.py @@ -57,7 +57,7 @@ def find_next_url(links): if link.find("rel=\"next\"") >= 0: match_result = re.match("", link) if match_result is None: - raise Exception("Next link[{}] is found but can't find url".format(link)) + raise Exception("❌ Next link[{}] is found but can't find url".format(link)) else: url_holder = match_result.group(0) return url_holder[1:-1] @@ -84,7 +84,7 @@ def find_next_url(links): previous_branch_first_commit = subprocess.check_output(command, shell=True).decode('UTF-8') match_result = re.match("(\w+) .*", previous_branch_first_commit) if match_result is None: - raise Exception("Can't find the first commit of the previous release.") + raise Exception("❌ Can't find the first commit of the previous release.") previous_branch_first_commit = match_result.group(1) print("Previous branch: {}, first commit: {}".format(previous_branch, previous_branch_first_commit)) @@ -110,14 +110,14 @@ def find_next_url(links): pr_url = "https://api.github.com/search/issues?per_page=50&page={}&q=milestone:{}+type:pr+is:merged+is:closed+repo:apache/druid".format(page,milestone_title) pr_resp = requests.get(pr_url, auth=(github_username, os.environ["GIT_TOKEN"])).json() if pr_resp['incomplete_results']: - sys.stderr.write('This script cannot handle incomplete results') + sys.stderr.write('❌ This script cannot handle incomplete results') sys.exit(1) pr_items.extend(pr_resp['items']) if len(pr_resp['items']) < 50: print("Total PRs for current milestone: {}".format(len(pr_items))) print("Total expected count: {}".format(pr_resp['total_count'])) if pr_resp['total_count'] != len(pr_items): - sys.stderr.write('Expected PR count does not match with number of PRs fetched') + sys.stderr.write('❌ Expected PR count does not match with number of PRs fetched') sys.exit(1) break find_missing_backports(pr_items, release_pr_subjects, release_pr_numbers) diff --git a/distribution/bin/get-milestone-prs.py b/distribution/bin/get-milestone-prs.py index 28e05968082b..e86a2b024bdd 100755 --- a/distribution/bin/get-milestone-prs.py +++ b/distribution/bin/get-milestone-prs.py @@ -71,7 +71,7 @@ milestone = pr["milestone"] if milestone is not None: milestone_found = True - print("COMMIT: {}, PR#: {}, MILESTONE: {}".format(commit_id, pr["number"], pr["milestone"]["url"])) + print("COMMIT: {}, PR#: {}, MILESTONE: {}".format(commit_id, pr["html_url"], pr["milestone"]["url"])) if not milestone_found: print("NO MILESTONE FOUND FOR COMMIT: {}, CLOSED PRs: {}".format(commit_id, closed_pr_nums)) diff --git a/distribution/bin/tag-missing-milestones.py b/distribution/bin/tag-missing-milestones.py index cf41afd63b65..2a797d3e9cbd 100755 --- a/distribution/bin/tag-missing-milestones.py +++ b/distribution/bin/tag-missing-milestones.py @@ -53,12 +53,19 @@ continue if pr['milestone'] is None: print("Tagging Pull Request {} with milestone {}".format(pr_number, milestone)) - url = "https://api.github.com/repos/apache/druid/issues/{}".format(pr_number) - requests.patch(url, json=milestone_json, auth=(github_username, os.environ["GIT_TOKEN"])) + if os.environ.get("DRY_RUN", "true").lower() == "false": + url = "https://api.github.com/repos/apache/druid/issues/{}".format(pr_number) + response = requests.patch(url, json=milestone_json, auth=(github_username, os.environ["GIT_TOKEN"])) + if response.status_code == 200: + print("✅ Tagged Pull Request {}".format(pr['html_url'])) + else: + print("❌ Response: {}".format(response.json())) + else: + print("💤 Dry run mode, not tagging Pull Request {}".format(pr['html_url'])) else: print("Skipping Pull Request {} since it's already tagged with milestone {}".format(pr_number, pr['milestone']['number'])) except Exception as e: - print("Got exception for commit: {} ex: {}".format(sha, e)) + print("❌ Got exception for commit: {} ex: {}".format(sha, e)) continue