changelog: Add a tool to extract changelog entries from the commits

Since the `CHANGELOG.md` file is a major source for merge conflicts I decided
to build a tiny tool that generates the entries for the changelog
automatically from the commit messages.

devtools/changelog.py

@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+from collections import namedtuple
+import re
+import sys
+import shlex
+import subprocess
+import requests
+from mako.template import Template
+import argparse
+from datetime import datetime
+
+# What sections do we support in the changelog:
+sections = [
+    'added',
+    'changed',
+    'deprecated',
+    'fixed',
+    'removed',
+    'security',
+]
+
+repo = 'ElementsProject/lightning'
+
+Entry = namedtuple("Entry", ["commit", "pullreq", "content", "section"])
+
+
+def git(cmd):
+    cmd = shlex.split(cmd)
+    out = subprocess.check_output(['git'] + cmd)
+    return out.decode('UTF-8')
+
+
+def get_commit_range():
+    """Find a commit range that we should collect the CHANGELOG for.
+    """
+    description = git("describe")
+    version = description.split('-')[0]
+    return "{version}..master".format(version=version)
+
+
+def get_log_entries(commitrange):
+    commit = None
+    logs = git("log {commitrange}".format(commitrange=commitrange))
+    entries = []
+
+    for l in logs.split('\n'):
+        m = re.match(r'^commit ([A-Fa-f0-9]{40})$', l)
+        if m:
+            commit = m.group(1)
+
+        m = re.match(
+            r'^\s+Changelog-({}): (.*)$'.format("|".join(sections)), l)
+        if not m:
+            continue
+
+        # Now try to resolve the pull request that originated this commit:
+        headers = {
+            'Accept': 'application/vnd.github.groot-preview+json',
+        }
+
+        url = 'https://api.github.com/repos/{repo}/commits/{commit}/pulls'.format(repo=repo, commit=commit)
+        content = requests.get(url, headers=headers).json()
+        if len(content):
+            pullreq = content[0]['number']
+        else:
+            pullreq = None
+
+        e = Entry(commit, pullreq, m.group(2), m.group(1))
+        entries.append(e)
+
+    return entries
+
+
+def group(entries):
+    groups = {s: [] for s in sections}
+    for e in entries:
+        groups[e.section].append(e)
+    return groups
+
+
+def commit_date(commitsha):
+    """Get the date of the specified commit.
+    """
+    line = git("show -s --format=%ci")
+    dt = datetime.strptime(line.strip(), '%Y-%m-%d %H:%M:%S %z')
+    return dt
+
+
+template = Template("""<%def name="group(entries)">
+% for e in entries:
+ - ${e.content} ([${e.pullreq}](https://github.com/ElementsProject/lightning/pull/${e.pullreq}))
+% endfor
+</%def>
+<!--
+TODO: Insert version codename, and username of the contributor that named the release.
+-->
+
+${h2} [${version}] - ${date.strftime("%Y-%m-%d")}: "CODENAME"
+
+This release named by @USERNAME.
+
+${h3} Added
+${group(groups['added']) | trim}
+${h3} Changed
+${group(groups['changed']) | trim}
+${h3} Deprecated
+
+Note: You should always set `allow-deprecated-apis=false` to test for changes.
+${group(groups['deprecated']) | trim}
+${h3} Removed
+${group(groups['removed']) | trim}
+${h3} Fixed
+${group(groups['fixed']) | trim}
+${h3} Security
+${group(groups['security']) | trim}""")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description='Generate a changelog summary for a given commit range'
+    )
+    parser.add_argument('commitrange', type=str, nargs='?',
+                        help='Range of commits to consider (format: <from_commit>..<to_commit>',
+                        default=get_commit_range())
+
+    args = parser.parse_args()
+
+    if '..' not in args.commitrange:
+        print("Commit range must include '..' to separate 'from_commit' and 'to_commit'")
+        sys.exit(1)
+
+    fromcommit, tocommit = args.commitrange.split('..')
+    entries = get_log_entries(args.commitrange)
+    groups = group(entries)
+    date = commit_date(tocommit)
+
+    print(template.render(
+        groups=groups,
+        h2='##',
+        h3='###',
+        version=tocommit,
+        date=date,
+    ))

doc/MAKING-RELEASES.md

@@ -19,13 +19,16 @@ Here's a checklist for the release process.
 
 ### Preparing for -rc1
 
-1. Check that CHANGELOG.md is well formatted, ordered in areas,
+1. Check that `CHANGELOG.md` is well formatted, ordered in areas,
    covers all signficant changes, and sub-ordered approximately by user impact
    & coolness.
-2. Update the CHANGELOG.md with [Unreleased] changed to v<VERSION>-rc1. Note that
+2. Use `devtools/changelog.py` to collect the changelog entries from pull
+   request commit messages and merge them into the manually maintained
+   `CHANGELOG.md`.
+3. Update the CHANGELOG.md with [Unreleased] changed to v<VERSION>-rc1. Note that
    you should exactly copy the date and name format from a previous
    release, as the `build-release.sh` script relies on this.
-3. Create a PR with the above.
+4. Create a PR with the above.
 
 ### Releasing -rc1
 

doc/STYLE.md

@@ -160,3 +160,40 @@ Try to make a single change at a time.  It's tempting to do "drive-by"
 fixes as you see other things, and a minimal amount is unavoidable, but
 you can end up shaving infinite yaks.  This is a good time to drop a 
 `/* FIXME: ...*/` comment and move on.
+
+## Github Workflows
+
+We have adopted a number of workflows to facilitate the development of
+c-lightning, and to make things more pleasant for contributors.
+
+### Changelog Entries in Commit Messages
+
+We are maintaining a chanelog in the top-level directory of this
+project. However since every pull request has a tendency to touch the file and
+therefore create merge-conflicts we decided to derive the changelog file from
+the pull requests that were added between releases. In order for a pull
+request to show up in the changelog at least one of its commits will have to
+have a line with one of the following prefixes:
+
+ - `Changelog-Added: ` if the pull request adds a new feature
+ - `Changelog-Changed: ` if a feature has been modified and might require
+   changes on the user side
+ - `Changelog-Deprecated: ` if a feature has been marked for deprecation, but
+   not yet removed
+ - `Changelog-Fixed: ` if a bug has been fixed
+ - `Changelog-Removed: ` if a (previously deprecated) feature has been removed
+ - `Changelog-Security: ` if a security issue has been addressed and the users
+   will need to upgrade in order to stay secure
+ 
+In case you think the pull request is small enough not to require a changelog
+entry please use `Changelog-None` in one of the commit messages to opt out.
+ 
+Under some circumstances a feature may be removed even without deprecation
+warning if it was not part of a released version yet, or the removal is
+urgent.
+
+In order to ensure that each pull request has the required `Changelog-*:` line
+for the changelog our trusty @bitcoin-bot will check logs whenever a pull
+request is created or updated and search for the required line. If there is no
+such line it'll mark the pull request as `pending` to call out the need for an
+entry.