leandrof/u-boot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add 'patman' patch generation, checking and submission script

Browse Source 
What is this?

=============

This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people

It is intended to automate patch creation and make it a less
error-prone process. It is useful for U-Boot and Linux work so far,
since it uses the checkpatch.pl script.

It is configured almost entirely by tags it finds in your commits.
This means that you can work on a number of different branches at
once, and keep the settings with each branch rather than having to
git format-patch, git send-email, etc. with the correct parameters
each time. So for example if you put:

in one of your commits, the series will be sent there.

See the README file for full details.
END

Signed-off-by: Simon Glass <sjg@chromium.org>
This commit is contained in:
Simon Glass 2012-01-14 15:12:45 +00:00 committed by Wolfgang Denk
parent 0d62032e5b
commit 0d24de9d55
13 changed files with 2354 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
tools/patman/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
*.pyc

408
tools/patman/README Normal file
View File

@ -0,0 +1,408 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
What is this?
=============
This tool is a Python script which:
- Creates patch directly from your branch
- Cleans them up by removing unwanted tags
- Inserts a cover letter with change lists
- Runs the patches through checkpatch.pl and its own checks
- Optionally emails them out to selected people
It is intended to automate patch creation and make it a less
error-prone process. It is useful for U-Boot and Linux work so far,
since it uses the checkpatch.pl script.
It is configured almost entirely by tags it finds in your commits.
This means that you can work on a number of different branches at
once, and keep the settings with each branch rather than having to
git format-patch, git send-email, etc. with the correct parameters
each time. So for example if you put:
Series-to: fred.blogs@napier.co.nz
in one of your commits, the series will be sent there.
How to use this tool
====================
This tool requires a certain way of working:
- Maintain a number of branches, one for each patch series you are
working on
- Add tags into the commits within each branch to indicate where the
series should be sent, cover letter, version, etc. Most of these are
normally in the top commit so it is easy to change them with 'git
commit --amend'
- Each branch tracks the upstream branch, so that this script can
automatically determine the number of commits in it (optional)
- Check out a branch, and run this script to create and send out your
patches. Weeks later, change the patches and repeat, knowing that you
will get a consistent result each time.
How to configure it
===================
For most cases patman will locate and use the file 'doc/git-mailrc' in
your U-Boot directory. This contains most of the aliases you will need.
To add your own, create a file ~/.config/patman directory like this:
>>>>
# patman alias file
[alias]
me: Simon Glass <sjg@chromium.org>
u-boot: U-Boot Mailing List <u-boot@lists.denx.de>
wolfgang: Wolfgang Denk <wd@denx.de>
others: Mike Frysinger <vapier@gentoo.org>, Fred Bloggs <f.bloggs@napier.net>
<<<<
Aliases are recursive.
The checkpatch.pl in the U-Boot tools/ subdirectory will be located and
used. Failing that you can put it into your path or ~/bin/checkpatch.pl
How to run it
=============
First do a dry run:
$ ./tools/scripts/patman/patman -n
If it can't detect the upstream branch, try telling it how many patches
there are in your series:
$ ./tools/scripts/patman/patman -n -c5
This will create patch files in your current directory and tell you who
it is thinking of sending them to. Take a look at the patch files.
$ ./tools/scripts/patman/patman -n -c5 -s1
Similar to the above, but skip the first commit and take the next 5. This
is useful if your top commit is for setting up testing.
How to add tags
===============
To make this script useful you must add tags like the following into any
commit. Most can only appear once in the whole series.
Series-to: email / alias
Email address / alias to send patch series to (you can add this
multiple times)
Series-cc: email / alias, ...
Email address / alias to Cc patch series to (you can add this
multiple times)
Series-version: n
Sets the version number of this patch series
Series-prefix: prefix
Sets the subject prefix. Normally empty but it can be RFC for
RFC patches, or RESEND if you are being ignored.
Cover-letter:
This is the patch set title
blah blah
more blah blah
END
Sets the cover letter contents for the series. The first line
will become the subject of the cover letter
Series-notes:
blah blah
blah blah
more blah blah
END
Sets some notes for the patch series, which you don't want in
the commit messages, but do want to send, The notes are joined
together and put after the cover letter. Can appear multiple
times.
Signed-off-by: Their Name <email>
A sign-off is added automatically to your patches (this is
probably a bug). If you put this tag in your patches, it will
override the default signoff that patman automatically adds.
Tested-by: Their Name <email>
Acked-by: Their Name <email>
These indicate that someone has acked or tested your patch.
When you get this reply on the mailing list, you can add this
tag to the relevant commit and the script will include it when
you send out the next version. If 'Tested-by:' is set to
yourself, it will be removed. No one will believe you.
Series-changes: n
- Guinea pig moved into its cage
- Other changes ending with a blank line
<blank line>
This can appear in any commit. It lists the changes for a
particular version n of that commit. The change list is
created based on this information. Each commit gets its own
change list and also the whole thing is repeated in the cover
letter (where duplicate change lines are merged).
By adding your change lists into your commits it is easier to
keep track of what happened. When you amend a commit, remember
to update the log there and then, knowing that the script will
do the rest.
Cc: Their Name <email>
This copies a single patch to another email address.
Various other tags are silently removed, like these Chrome OS and
Gerrit tags:
BUG=...
TEST=...
Change-Id:
Review URL:
Reviewed-on:
Reviewed-by:
Exercise for the reader: Try adding some tags to one of your current
patch series and see how the patches turn out.
Where Patches Are Sent
======================
Once the patches are created, patman sends them using gti send-email. The
whole series is sent to the recipients in Series-to: and Series-cc.
You can Cc individual patches to other people with the Cc: tag. Tags in the
subject are also picked up to Cc patches. For example, a commit like this:
>>>>
commit 10212537b85ff9b6e09c82045127522c0f0db981
Author: Mike Frysinger <vapier@gentoo.org>
Date: Mon Nov 7 23:18:44 2011 -0500
x86: arm: add a git mailrc file for maintainers
This should make sending out e-mails to the right people easier.
Cc: sandbox, mikef, ag
Cc: afleming
<<<<
will create a patch which is copied to x86, arm, sandbox, mikef, ag and
afleming.
Example Work Flow
=================
The basic workflow is to create your commits, add some tags to the top
commit, and type 'patman' to check and send them.
Here is an example workflow for a series of 4 patches. Let's say you have
these rather contrived patches in the following order in branch us-cmd in
your tree where 'us' means your upstreaming activity (newest to oldest as
output by git log --oneline):
7c7909c wip
89234f5 Don't include standard parser if hush is used
8d640a7 mmc: sparc: Stop using builtin_run_command()
0c859a9 Rename run_command2() to run_command()
a74443f sandbox: Rename run_command() to builtin_run_command()
The first patch is some test things that enable your code to be compiled,
but that you don't want to submit because there is an existing patch for it
on the list. So you can tell patman to create and check some patches
(skipping the first patch) with:
patman -s1 -n
If you want to do all of them including the work-in-progress one, then
(if you are tracking an upstream branch):
patman -n
Let's say that patman reports an error in the second patch. Then:
git rebase -i HEAD~6
<change 'pick' to 'edit' in 89234f5>
<use editor to make code changes>
git add -u
git rebase --continue
Now you have an updated patch series. To check it:
patman -s1 -n
Let's say it is now clean and you want to send it. Now you need to set up
the destination. So amend the top commit with:
git commit --amend
Use your editor to add some tags, so that the whole commit message is:
The current run_command() is really only one of the options, with
hush providing the other. It really shouldn't be called directly
in case the hush parser is bring used, so rename this function to
better explain its purpose.
Series-to: u-boot
Series-cc: bfin, marex
Series-prefix: RFC
Cover-letter:
Unified command execution in one place
At present two parsers have similar code to execute commands. Also
cmd_usage() is called all over the place. This series adds a single
function which processes commands called cmd_process().
END
Change-Id: Ica71a14c1f0ecb5650f771a32fecb8d2eb9d8a17
You want this to be an RFC and Cc the whole series to the bfin alias and
to Marek. Two of the patches have tags (those are the bits at the front of
the subject that say mmc: sparc: and sandbox:), so 8d640a7 will be Cc'd to
mmc and sparc, and the last one to sandbox.
Now to send the patches, take off the -n flag:
patman -s1
The patches will be created, shown in your editor, and then sent along with
the cover letter. Note that patman's tags are automatically removed so that
people on the list don't see your secret info.
Of course patches often attract comments and you need to make some updates.
Let's say one person sent comments and you get an Acked-by: on one patch.
Also, the patch on the list that you were waiting for has been merged,
so you can drop your wip commit. So you resync with upstream:
git fetch origin (or whatever upstream is called)
git rebase origin/master
and use git rebase -i to edit the commits, dropping the wip one. You add
the ack tag to one commit:
Acked-by: Heiko Schocher <hs@denx.de>
update the Series-cc: in the top commit:
Series-cc: bfin, marex, Heiko Schocher <hs@denx.de>
and remove the Series-prefix: tag since it it isn't an RFC any more. The
series is now version two, so the series info in the top commit looks like
this:
Series-to: u-boot
Series-cc: bfin, marex, Heiko Schocher <hs@denx.de>
Series-version: 2
Cover-letter:
...
Finally, you need to add a change log to the two commits you changed. You
add change logs to each individual commit where the changes happened, like
this:
Series-changes: 2
- Updated the command decoder to reduce code size
- Wound the torque propounder up a little more
(note the blank line at the end of the list)
When you run patman it will collect all the change logs from the different
commits and combine them into the cover letter, if you have one. So finally
you have a new series of commits:
faeb973 Don't include standard parser if hush is used
1b2f2fe mmc: sparc: Stop using builtin_run_command()
cfbe330 Rename run_command2() to run_command()
0682677 sandbox: Rename run_command() to builtin_run_command()
so to send them:
patman
and it will create and send the version 2 series.
General points:
1. When you change back to the us-cmd branch days or weeks later all your
information is still there, safely stored in the commits. You don't need
to remember what version you are up to, who you sent the last lot of patches
to, or anything about the change logs.
2. If you put tags in the subject, patman will Cc the maintainers
automatically in many cases.
3. If you want to keep the commits from each series you sent so that you can
compare change and see what you did, you can either create a new branch for
each version, or just tag the branch before you start changing it:
git tag sent/us-cmd-rfc
...later...
git tag sent/us-cmd-v2
4. If you want to modify the patches a little before sending, you can do
this in your editor, but be careful!
5. If you want to run git send-email yourself, use the -n flag which will
print out the command line patman would have used.
6. It is a good idea to add the change log info as you change the commit,
not later when you can't remember which patch you changed. You can always
go back and change or remove logs from commits.
Other thoughts
==============
This script has been split into sensible files but still needs work.
Most of these are indicated by a TODO in the code.
It would be nice if this could handle the In-reply-to side of things.
The tests are incomplete, as is customary. Use the -t flag to run them,
and make sure you are in the tools/scripts/patman directory first:
$ cd /path/to/u-boot
$ cd tools/scripts/patman
$ patman -t
Error handling doesn't always produce friendly error messages - e.g.
putting an incorrect tag in a commit may provide a confusing message.
There might be a few other features not mentioned in this README. They
might be bugs. In particular, tags are case sensitive which is probably
a bad thing.
Simon Glass <sjg@chromium.org>
v1, v2, 19-Oct-11
revised v3 24-Nov-11

161
tools/patman/checkpatch.py Normal file
View File

@ -0,0 +1,161 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import command
import gitutil
import os
import re
import terminal
def FindCheckPatch():
try_list = [
os.getcwd(),
os.path.join(os.getcwd(), '..', '..'),
os.path.join(gitutil.GetTopLevel(), 'tools'),
'%s/bin' % os.getenv('HOME'),
]
# Look in current dir
for path in try_list:
fname = os.path.join(path, 'checkpatch.pl')
if os.path.isfile(fname):
return fname
# Look upwwards for a Chrome OS tree
while not os.path.ismount(path):
fname = os.path.join(path, 'src', 'third_party', 'kernel', 'files',
'scripts', 'checkpatch.pl')
if os.path.isfile(fname):
return fname
path = os.path.dirname(path)
print 'Could not find checkpatch.pl'
return None
def CheckPatch(fname, verbose=False):
"""Run checkpatch.pl on a file.
Returns:
4-tuple containing:
result: False=failure, True=ok
problems: List of problems, each a dict:
'type'; error or warning
'msg': text message
'file' : filename
'line': line number
lines: Number of lines
"""
result = False
error_count, warning_count, lines = 0, 0, 0
problems = []
chk = FindCheckPatch()
if not chk:
raise OSError, ('Cannot find checkpatch.pl - please put it in your ' +
'~/bin directory')
item = {}
stdout = command.Output(chk, '--no-tree', fname)
#pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE)
#stdout, stderr = pipe.communicate()
# total: 0 errors, 0 warnings, 159 lines checked
re_stats = re.compile('total: (\\d+) errors, (\d+) warnings, (\d+)')
re_ok = re.compile('.*has no obvious style problems')
re_bad = re.compile('.*has style problems, please review')
re_error = re.compile('ERROR: (.*)')
re_warning = re.compile('WARNING: (.*)')
re_file = re.compile('#\d+: FILE: ([^:]*):(\d+):')
for line in stdout.splitlines():
if verbose:
print line
# A blank line indicates the end of a message
if not line and item:
problems.append(item)
item = {}
match = re_stats.match(line)
if match:
error_count = int(match.group(1))
warning_count = int(match.group(2))
lines = int(match.group(3))
elif re_ok.match(line):
result = True
elif re_bad.match(line):
result = False
match = re_error.match(line)
if match:
item['msg'] = match.group(1)
item['type'] = 'error'
match = re_warning.match(line)
if match:
item['msg'] = match.group(1)
item['type'] = 'warning'
match = re_file.match(line)
if match:
item['file'] = match.group(1)
item['line'] = int(match.group(2))
return result, problems, error_count, warning_count, lines, stdout
def GetWarningMsg(col, msg_type, fname, line, msg):
'''Create a message for a given file/line
Args:
msg_type: Message type ('error' or 'warning')
fname: Filename which reports the problem
line: Line number where it was noticed
msg: Message to report
'''
if msg_type == 'warning':
msg_type = col.Color(col.YELLOW, msg_type)
elif msg_type == 'error':
msg_type = col.Color(col.RED, msg_type)
return '%s: %s,%d: %s' % (msg_type, fname, line, msg)
def CheckPatches(verbose, args):
'''Run the checkpatch.pl script on each patch'''
error_count = 0
warning_count = 0
col = terminal.Color()
for fname in args:
ok, problems, errors, warnings, lines, stdout = CheckPatch(fname,
verbose)
if not ok:
error_count += errors
warning_count += warnings
print '%d errors, %d warnings for %s:' % (errors,
warnings, fname)
if len(problems) != error_count + warning_count:
print "Internal error: some problems lost"
for item in problems:
print GetWarningMsg(col, item['type'], item['file'],
item['line'], item['msg'])
#print stdout
if error_count != 0 or warning_count != 0:
str = 'checkpatch.pl found %d error(s), %d warning(s)' % (
error_count, warning_count)
color = col.GREEN
if warning_count:
color = col.YELLOW
if error_count:
color = col.RED
print col.Color(color, str)
return False
return True

72
tools/patman/command.py Normal file
View File

@ -0,0 +1,72 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import os
import subprocess
"""Shell command ease-ups for Python."""
def RunPipe(pipeline, infile=None, outfile=None,
capture=False, oneline=False, hide_stderr=False):
"""
Perform a command pipeline, with optional input/output filenames.
hide_stderr Don't allow output of stderr (default False)
"""
last_pipe = None
while pipeline:
cmd = pipeline.pop(0)
kwargs = {}
if last_pipe is not None:
kwargs['stdin'] = last_pipe.stdout
elif infile:
kwargs['stdin'] = open(infile, 'rb')
if pipeline or capture:
kwargs['stdout'] = subprocess.PIPE
elif outfile:
kwargs['stdout'] = open(outfile, 'wb')
if hide_stderr:
kwargs['stderr'] = open('/dev/null', 'wb')
last_pipe = subprocess.Popen(cmd, **kwargs)
if capture:
ret = last_pipe.communicate()[0]
if not ret:
return None
elif oneline:
return ret.rstrip('\r\n')
else:
return ret
else:
return os.waitpid(last_pipe.pid, 0)[1] == 0
def Output(*cmd):
return RunPipe([cmd], capture=True)
def OutputOneLine(*cmd):
return RunPipe([cmd], capture=True, oneline=True)
def Run(*cmd, **kwargs):
return RunPipe([cmd], **kwargs)
def RunList(cmd):
return RunPipe([cmd], capture=True)

87
tools/patman/commit.py Normal file
View File

@ -0,0 +1,87 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import re
# Separates a tag: at the beginning of the subject from the rest of it
re_subject_tag = re.compile('([^:]*):\s*(.*)')
class Commit:
"""Holds information about a single commit/patch in the series.
Args:
hash: Commit hash (as a string)
Variables:
hash: Commit hash
subject: Subject line
tags: List of maintainer tag strings
changes: Dict containing a list of changes (single line strings).
The dict is indexed by change version (an integer)
cc_list: List of people to aliases/emails to cc on this commit
"""
def __init__(self, hash):
self.hash = hash
self.subject = None
self.tags = []
self.changes = {}
self.cc_list = []
def AddChange(self, version, info):
"""Add a new change line to the change list for a version.
Args:
version: Patch set version (integer: 1, 2, 3)
info: Description of change in this version
"""
if not self.changes.get(version):
self.changes[version] = []
self.changes[version].append(info)
def CheckTags(self):
"""Create a list of subject tags in the commit
Subject tags look like this:
propounder: Change the widget to propound correctly
Multiple tags are supported. The list is updated in self.tag
Returns:
None if ok, else the name of a tag with no email alias
"""
str = self.subject
m = True
while m:
m = re_subject_tag.match(str)
if m:
tag = m.group(1)
self.tags.append(tag)
str = m.group(2)
return None
def AddCc(self, cc_list):
"""Add a list of people to Cc when we send this patch.
Args:
cc_list: List of aliases or email addresses
"""
self.cc_list += cc_list

372
tools/patman/gitutil.py Normal file
View File

@ -0,0 +1,372 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import command
import re
import os
import series
import settings
import subprocess
import sys
import terminal
def CountCommitsToBranch():
"""Returns number of commits between HEAD and the tracking branch.
This looks back to the tracking branch and works out the number of commits
since then.
Return:
Number of patches that exist on top of the branch
"""
pipe = [['git', 'log', '--oneline', '@{upstream}..'],
['wc', '-l']]
stdout = command.RunPipe(pipe, capture=True, oneline=True)
patch_count = int(stdout)
return patch_count
def CreatePatches(start, count, series):
"""Create a series of patches from the top of the current branch.
The patch files are written to the current directory using
git format-patch.
Args:
start: Commit to start from: 0=HEAD, 1=next one, etc.
count: number of commits to include
Return:
Filename of cover letter
List of filenames of patch files
"""
if series.get('version'):
version = '%s ' % series['version']
cmd = ['git', 'format-patch', '-M', '--signoff']
if series.get('cover'):
cmd.append('--cover-letter')
prefix = series.GetPatchPrefix()
if prefix:
cmd += ['--subject-prefix=%s' % prefix]
cmd += ['HEAD~%d..HEAD~%d' % (start + count, start)]
stdout = command.RunList(cmd)
files = stdout.splitlines()
# We have an extra file if there is a cover letter
if series.get('cover'):
return files[0], files[1:]
else:
return None, files
def ApplyPatch(verbose, fname):
"""Apply a patch with git am to test it
TODO: Convert these to use command, with stderr option
Args:
fname: filename of patch file to apply
"""
cmd = ['git', 'am', fname]
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = pipe.communicate()
re_error = re.compile('^error: patch failed: (.+):(\d+)')
for line in stderr.splitlines():
if verbose:
print line
match = re_error.match(line)
if match:
print GetWarningMsg('warning', match.group(1), int(match.group(2)),
'Patch failed')
return pipe.returncode == 0, stdout
def ApplyPatches(verbose, args, start_point):
"""Apply the patches with git am to make sure all is well
Args:
verbose: Print out 'git am' output verbatim
args: List of patch files to apply
start_point: Number of commits back from HEAD to start applying.
Normally this is len(args), but it can be larger if a start
offset was given.
"""
error_count = 0
col = terminal.Color()
# Figure out our current position
cmd = ['git', 'name-rev', 'HEAD', '--name-only']
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE)
stdout, stderr = pipe.communicate()
if pipe.returncode:
str = 'Could not find current commit name'
print col.Color(col.RED, str)
print stdout
return False
old_head = stdout.splitlines()[0]
# Checkout the required start point
cmd = ['git', 'checkout', 'HEAD~%d' % start_point]
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = pipe.communicate()
if pipe.returncode:
str = 'Could not move to commit before patch series'
print col.Color(col.RED, str)
print stdout, stderr
return False
# Apply all the patches
for fname in args:
ok, stdout = ApplyPatch(verbose, fname)
if not ok:
print col.Color(col.RED, 'git am returned errors for %s: will '
'skip this patch' % fname)
if verbose:
print stdout
error_count += 1
cmd = ['git', 'am', '--skip']
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE)
stdout, stderr = pipe.communicate()
if pipe.returncode != 0:
print col.Color(col.RED, 'Unable to skip patch! Aborting...')
print stdout
break
# Return to our previous position
cmd = ['git', 'checkout', old_head]
pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = pipe.communicate()
if pipe.returncode:
print col.Color(col.RED, 'Could not move back to head commit')
print stdout, stderr
return error_count == 0
def BuildEmailList(in_list, tag=None, alias=None):
"""Build a list of email addresses based on an input list.
Takes a list of email addresses and aliases, and turns this into a list
of only email address, by resolving any aliases that are present.
If the tag is given, then each email address is prepended with this
tag and a space. If the tag starts with a minus sign (indicating a
command line parameter) then the email address is quoted.
Args:
in_list: List of aliases/email addresses
tag: Text to put before each address
Returns:
List of email addresses
>>> alias = {}
>>> alias['fred'] = ['f.bloggs@napier.co.nz']
>>> alias['john'] = ['j.bloggs@napier.co.nz']
>>> alias['mary'] = ['Mary Poppins <m.poppins@cloud.net>']
>>> alias['boys'] = ['fred', ' john']
>>> alias['all'] = ['fred ', 'john', ' mary ']
>>> BuildEmailList(['john', 'mary'], None, alias)
['j.bloggs@napier.co.nz', 'Mary Poppins <m.poppins@cloud.net>']
>>> BuildEmailList(['john', 'mary'], '--to', alias)
['--to "j.bloggs@napier.co.nz"', \
'--to "Mary Poppins <m.poppins@cloud.net>"']
>>> BuildEmailList(['john', 'mary'], 'Cc', alias)
['Cc j.bloggs@napier.co.nz', 'Cc Mary Poppins <m.poppins@cloud.net>']
"""
quote = '"' if tag and tag[0] == '-' else ''
raw = []
for item in in_list:
raw += LookupEmail(item, alias)
result = []
for item in raw:
if not item in result:
result.append(item)
if tag:
return ['%s %s%s%s' % (tag, quote, email, quote) for email in result]
return result
def EmailPatches(series, cover_fname, args, dry_run, cc_fname,
self_only=False, alias=None):
"""Email a patch series.
Args:
series: Series object containing destination info
cover_fname: filename of cover letter
args: list of filenames of patch files
dry_run: Just return the command that would be run
cc_fname: Filename of Cc file for per-commit Cc
self_only: True to just email to yourself as a test
Returns:
Git command that was/would be run
>>> alias = {}
>>> alias['fred'] = ['f.bloggs@napier.co.nz']
>>> alias['john'] = ['j.bloggs@napier.co.nz']
>>> alias['mary'] = ['m.poppins@cloud.net']
>>> alias['boys'] = ['fred', ' john']
>>> alias['all'] = ['fred ', 'john', ' mary ']
>>> alias[os.getenv('USER')] = ['this-is-me@me.com']
>>> series = series.Series()
>>> series.to = ['fred']
>>> series.cc = ['mary']
>>> EmailPatches(series, 'cover', ['p1', 'p2'], True, 'cc-fname', False, \
alias)
'git send-email --annotate --to "f.bloggs@napier.co.nz" --cc \
"m.poppins@cloud.net" --cc-cmd "./patman --cc-cmd cc-fname" cover p1 p2'
>>> EmailPatches(series, None, ['p1'], True, 'cc-fname', False, alias)
'git send-email --annotate --to "f.bloggs@napier.co.nz" --cc \
"m.poppins@cloud.net" --cc-cmd "./patman --cc-cmd cc-fname" p1'
>>> series.cc = ['all']
>>> EmailPatches(series, 'cover', ['p1', 'p2'], True, 'cc-fname', True, \
alias)
'git send-email --annotate --to "this-is-me@me.com" --cc-cmd "./patman \
--cc-cmd cc-fname" cover p1 p2'
>>> EmailPatches(series, 'cover', ['p1', 'p2'], True, 'cc-fname', False, \
alias)
'git send-email --annotate --to "f.bloggs@napier.co.nz" --cc \
"f.bloggs@napier.co.nz" --cc "j.bloggs@napier.co.nz" --cc \
"m.poppins@cloud.net" --cc-cmd "./patman --cc-cmd cc-fname" cover p1 p2'
"""
to = BuildEmailList(series.get('to'), '--to', alias)
if not to:
print ("No recipient, please add something like this to a commit\n"
"Series-to: Fred Bloggs <f.blogs@napier.co.nz>")
return
cc = BuildEmailList(series.get('cc'), '--cc', alias)
if self_only:
to = BuildEmailList([os.getenv('USER')], '--to', alias)
cc = []
cmd = ['git', 'send-email', '--annotate']
cmd += to
cmd += cc
cmd += ['--cc-cmd', '"%s --cc-cmd %s"' % (sys.argv[0], cc_fname)]
if cover_fname:
cmd.append(cover_fname)
cmd += args
str = ' '.join(cmd)
if not dry_run:
os.system(str)
return str
def LookupEmail(lookup_name, alias=None, level=0):
"""If an email address is an alias, look it up and return the full name
TODO: Why not just use git's own alias feature?
Args:
lookup_name: Alias or email address to look up
Returns:
tuple:
list containing a list of email addresses
Raises:
OSError if a recursive alias reference was found
ValueError if an alias was not found
>>> alias = {}
>>> alias['fred'] = ['f.bloggs@napier.co.nz']
>>> alias['john'] = ['j.bloggs@napier.co.nz']
>>> alias['mary'] = ['m.poppins@cloud.net']
>>> alias['boys'] = ['fred', ' john', 'f.bloggs@napier.co.nz']
>>> alias['all'] = ['fred ', 'john', ' mary ']
>>> alias['loop'] = ['other', 'john', ' mary ']
>>> alias['other'] = ['loop', 'john', ' mary ']
>>> LookupEmail('mary', alias)
['m.poppins@cloud.net']
>>> LookupEmail('arthur.wellesley@howe.ro.uk', alias)
['arthur.wellesley@howe.ro.uk']
>>> LookupEmail('boys', alias)
['f.bloggs@napier.co.nz', 'j.bloggs@napier.co.nz']
>>> LookupEmail('all', alias)
['f.bloggs@napier.co.nz', 'j.bloggs@napier.co.nz', 'm.poppins@cloud.net']
>>> LookupEmail('odd', alias)
Traceback (most recent call last):
...
ValueError: Alias 'odd' not found
>>> LookupEmail('loop', alias)
Traceback (most recent call last):
...
OSError: Recursive email alias at 'other'
"""
if not alias:
alias = settings.alias
lookup_name = lookup_name.strip()
if '@' in lookup_name: # Perhaps a real email address
return [lookup_name]
lookup_name = lookup_name.lower()
if level > 10:
raise OSError, "Recursive email alias at '%s'" % lookup_name
out_list = []
if lookup_name:
if not lookup_name in alias:
raise ValueError, "Alias '%s' not found" % lookup_name
for item in alias[lookup_name]:
todo = LookupEmail(item, alias, level + 1)
for new_item in todo:
if not new_item in out_list:
out_list.append(new_item)
#print "No match for alias '%s'" % lookup_name
return out_list
def GetTopLevel():
"""Return name of top-level directory for this git repo.
Returns:
Full path to git top-level directory
This test makes sure that we are running tests in the right subdir
>>> os.path.realpath(os.getcwd()) == \
os.path.join(GetTopLevel(), 'tools', 'scripts', 'patman')
True
"""
return command.OutputOneLine('git', 'rev-parse', '--show-toplevel')
def GetAliasFile():
"""Gets the name of the git alias file.
Returns:
Filename of git alias file, or None if none
"""
fname = command.OutputOneLine('git', 'config', 'sendemail.aliasesfile')
if fname:
fname = os.path.join(GetTopLevel(), fname.strip())
return fname
def Setup():
"""Set up git utils, by reading the alias files."""
settings.Setup('')
# Check for a git alias file also
alias_fname = GetAliasFile()
if alias_fname:
settings.ReadGitAliases(alias_fname)
if __name__ == "__main__":
import doctest
doctest.testmod()

444
tools/patman/patchstream.py Normal file
View File

@ -0,0 +1,444 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import os
import re
import shutil
import tempfile
import command
import commit
import gitutil
from series import Series
# Tags that we detect and remove
re_remove = re.compile('^BUG=|^TEST=|^Change-Id:|^Review URL:'
'|Reviewed-on:|Reviewed-by:')
# Lines which are allowed after a TEST= line
re_allowed_after_test = re.compile('^Signed-off-by:')
# The start of the cover letter
re_cover = re.compile('^Cover-letter:')
# Patch series tag
re_series = re.compile('^Series-(\w*): *(.*)')
# Commit tags that we want to collect and keep
re_tag = re.compile('^(Tested-by|Acked-by|Signed-off-by|Cc): (.*)')
# The start of a new commit in the git log
re_commit = re.compile('^commit (.*)')
# We detect these since checkpatch doesn't always do it
re_space_before_tab = re.compile('^[+].* \t')
# States we can be in - can we use range() and still have comments?
STATE_MSG_HEADER = 0 # Still in the message header
STATE_PATCH_SUBJECT = 1 # In patch subject (first line of log for a commit)
STATE_PATCH_HEADER = 2 # In patch header (after the subject)
STATE_DIFFS = 3 # In the diff part (past --- line)
class PatchStream:
"""Class for detecting/injecting tags in a patch or series of patches
We support processing the output of 'git log' to read out the tags we
are interested in. We can also process a patch file in order to remove
unwanted tags or inject additional ones. These correspond to the two
phases of processing.
"""
def __init__(self, series, name=None, is_log=False):
self.skip_blank = False # True to skip a single blank line
self.found_test = False # Found a TEST= line
self.lines_after_test = 0 # MNumber of lines found after TEST=
self.warn = [] # List of warnings we have collected
self.linenum = 1 # Output line number we are up to
self.in_section = None # Name of start...END section we are in
self.notes = [] # Series notes
self.section = [] # The current section...END section
self.series = series # Info about the patch series
self.is_log = is_log # True if indent like git log
self.in_change = 0 # Non-zero if we are in a change list
self.blank_count = 0 # Number of blank lines stored up
self.state = STATE_MSG_HEADER # What state are we in?
self.tags = [] # Tags collected, like Tested-by...
self.signoff = [] # Contents of signoff line
self.commit = None # Current commit
def AddToSeries(self, line, name, value):
"""Add a new Series-xxx tag.
When a Series-xxx tag is detected, we come here to record it, if we
are scanning a 'git log'.
Args:
line: Source line containing tag (useful for debug/error messages)
name: Tag name (part after 'Series-')
value: Tag value (part after 'Series-xxx: ')
"""
if name == 'notes':
self.in_section = name
self.skip_blank = False
if self.is_log:
self.series.AddTag(self.commit, line, name, value)
def CloseCommit(self):
"""Save the current commit into our commit list, and reset our state"""
if self.commit and self.is_log:
self.series.AddCommit(self.commit)
self.commit = None
def FormatTags(self, tags):
out_list = []
for tag in sorted(tags):
if tag.startswith('Cc:'):
tag_list = tag[4:].split(',')
out_list += gitutil.BuildEmailList(tag_list, 'Cc:')
else:
out_list.append(tag)
return out_list
def ProcessLine(self, line):
"""Process a single line of a patch file or commit log
This process a line and returns a list of lines to output. The list
may be empty or may contain multiple output lines.
This is where all the complicated logic is located. The class's
state is used to move between different states and detect things
properly.
We can be in one of two modes:
self.is_log == True: This is 'git log' mode, where most output is
indented by 4 characters and we are scanning for tags
self.is_log == False: This is 'patch' mode, where we already have
all the tags, and are processing patches to remove junk we
don't want, and add things we think are required.
Args:
line: text line to process
Returns:
list of output lines, or [] if nothing should be output
"""
# Initially we have no output. Prepare the input line string
out = []
line = line.rstrip('\n')
if self.is_log:
if line[:4] == ' ':
line = line[4:]
# Handle state transition and skipping blank lines
series_match = re_series.match(line)
commit_match = re_commit.match(line) if self.is_log else None
tag_match = None
if self.state == STATE_PATCH_HEADER:
tag_match = re_tag.match(line)
is_blank = not line.strip()
if is_blank:
if (self.state == STATE_MSG_HEADER
or self.state == STATE_PATCH_SUBJECT):
self.state += 1
# We don't have a subject in the text stream of patch files
# It has its own line with a Subject: tag
if not self.is_log and self.state == STATE_PATCH_SUBJECT:
self.state += 1
elif commit_match:
self.state = STATE_MSG_HEADER
# If we are in a section, keep collecting lines until we see END
if self.in_section:
if line == 'END':
if self.in_section == 'cover':
self.series.cover = self.section
elif self.in_section == 'notes':
if self.is_log:
self.series.notes += self.section
else:
self.warn.append("Unknown section '%s'" % self.in_section)
self.in_section = None
self.skip_blank = True
self.section = []
else:
self.section.append(line)
# Detect the commit subject
elif not is_blank and self.state == STATE_PATCH_SUBJECT:
self.commit.subject = line
# Detect the tags we want to remove, and skip blank lines
elif re_remove.match(line):
self.skip_blank = True
# TEST= should be the last thing in the commit, so remove
# everything after it
if line.startswith('TEST='):
self.found_test = True
elif self.skip_blank and is_blank:
self.skip_blank = False
# Detect the start of a cover letter section
elif re_cover.match(line):
self.in_section = 'cover'
self.skip_blank = False
# If we are in a change list, key collected lines until a blank one
elif self.in_change:
if is_blank:
# Blank line ends this change list
self.in_change = 0
else:
self.series.AddChange(self.in_change, self.commit, line)
self.skip_blank = False
# Detect Series-xxx tags
elif series_match:
name = series_match.group(1)
value = series_match.group(2)
if name == 'changes':
# value is the version number: e.g. 1, or 2
try:
value = int(value)
except ValueError as str:
raise ValueError("%s: Cannot decode version info '%s'" %
(self.commit.hash, line))
self.in_change = int(value)
else:
self.AddToSeries(line, name, value)
self.skip_blank = True
# Detect the start of a new commit
elif commit_match:
self.CloseCommit()
self.commit = commit.Commit(commit_match.group(1)[:7])
# Detect tags in the commit message
elif tag_match:
# Onlly allow a single signoff tag
if tag_match.group(1) == 'Signed-off-by':
if self.signoff:
self.warn.append('Patch has more than one Signed-off-by '
'tag')
self.signoff += [line]
# Remove Tested-by self, since few will take much notice
elif (tag_match.group(1) == 'Tested-by' and
tag_match.group(2).find(os.getenv('USER') + '@') != -1):
self.warn.append("Ignoring %s" % line)
elif tag_match.group(1) == 'Cc':
self.commit.AddCc(tag_match.group(2).split(','))
else:
self.tags.append(line);
# Well that means this is an ordinary line
else:
pos = 1
# Look for ugly ASCII characters
for ch in line:
# TODO: Would be nicer to report source filename and line
if ord(ch) > 0x80:
self.warn.append("Line %d/%d ('%s') has funny ascii char" %
(self.linenum, pos, line))
pos += 1
# Look for space before tab
m = re_space_before_tab.match(line)
if m:
self.warn.append('Line %d/%d has space before tab' %
(self.linenum, m.start()))
# OK, we have a valid non-blank line
out = [line]
self.linenum += 1
self.skip_blank = False
if self.state == STATE_DIFFS:
pass
# If this is the start of the diffs section, emit our tags and
# change log
elif line == '---':
self.state = STATE_DIFFS
# Output the tags (signeoff first), then change list
out = []
if self.signoff:
out += self.signoff
log = self.series.MakeChangeLog(self.commit)
out += self.FormatTags(self.tags)
out += [line] + log
elif self.found_test:
if not re_allowed_after_test.match(line):
self.lines_after_test += 1
return out
def Finalize(self):
"""Close out processing of this patch stream"""
self.CloseCommit()
if self.lines_after_test:
self.warn.append('Found %d lines after TEST=' %
self.lines_after_test)
def ProcessStream(self, infd, outfd):
"""Copy a stream from infd to outfd, filtering out unwanting things.
This is used to process patch files one at a time.
Args:
infd: Input stream file object
outfd: Output stream file object
"""
# Extract the filename from each diff, for nice warnings
fname = None
last_fname = None
re_fname = re.compile('diff --git a/(.*) b/.*')
while True:
line = infd.readline()
if not line:
break
out = self.ProcessLine(line)
# Try to detect blank lines at EOF
for line in out:
match = re_fname.match(line)
if match:
last_fname = fname
fname = match.group(1)
if line == '+':
self.blank_count += 1
else:
if self.blank_count and (line == '-- ' or match):
self.warn.append("Found possible blank line(s) at "
"end of file '%s'" % last_fname)
outfd.write('+\n' * self.blank_count)
outfd.write(line + '\n')
self.blank_count = 0
self.Finalize()
def GetMetaData(start, count):
"""Reads out patch series metadata from the commits
This does a 'git log' on the relevant commits and pulls out the tags we
are interested in.
Args:
start: Commit to start from: 0=HEAD, 1=next one, etc.
count: Number of commits to list
"""
pipe = [['git', 'log', '--reverse', 'HEAD~%d' % start, '-n%d' % count]]
stdout = command.RunPipe(pipe, capture=True)
series = Series()
ps = PatchStream(series, is_log=True)
for line in stdout.splitlines():
ps.ProcessLine(line)
ps.Finalize()
return series
def FixPatch(backup_dir, fname, series, commit):
"""Fix up a patch file, by adding/removing as required.
We remove our tags from the patch file, insert changes lists, etc.
The patch file is processed in place, and overwritten.
A backup file is put into backup_dir (if not None).
Args:
fname: Filename to patch file to process
series: Series information about this patch set
commit: Commit object for this patch file
Return:
A list of errors, or [] if all ok.
"""
handle, tmpname = tempfile.mkstemp()
outfd = os.fdopen(handle, 'w')
infd = open(fname, 'r')
ps = PatchStream(series)
ps.commit = commit
ps.ProcessStream(infd, outfd)
infd.close()
outfd.close()
# Create a backup file if required
if backup_dir:
shutil.copy(fname, os.path.join(backup_dir, os.path.basename(fname)))
shutil.move(tmpname, fname)
return ps.warn
def FixPatches(series, fnames):
"""Fix up a list of patches identified by filenames
The patch files are processed in place, and overwritten.
Args:
series: The series object
fnames: List of patch files to process
"""
# Current workflow creates patches, so we shouldn't need a backup
backup_dir = None #tempfile.mkdtemp('clean-patch')
count = 0
for fname in fnames:
commit = series.commits[count]
commit.patch = fname
result = FixPatch(backup_dir, fname, series, commit)
if result:
print '%d warnings for %s:' % (len(result), fname)
for warn in result:
print '\t', warn
print
count += 1
print 'Cleaned %d patches' % count
return series
def InsertCoverLetter(fname, series, count):
"""Inserts a cover letter with the required info into patch 0
Args:
fname: Input / output filename of the cover letter file
series: Series object
count: Number of patches in the series
"""
fd = open(fname, 'r')
lines = fd.readlines()
fd.close()
fd = open(fname, 'w')
text = series.cover
prefix = series.GetPatchPrefix()
for line in lines:
if line.startswith('Subject:'):
# TODO: if more than 10 patches this should save 00/xx, not 0/xx
line = 'Subject: [%s 0/%d] %s\n' % (prefix, count, text[0])
# Insert our cover letter
elif line.startswith('*** BLURB HERE ***'):
# First the blurb test
line = '\n'.join(text[1:]) + '\n'
if series.get('notes'):
line += '\n'.join(series.notes) + '\n'
# Now the change list
out = series.MakeChangeLog(None)
line += '\n' + '\n'.join(out)
fd.write(line)
fd.close()

1
tools/patman/patman Symbolic link
View File

@ -0,0 +1 @@
patman.py

153
tools/patman/patman.py Executable file
View File

@ -0,0 +1,153 @@
#!/usr/bin/python
#
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
"""See README for more information"""
from optparse import OptionParser
import os
import re
import sys
import unittest
# Our modules
import checkpatch
import command
import gitutil
import patchstream
import terminal
import test
parser = OptionParser()
parser.add_option('-H', '--full-help', action='store_true', dest='full_help',
default=False, help='Display the README file')
parser.add_option('-c', '--count', dest='count', type='int',
default=-1, help='Automatically create patches from top n commits')
parser.add_option('-i', '--ignore-errors', action='store_true',
dest='ignore_errors', default=False,
help='Send patches email even if patch errors are found')
parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
default=False, help="Do a try run (create but don't email patches)")
parser.add_option('-s', '--start', dest='start', type='int',
default=0, help='Commit to start creating patches from (0 = HEAD)')
parser.add_option('-t', '--test', action='store_true', dest='test',
default=False, help='run tests')
parser.add_option('-v', '--verbose', action='store_true', dest='verbose',
default=False, help='Verbose output of errors and warnings')
parser.add_option('--cc-cmd', dest='cc_cmd', type='string', action='store',
default=None, help='Output cc list for patch file (used by git)')
parser.add_option('--no-tags', action='store_false', dest='process_tags',
default=True, help="Don't process subject tags as aliaes")
parser.usage = """patman [options]
Create patches from commits in a branch, check them and email them as
specified by tags you place in the commits. Use -n to """
(options, args) = parser.parse_args()
# Run our meagre tests
if options.test:
import doctest
sys.argv = [sys.argv[0]]
suite = unittest.TestLoader().loadTestsFromTestCase(test.TestPatch)
result = unittest.TestResult()
suite.run(result)
suite = doctest.DocTestSuite('gitutil')
suite.run(result)
# TODO: Surely we can just 'print' result?
print result
for test, err in result.errors:
print err
for test, err in result.failures:
print err
# Called from git with a patch filename as argument
# Printout a list of additional CC recipients for this patch
elif options.cc_cmd:
fd = open(options.cc_cmd, 'r')
re_line = re.compile('(\S*) (.*)')
for line in fd.readlines():
match = re_line.match(line)
if match and match.group(1) == args[0]:
for cc in match.group(2).split(', '):
cc = cc.strip()
if cc:
print cc
fd.close()
elif options.full_help:
pager = os.getenv('PAGER')
if not pager:
pager = 'more'
fname = os.path.join(os.path.dirname(sys.argv[0]), 'README')
command.Run(pager, fname)
# Process commits, produce patches files, check them, email them
else:
gitutil.Setup()
if options.count == -1:
# Work out how many patches to send if we can
options.count = gitutil.CountCommitsToBranch() - options.start
col = terminal.Color()
if not options.count:
str = 'No commits found to process - please use -c flag'
print col.Color(col.RED, str)
sys.exit(1)
# Read the metadata from the commits
if options.count:
series = patchstream.GetMetaData(options.start, options.count)
cover_fname, args = gitutil.CreatePatches(options.start, options.count,
series)
# Fix up the patch files to our liking, and insert the cover letter
series = patchstream.FixPatches(series, args)
if series and cover_fname and series.get('cover'):
patchstream.InsertCoverLetter(cover_fname, series, options.count)
# Do a few checks on the series
series.DoChecks()
# Check the patches, and run them through 'git am' just to be sure
ok = checkpatch.CheckPatches(options.verbose, args)
if not gitutil.ApplyPatches(options.verbose, args,
options.count + options.start):
ok = False
# Email the patches out (giving the user time to check / cancel)
cmd = ''
if ok or options.ignore_errors:
cc_file = series.MakeCcFile(options.process_tags)
cmd = gitutil.EmailPatches(series, cover_fname, args,
options.dry_run, cc_file)
os.remove(cc_file)
# For a dry run, just show our actions as a sanity check
if options.dry_run:
series.ShowActions(args, cmd, options.process_tags)

238
tools/patman/series.py Normal file
View File

@ -0,0 +1,238 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import os
import gitutil
import terminal
# Series-xxx tags that we understand
valid_series = ['to', 'cc', 'version', 'changes', 'prefix', 'notes'];
class Series(dict):
"""Holds information about a patch series, including all tags.
Vars:
cc: List of aliases/emails to Cc all patches to
commits: List of Commit objects, one for each patch
cover: List of lines in the cover letter
notes: List of lines in the notes
changes: (dict) List of changes for each version, The key is
the integer version number
"""
def __init__(self):
self.cc = []
self.to = []
self.commits = []
self.cover = None
self.notes = []
self.changes = {}
# These make us more like a dictionary
def __setattr__(self, name, value):
self[name] = value
def __getattr__(self, name):
return self[name]
def AddTag(self, commit, line, name, value):
"""Add a new Series-xxx tag along with its value.
Args:
line: Source line containing tag (useful for debug/error messages)
name: Tag name (part after 'Series-')
value: Tag value (part after 'Series-xxx: ')
"""
# If we already have it, then add to our list
if name in self:
values = value.split(',')
values = [str.strip() for str in values]
if type(self[name]) != type([]):
raise ValueError("In %s: line '%s': Cannot add another value "
"'%s' to series '%s'" %
(commit.hash, line, values, self[name]))
self[name] += values
# Otherwise just set the value
elif name in valid_series:
self[name] = value
else:
raise ValueError("In %s: line '%s': Unknown 'Series-%s': valid "
"options are %s" % (self.commit.hash, line, name,
', '.join(valid_series)))
def AddCommit(self, commit):
"""Add a commit into our list of commits
We create a list of tags in the commit subject also.
Args:
commit: Commit object to add
"""
commit.CheckTags()
self.commits.append(commit)
def ShowActions(self, args, cmd, process_tags):
"""Show what actions we will/would perform
Args:
args: List of patch files we created
cmd: The git command we would have run
process_tags: Process tags as if they were aliases
"""
col = terminal.Color()
print 'Dry run, so not doing much. But I would do this:'
print
print 'Send a total of %d patch%s with %scover letter.' % (
len(args), '' if len(args) == 1 else 'es',
self.get('cover') and 'a ' or 'no ')
# TODO: Colour the patches according to whether they passed checks
for upto in range(len(args)):
commit = self.commits[upto]
print col.Color(col.GREEN, ' %s' % args[upto])
cc_list = []
if process_tags:
cc_list += gitutil.BuildEmailList(commit.tags)
cc_list += gitutil.BuildEmailList(commit.cc_list)
for email in cc_list:
if email == None:
email = col.Color(col.YELLOW, "<alias '%s' not found>"
% tag)
if email:
print ' Cc: ',email
print
for item in gitutil.BuildEmailList(self.get('to', '<none>')):
print 'To:\t ', item
for item in gitutil.BuildEmailList(self.cc):
print 'Cc:\t ', item
print 'Version: ', self.get('version')
print 'Prefix:\t ', self.get('prefix')
if self.cover:
print 'Cover: %d lines' % len(self.cover)
if cmd:
print 'Git command: %s' % cmd
def MakeChangeLog(self, commit):
"""Create a list of changes for each version.
Return:
The change log as a list of strings, one per line
Changes in v1:
- Fix the widget
- Jog the dial
Changes in v2:
- Jog the dial back closer to the widget
etc.
"""
final = []
need_blank = False
for change in sorted(self.changes):
out = []
for this_commit, text in self.changes[change]:
if commit and this_commit != commit:
continue
if text not in out:
out.append(text)
if out:
out = ['Changes in v%d:' % change] + sorted(out)
if need_blank:
out = [''] + out
final += out
need_blank = True
if self.changes:
final.append('')
return final
def DoChecks(self):
"""Check that each version has a change log
Print an error if something is wrong.
"""
col = terminal.Color()
if self.get('version'):
changes_copy = dict(self.changes)
for version in range(2, int(self.version) + 1):
if self.changes.get(version):
del changes_copy[version]
else:
str = 'Change log missing for v%d' % version
print col.Color(col.RED, str)
for version in changes_copy:
str = 'Change log for unknown version v%d' % version
print col.Color(col.RED, str)
elif self.changes:
str = 'Change log exists, but no version is set'
print col.Color(col.RED, str)
def MakeCcFile(self, process_tags):
"""Make a cc file for us to use for per-commit Cc automation
Args:
process_tags: Process tags as if they were aliases
Return:
Filename of temp file created
"""
# Look for commit tags (of the form 'xxx:' at the start of the subject)
fname = '/tmp/patman.%d' % os.getpid()
fd = open(fname, 'w')
for commit in self.commits:
list = []
if process_tags:
list += gitutil.BuildEmailList(commit.tags)
list += gitutil.BuildEmailList(commit.cc_list)
print >>fd, commit.patch, ', '.join(list)
fd.close()
return fname
def AddChange(self, version, commit, info):
"""Add a new change line to a version.
This will later appear in the change log.
Args:
version: version number to add change list to
info: change line for this version
"""
if not self.changes.get(version):
self.changes[version] = []
self.changes[version].append([commit, info])
def GetPatchPrefix(self):
"""Get the patch version string
Return:
Patch string, like 'RFC PATCH v5' or just 'PATCH'
"""
version = ''
if self.get('version'):
version = ' v%s' % self['version']
# Get patch name prefix
prefix = ''
if self.get('prefix'):
prefix = '%s ' % self['prefix']
return '%sPATCH%s' % (prefix, version)

81
tools/patman/settings.py Normal file
View File

@ -0,0 +1,81 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import ConfigParser
import os
import re
import command
def ReadGitAliases(fname):
"""Read a git alias file. This is in the form used by git:
alias uboot u-boot@lists.denx.de
alias wd Wolfgang Denk <wd@denx.de>
Args:
fname: Filename to read
"""
try:
fd = open(fname, 'r')
except IOError:
print "Warning: Cannot find alias file '%s'" % fname
return
re_line = re.compile('alias\s+(\S+)\s+(.*)')
for line in fd.readlines():
line = line.strip()
if not line or line[0] == '#':
continue
m = re_line.match(line)
if not m:
print "Warning: Alias file line '%s' not understood" % line
continue
list = alias.get(m.group(1), [])
for item in m.group(2).split(','):
item = item.strip()
if item:
list.append(item)
alias[m.group(1)] = list
fd.close()
def Setup(config_fname=''):
"""Set up the settings module by reading config files.
Args:
config_fname: Config filename to read ('' for default)
"""
settings = ConfigParser.SafeConfigParser()
if config_fname == '':
config_fname = '%s/.config/patman' % os.getenv('HOME')
if config_fname:
settings.read(config_fname)
for name, value in settings.items('alias'):
alias[name] = value.split(',')
# These are the aliases we understand, indexed by alias. Each member is a list.
alias = {}

86
tools/patman/terminal.py Normal file
View File

@ -0,0 +1,86 @@
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
"""Terminal utilities
This module handles terminal interaction including ANSI color codes.
"""
class Color(object):
"""Conditionally wraps text in ANSI color escape sequences."""
BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE = range(8)
BOLD = -1
COLOR_START = '\033[1;%dm'
BOLD_START = '\033[1m'
RESET = '\033[0m'
def __init__(self, enabled=True):
"""Create a new Color object, optionally disabling color output.
Args:
enabled: True if color output should be enabled. If False then this
class will not add color codes at all.
"""
self._enabled = enabled
def Start(self, color):
"""Returns a start color code.
Args:
color: Color to use, .e.g BLACK, RED, etc.
Returns:
If color is enabled, returns an ANSI sequence to start the given color,
otherwise returns empty string
"""
if self._enabled:
return self.COLOR_START % (color + 30)
return ''
def Stop(self):
"""Retruns a stop color code.
Returns:
If color is enabled, returns an ANSI color reset sequence, otherwise
returns empty string
"""
if self._enabled:
return self.RESET
return ''
def Color(self, color, text):
"""Returns text with conditionally added color escape sequences.
Keyword arguments:
color: Text color -- one of the color constants defined in this class.
text: The text to color.
Returns:
If self._enabled is False, returns the original text. If it's True,
returns text with color escape sequences based on the value of color.
"""
if not self._enabled:
return text
if color == self.BOLD:
start = self.BOLD_START
else:
start = self.COLOR_START % (color + 30)
return start + text + self.RESET

250
tools/patman/test.py Normal file
View File

@ -0,0 +1,250 @@
#
# Copyright (c) 2011 The Chromium OS Authors.
#
# See file CREDITS for list of people who contributed to this
# project.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of
# the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
# MA 02111-1307 USA
#
import os
import tempfile
import unittest
import checkpatch
import gitutil
import patchstream
import series
class TestPatch(unittest.TestCase):
"""Test this program
TODO: Write tests for the rest of the functionality
"""
def testBasic(self):
"""Test basic filter operation"""
data='''
From 656c9a8c31fa65859d924cd21da920d6ba537fad Mon Sep 17 00:00:00 2001
From: Simon Glass <sjg@chromium.org>
Date: Thu, 28 Apr 2011 09:58:51 -0700
Subject: [PATCH (resend) 3/7] Tegra2: Add more clock support
This adds functions to enable/disable clocks and reset to on-chip peripherals.
BUG=chromium-os:13875
TEST=build U-Boot for Seaboard, boot
Change-Id: I80fe1d0c0b7dd10aa58ce5bb1d9290b6664d5413
Review URL: http://codereview.chromium.org/6900006
Signed-off-by: Simon Glass <sjg@chromium.org>
---
arch/arm/cpu/armv7/tegra2/Makefile | 2 +-
arch/arm/cpu/armv7/tegra2/ap20.c | 57 ++----
arch/arm/cpu/armv7/tegra2/clock.c | 163 +++++++++++++++++
'''
expected='''
From 656c9a8c31fa65859d924cd21da920d6ba537fad Mon Sep 17 00:00:00 2001
From: Simon Glass <sjg@chromium.org>
Date: Thu, 28 Apr 2011 09:58:51 -0700
Subject: [PATCH (resend) 3/7] Tegra2: Add more clock support
This adds functions to enable/disable clocks and reset to on-chip peripherals.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
arch/arm/cpu/armv7/tegra2/Makefile | 2 +-
arch/arm/cpu/armv7/tegra2/ap20.c | 57 ++----
arch/arm/cpu/armv7/tegra2/clock.c | 163 +++++++++++++++++
'''
out = ''
inhandle, inname = tempfile.mkstemp()
infd = os.fdopen(inhandle, 'w')
infd.write(data)
infd.close()
exphandle, expname = tempfile.mkstemp()
expfd = os.fdopen(exphandle, 'w')
expfd.write(expected)
expfd.close()
patchstream.FixPatch(None, inname, series.Series(), None)
rc = os.system('diff -u %s %s' % (inname, expname))
self.assertEqual(rc, 0)
os.remove(inname)
os.remove(expname)
def GetData(self, data_type):
data='''
From 4924887af52713cabea78420eff03badea8f0035 Mon Sep 17 00:00:00 2001
From: Simon Glass <sjg@chromium.org>
Date: Thu, 7 Apr 2011 10:14:41 -0700
Subject: [PATCH 1/4] Add microsecond boot time measurement
This defines the basics of a new boot time measurement feature. This allows
logging of very accurate time measurements as the boot proceeds, by using
an available microsecond counter.
%s
---
README | 11 ++++++++
common/bootstage.c | 50 ++++++++++++++++++++++++++++++++++++
include/bootstage.h | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++
include/common.h | 8 ++++++
5 files changed, 141 insertions(+), 0 deletions(-)
create mode 100644 common/bootstage.c
create mode 100644 include/bootstage.h
diff --git a/README b/README
index 6f3748d..f9e4e65 100644
--- a/README
+++ b/README
@@ -2026,6 +2026,17 @@ The following options need to be configured:
example, some LED's) on your board. At the moment,
the following checkpoints are implemented:
+- Time boot progress
+ CONFIG_BOOTSTAGE
+
+ Define this option to enable microsecond boot stage timing
+ on supported platforms. For this to work your platform
+ needs to define a function timer_get_us() which returns the
+ number of microseconds since reset. This would normally
+ be done in your SOC or board timer.c file.
+
+ You can add calls to bootstage_mark() to set time markers.
+
- Standalone program support:
CONFIG_STANDALONE_LOAD_ADDR
diff --git a/common/bootstage.c b/common/bootstage.c
new file mode 100644
index 0000000..2234c87
--- /dev/null
+++ b/common/bootstage.c
@@ -0,0 +1,50 @@
+/*
+ * Copyright (c) 2011, Google Inc. All rights reserved.
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+
+/*
+ * This module records the progress of boot and arbitrary commands, and
+ * permits accurate timestamping of each. The records can optionally be
+ * passed to kernel in the ATAGs
+ */
+
+#include <common.h>
+
+
+struct bootstage_record {
+ uint32_t time_us;
+ const char *name;
+};
+
+static struct bootstage_record record[BOOTSTAGE_COUNT];
+
+uint32_t bootstage_mark(enum bootstage_id id, const char *name)
+{
+ struct bootstage_record *rec = &record[id];
+
+ /* Only record the first event for each */
+%sif (!rec->name) {
+ rec->time_us = (uint32_t)timer_get_us();
+ rec->name = name;
+ }
+%sreturn rec->time_us;
+}
--
1.7.3.1
'''
signoff = 'Signed-off-by: Simon Glass <sjg@chromium.org>\n'
tab = ' '
if data_type == 'good':
pass
elif data_type == 'no-signoff':
signoff = ''
elif data_type == 'spaces':
tab = ' '
else:
print 'not implemented'
return data % (signoff, tab, tab)
def SetupData(self, data_type):
inhandle, inname = tempfile.mkstemp()
infd = os.fdopen(inhandle, 'w')
data = self.GetData(data_type)
infd.write(data)
infd.close()
return inname
def testCheckpatch(self):
"""Test checkpatch operation"""
inf = self.SetupData('good')
result, problems, err, warn, lines, stdout = checkpatch.CheckPatch(inf)
self.assertEqual(result, True)
self.assertEqual(problems, [])
self.assertEqual(err, 0)
self.assertEqual(warn, 0)
self.assertEqual(lines, 67)
os.remove(inf)
inf = self.SetupData('no-signoff')
result, problems, err, warn, lines, stdout = checkpatch.CheckPatch(inf)
self.assertEqual(result, False)
self.assertEqual(len(problems), 1)
self.assertEqual(err, 1)
self.assertEqual(warn, 0)
self.assertEqual(lines, 67)
os.remove(inf)
inf = self.SetupData('spaces')
result, problems, err, warn, lines, stdout = checkpatch.CheckPatch(inf)
self.assertEqual(result, False)
self.assertEqual(len(problems), 2)
self.assertEqual(err, 0)
self.assertEqual(warn, 2)
self.assertEqual(lines, 67)
os.remove(inf)
if __name__ == "__main__":
unittest.main()
gitutil.RunTests()