This patch optimizes the set target port group(STPG) command. During our
testing, we found that it is not optimal to send stpg command every time
the path group switch happens. This patch uses PREF (preferred target port)
bit with combination of flags passed by multipath user level tool to
optimize this behaviour. If PREF bit is set then it issues a STPG command,
otherwise it will let implicit transfer take place.
By default there is no change in the behaviour. User tool needs to pass the
parameter to make this change take effect. Patch has been tested on NetApp
E series storage.
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
PREF bit indicates preferred target port group for accessing a logical
unit. This bit is used to optimize the STPG command handling.
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Handler expects only one parameter to set the flag ALUA_OPTIMIZE_STPG.
This flag is used to optimize the STPG behaviour. There is no change in
behaviour by default.
For example, to set the flag pass the following parameters from multipath.conf
hardware_handler "2 alua 1"
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch adds a check-condition in scsi_dh_alua handler for a retry.
Sometimes, I have seen attach failing due to this check-condition with
following error messages on NetApp E series storage.
Dec 7 15:31:01 nilgiris kernel: [102979.696673] scsi 3:0:2:9: alua: port group 00 rel port 01
Dec 7 15:31:01 nilgiris kernel: [102979.697082] scsi 3:0:2:9: alua: rtpg failed with 8000002
Dec 7 15:31:01 nilgiris kernel: [102979.697086] scsi 3:0:2:9: alua: rtpg sense code 06/2a/01
Dec 7 15:31:01 nilgiris kernel: [102979.697088] scsi 3:0:2:9: alua: not attached
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
If create_singlethread_workqueue() failes, rdac_init should fail too.
Signed-off-by: Richard Weinberger <richard@nod.at>
Acked-by: "Moger, Babu" <Babu.Moger@netapp.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
All the handlers have now implemented the match function so We don't need to
use scsi_dev_info any more for matching purposes.
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Acked-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch introduces the match function for rdac device handler. Without
this, sometimes handler attach fails during the device_add. Included check for
TPGS bit before proceeding further. The match function was introduced by
commit 6c3633d08a
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Acked-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch introduces the match function for hp_sw device handler. Included
the check for TPGS bit before proceeding further per Hannes comment. The
match function was introduced by commit
6c3633d08a
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Acked-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch introduces the match function for emc device handler. Included
check for TPGS bit before proceeding further. The match function was
introduced by commit 6c3633d08a
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Acked-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
* 'modsplit-Oct31_2011' of git://git.kernel.org/pub/scm/linux/kernel/git/paulg/linux: (230 commits)
Revert "tracing: Include module.h in define_trace.h"
irq: don't put module.h into irq.h for tracking irqgen modules.
bluetooth: macroize two small inlines to avoid module.h
ip_vs.h: fix implicit use of module_get/module_put from module.h
nf_conntrack.h: fix up fallout from implicit moduleparam.h presence
include: replace linux/module.h with "struct module" wherever possible
include: convert various register fcns to macros to avoid include chaining
crypto.h: remove unused crypto_tfm_alg_modname() inline
uwb.h: fix implicit use of asm/page.h for PAGE_SIZE
pm_runtime.h: explicitly requires notifier.h
linux/dmaengine.h: fix implicit use of bitmap.h and asm/page.h
miscdevice.h: fix up implicit use of lists and types
stop_machine.h: fix implicit use of smp.h for smp_processor_id
of: fix implicit use of errno.h in include/linux/of.h
of_platform.h: delete needless include <linux/module.h>
acpi: remove module.h include from platform/aclinux.h
miscdevice.h: delete unnecessary inclusion of module.h
device_cgroup.h: delete needless include <linux/module.h>
net: sch_generic remove redundant use of <linux/module.h>
net: inet_timewait_sock doesnt need <linux/module.h>
...
Fix up trivial conflicts (other header files, and removal of the ab3550 mfd driver) in
- drivers/media/dvb/frontends/dibx000_common.c
- drivers/media/video/{mt9m111.c,ov6650.c}
- drivers/mfd/ab3550-core.c
- include/linux/dmaengine.h
This patch corrects the retry interval for alua rtpg command. Purpose was to retry the commands in seconds.
But that was not happening. Reason is msleep takes argument in milliseconds.
Also added minor text after successful attach.
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Acked-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
The module.h header was implicitly present everywhere, so files
with no explicit include of the module infrastructure would build
anyway. We are now removing the implicit include, and so we need
to call out the module.h file that we need explicitly.
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
This patch adds one more check-condition for scsi_dh_alua handler. Without
this, the handler attach fails sometimes during the discovery. I have noticed
this with NetApp E-Series storage with alua mode. Also removed some
unnecessary brackets {} for consistency.
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
This patch adds couple more Vendor/Product IDs for RDAC.. There are no
functional changes.
Signed-off-by: Babu Moger <babu.moger@netapp.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
The alua device handler starts the first retry after 10 seconds,
and increases it times 10 for each round.
This leads to an unnecessary delay. This patch modifies it to
start after one second, and increase by a factor of two.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
For Target Portal Group IDs occupying the full 2 bytes in the
RTPG response, the following group_id check in the alua_rtpg
routine always fails in scsi_dh_alua.c:
if (h->group_id == (ucp[2] << 8) + ucp[3]) {
This causes the ALUA handler to wrongly identify the AAS of
a specified device as well as incorrectly interpreting the
supported AAS of the target as seen by the following entries
in the /var/log/messages:
"alua: port group 3ea state A supports tousna"
"alua: port group 3e9 state A supports tousna"
This is because 'ucp' is wrongly declared in alua_rtpg as
a character pointer instead of an unsigned character pointer.
Signed-off-by: Martin George <marting@netapp.com>
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Avoid attaching a hardware handler to a device which is
already scheduled for deletion.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
When activating a patch we should always update the TPGS state
as it might have changed in between.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Some device handler types are not tied to the vendor/model
but rather to a specific capability. Eg ALUA is supported
if the 'TPGS' setting in the standard inquiry is set.
This patch implements a 'match' callback for device handler
which supersedes the original vendor/model lookup and
implements the callback for the ALUA handler.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Fixup some kernel-doc comments to reference to the
correct function name.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
Instead of issuing a standard inquiry from within the
alua device handler we can evaluate the TPGS setting from
the existing inquiry data of the sdev and save us the I/O.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
rdac hardware handler assumes that there is one-to-one relation ship
between the host and the controller w.r.t lun. IOW, it does not
support "multiple storage partitions" within a storage.
Example:
HBA1 and HBA2 see lun 0 and 1 in storage A (1)
HBA3 and HBA4 see lun 0 and 1 in storage A (2)
HBA5 and HBA6 see lun 0 and 1 in storage A (3)
luns 0 and 1 in (1), (2) and (3) are totally different.
But, rdac handler treats the lun 0s (and lun 1s) as the same when
sending a mode select to the controller, which is wrong.
This patch makes the rdac hardware handler associate HBA and the
storage w.r.t lun (and not the host itself).
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
rdac hardware handler uses "Subsystem Identifier" from C4 inquiry page
to uniquely identify a storage. The problem with that is that if any
any of the bytes are non-ascii, subsys_id will all be spaces (hex
0x20). This creates lot of problems especially when there are multiple
rdac storages are connected to the server.
Use "Storage Array Unique Identifier" from C8 inquiry page, which is the
world wide unique identifier for the storage array, to uniquely identify
the storage.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
All these are instances of
#define NAME value;
or
#define NAME(params_opt) value;
These of course fail to build when used in contexts like
if(foo $OP NAME)
while(bar $OP NAME)
and may silently generate the wrong code in contexts such as
foo = NAME + 1; /* foo = value; + 1; */
bar = NAME - 1; /* bar = value; - 1; */
baz = NAME & quux; /* baz = value; & quux; */
Reported on comp.lang.c,
Message-ID: <ab0d55fe-25e5-482b-811e-c475aa6065c3@c29g2000yqd.googlegroups.com>
Initial analysis of the dangers provided by Keith Thompson in that thread.
There are many more instances of more complicated macros having unnecessary
trailing semicolons, but this pile seems to be all of the cases of simple
values suffering from the problem. (Thus things that are likely to be found
in one of the contexts above, more complicated ones aren't.)
Signed-off-by: Phil Carmody <ext-phil.2.carmody@nokia.com>
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
* git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6: (110 commits)
[SCSI] qla2xxx: Refactor call to qla2xxx_read_sfp for thermal temperature.
[SCSI] qla2xxx: Unify the read/write sfp mailbox command routines.
[SCSI] qla2xxx: Clear complete initialization control block.
[SCSI] qla2xxx: Allow an override of the registered maximum LUN.
[SCSI] qla2xxx: Add host number in reset and quiescent message logs.
[SCSI] qla2xxx: Correctly read sfp single byte mailbox register.
[SCSI] qla2xxx: Add qla82xx_rom_unlock() function.
[SCSI] qla2xxx: Log if qla82xx firmware fails to load from flash.
[SCSI] qla2xxx: Use passed in host to initialize local scsi_qla_host in queuecommand function
[SCSI] qla2xxx: Correct buffer start in edc sysfs debug print.
[SCSI] qla2xxx: Update firmware version after flash update for ISP82xx.
[SCSI] qla2xxx: Fix hang during driver unload when vport is active.
[SCSI] qla2xxx: Properly set the dsd_list_len for dsd_chaining in cmd type 6.
[SCSI] qla2xxx: Fix virtual port failing to login after chip reset.
[SCSI] qla2xxx: Fix vport delete hang when logins are outstanding.
[SCSI] hpsa: Change memset using sizeof(ptr) to sizeof(*ptr)
[SCSI] ipr: Rate limit DMA mapping errors
[SCSI] hpsa: add P2000 to list of shared SAS devices
[SCSI] hpsa: do not attempt PCI power management reset method if we know it won't work.
[SCSI] hpsa: remove superfluous sleeps around reset code
...
The SCSI ALUA handler currently fails to attach to devices
reporting an UNAVAILABLE/OFFLINE AAS. But given that an
UNAVAILABLE/OFFLINE AAS can transition to other states
like ACTIVE/OPTIMIZED, ACTIVE/NON-OPTIMIZED, etc. as per
SPC4, this ALUA handler behavior should be rectified so
as to attach to devices which also report an
UNAVAILABLE/OFFLINE AAS.
Signed-off-by: Martin George <marting@netapp.com>
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Based on the operating modes, handler decides whether to send mode
select or not. Purpose here is to reduce io-shipping as much as
possible whenever there is an option.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Reviewed-by: Yanling Qi <yanling.qi@lsi.com>
Reviewed-by: Sudhir Dachepalli <Sudhir.Dachepalli@lis.com>
Reviewed-by: Somasundaram Krishnasamy <Somasundaram.Krishnasamy@lsi.com>
Reviewed-by: Bob Stankey <Robert.Stankey@lsi.com>
Reviewed-by: Vijay Chauhan <Vijay.Chauhan@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch detects different operating RDAC modes during the
discovery. It also collects the information about the preferred path.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Reviewed-by: Yanling Qi <yanling.qi@lsi.com>
Reviewed-by: Sudhir Dachepalli <Sudhir.Dachepalli@lis.com>
Reviewed-by: Somasundaram Krishnasamy <Somasundaram.Krishnasamy@lsi.com>
Reviewed-by: Bob Stankey <Robert.Stankey@lsi.com>
Reviewed-by: Vijay Chauhan <Vijay.Chauhan@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch adds definitions to support for different operating modes
for LSI rdac storage. Currently, rdac support 3 operation modes.
1. RDAC mode(legacy)
2. AVT mode
3. IOSHIP mode
These definitions are used while activating the path(rdac_activate).
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Reviewed-by: Yanling Qi <yanling.qi@lsi.com>
Reviewed-by: Sudhir Dachepalli <Sudhir.Dachepalli@lis.com>
Reviewed-by: Somasundaram Krishnasamy <Somasundaram.Krishnasamy@lsi.com>
Reviewed-by: Bob Stankey <Robert.Stankey@lsi.com>
Reviewed-by: Vijay Chauhan <Vijay.Chauhan@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Commit db422318cb ([SCSI] scsi_dh:
propagate SCSI device deletion) introduced a regression where the device
reference is not dropped prior to scsi_dh_activate's early return from
the error path.
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Cc: stable@kernel.org # 2.6.38
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch is to add Dell MD36xxf array into the RDAC handler device list.
Singed-off-by: Yanqing Liu <Yanqing_Liu@Dell.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adding MODULE_VERSION for scsi_dh_rdac. This will be helpful sometimes
to get the code level without looking at the code.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
instead of doing sizeof(struct X) it's better to do sizeof(*v) where v
is the variable pointing to struct X.
Signed-off-by: Hillf Danton <dhillf@gmail.com>
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
During one of our testing, we noticed that mode select command sent
from the host did not have the lun_table updated.
Problem is root caused to the way lun table is updated. Lun table
update was done after the call to blk_rq_map_kern is made. This was
causing problem because kernel uses bounce buffer(bio_copy_kern) if
the address is not aligned. The command buffer updated after the
call(blk_rq_map_kern) was not going on the wire. Moved the code to
update the lun_table before the call to fix the problem.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Signed-off-by: Somasundaram Krishnasamy <Somasundaram.Krishnasamy@lsi.com>
Signed-off-by: Yanling Qi <Yanling.Qi@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The use of blk_execute_rq_nowait() implies __blk_put_request() is needed
in start_stop_endio() rather than blk_put_request() --
blk_finish_request() is called with queue lock already held.
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Previously we were using strncmp in order to avoid having to include
whitespace in the devlist, but this means "HSV1000" matches a device
list entry that says "HSV100", which is wrong. This patch changes
scsi_dh.c to use scsi_devinfo's matching functions instead, since they
handle these cases correctly.
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Currently NetApp's VID/PID details in the INQUIRY response shows up as
'NETAPP' and 'LUN'. With upcoming scalable SAN ONTAP version on NetApp
controllers, the PID entry alone is being modified to 'LUN C-Mode' (to
distinguish current ONTAP LUNs from scalable ONTAP LUNs).
'LUN' would still suffice for matching 'LUN C-Mode' but best to
explicitly add these new NetApp LUNs to the device list.
Reported-by: Martin George <marting@netapp.com>
Acked-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adds Promise VTrak devices to the ALUA device handler.
Signed-off-by: Ilgu Hong <ilgu.hong@promise.com>
Signed-off-by: Joseph Gruher <joseph.r.gruher@intel.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Initialize stpg_endio() 'err' to SCSI_DH_OK and only change it to
SCSI_DH_IO accordingly. This allows the switching of target group state
to be properly reported when no error has occurred.
Signed-off-by: Joseph Gruher <joseph.r.gruher@intel.com>
Signed-off-by: Ilgu Hong <ilgu.hong@promise.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The use of blk_execute_rq_nowait() implies __blk_put_request() is needed
in stpg_endio() rather than blk_put_request() -- blk_finish_request() is
called with queue lock already held.
Signed-off-by: Joseph Gruher <joseph.r.gruher@intel.com>
Signed-off-by: Ilgu Hong <ilgu.hong@promise.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Cc: stable@kernel.org
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
submit_stpg() will always return failure so alua_activate() will report
failure via dm-multipath callback function. Even though the stpg fired
successfuly dm-multipath does not know and always fails to change the
valid path.
By returning SCSI_DH_OK we're now skipping alua_activate()'s call to
activate_complete 'fn'. But this is fine because stpg_endio() will call
it via h->callback_fn().
Signed-off-by: Joseph Gruher <joseph.r.gruher@intel.com>
Signed-off-by: Ilgu Hong <ilgu.hong@promise.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Currently, when scsi_dh_activate() returns with an error
(e.g. SCSI_DH_NOSYS) the activate_complete callback is not called and
the error is not propagated to DM mpath.
When a SCSI device attached to a device handler is deleted, userland
processes currently performing I/O on the device will have their I/O
hang forever.
- Set SCSI_DH_NOSYS error when the handler is in the process of being
deleted (e.g. the SCSI device is in a SDEV_CANCEL or SDEV_DEL state).
- Set SCSI_DH_DEV_OFFLINED error when device is in SDEV_OFFLINE state.
- Call the activate_complete callback function directly from
scsi_dh_activate if an error has been set (when either the scsi_dh
internal data has already been deleted or is in the process of being
deleted).
The patch was tested in an iSCSI environment, RDAC H/W handler and
multipath. In the following reproduction process, dd will I/O hang
forever and the only way to release it will be to reboot the machine:
1) Perform I/O on a multipath device:
dd if=/dev/dm-0 of=/dev/zero bs=8k count=1000000 &
2) Delete all slave SCSI devices contained in the mpath device:
I) In an iSCSI environment, the easiest way to do this is by
stopping iSCSI:
/etc/init.d/iscsi stop
II) Another way to delete the devices is by applying the following
bash scriptlet:
dm_devs=$(ls /sys/block/ | grep dm- | xargs)
for dm_dev in $dm_devs; do
devices=$(ls /sys/block/$dm_dev/slaves)
for device in $devices; do
echo 1 > /sys/block/$device/device/delete
done
done
NOTE: when DM mpath's fail_path uses blk_abort_queue this scsi_dh change
isn't strictly required. However, DM mpath's call to blk_abort_queue
will soon be reverted because it has proven to be unsafe due to a race
(between blk_abort_queue and scsi_request_fn) that can lead to list
corruption. Therefore we cannot rely on blk_abort_queue via fail_path,
but even if we could this scsi_dh change is still preferrable.
Signed-off-by: Menny Hamburger <Menny_Hamburger@Dell.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Reviewed-by: Babu Moger <babu.moger@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
For ALUA we should be handling all states, independent of whether
the mode is explicit or implicit. For 'Transitioning' we should retry
for a certain amount of time; after that we're setting the port
to 'Standby' and return SCSI_DH_RETRY to signal upper layers
a retry is in order here.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Acked-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch is to add next generation of Dell iSCSI PowerVault
controller MD36xxi into RDAC device list.
Signed-off-by: Yanqing Liu <Yanqing_Liu@Dell.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
blk_get_request sets the cmd_flags, so we should not and do not
need to set them. If we did set them to a different value then
it can cause a oops in the elevator code.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
This patch fixes the request setup code for mode selects. I got the fixes from
Hannes Reinecke while trying to hunt down some problems and merged it
into one patch. I am sending it because Hannes is busy with other things.
The patch fixes:
- setting of the length for mode selects.
- setting of the data direction for mode select 10.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adds IBM Power Virtual SCSI ALUA devices to the ALUA device handler.
Signed-off-by: Brian King <brking@linux.vnet.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Newer Netapp target software supports ALUA, so
this patch adds them to the scsi_dev_alua dev list.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch adds two new IBM storage devices which can use rdac device handlers.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Create the sysfs file, dh_state even if the new SCSI device is not
in the any of the device handler's internal lists.
Signed-Off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Acked-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Make the activate function asynchronous by using blk_execute_rq_nowait()
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Make the activate function asynchronous by using blk_execute_rq_nowait()
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Batch up MODE_SELECT in rdac device handler.
LSI RDAC storage has the capability of handling mode selects for
multiple luns in a same command. Make use of that ability to send
as few MODE SELECTs as possible to the storage controller as possible.
This patch creates a work queue and queues up activate requests
when a MODE SELECT is sent down the wire. When that MODE SELECT
completes, it compiles queued up activate requests for multiple
luns into a single MODE SELECT.
This reduces the time to do failover/failback of large number of LUNS.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Make scsi_dh_activate() function asynchronous, by taking in two additional
parameters, one is the callback function and the other is the data to call
the callback function with.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The function mode_select_handle_sense returns SCSI_DH_OK even when there is a sense code which is incorrect. Removing it so that it returns SCSI_DH_IO when there is sense that is not handled by this function.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Reviewed-by: Vijay Chauhan <vijay.chauhan@lsi.com>
Reviewed-by: Bob Stankey <Robert.stankey@lsi.com>
Reviewed-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Patch to add debugging stuff for rdac device handler.
- Added a bit mask "module parameter" rdac_logging with 2 bits for each type
of logging.
- currently defined only two types of logging(failover and sense logging). Can
be enhanced later if required.
- By default only failover logging is enabled which is equivalent of current
logging.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Reviewed-by: Vijay Chauhan <vijay.chauhan@lsi.com>
Reviewed-by: Bob Stankey <Robert.stankey@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adding the code to read the debug information during initialization. This
patch collects the information about storage and controllers during
rdac_activate.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Reviewed-by: Vijay Chauhan <vijay.chauhan@lsi.com>
Reviewed-by: Bob Stankey <Robert.stankey@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Moving the initialization code from rdac_activate to rdac_bus_attach which is
more efficient. We don't have to collect all the information during every
activate.
Signed-off-by: Babu Moger <babu.moger@lsi.com>
Reviewed-by: Vijay Chauhan <vijay.chauhan@lsi.com>
Reviewed-by: Bob Stankey <Robert.stankey@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch is to add DM support for next generation of Dell PowerVault
storage array.
Signed-off-by: Yanqing Liu <Yanqing_Liu@Dell.com>
Acked-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Handle the parameters provided by user thru multipath.
This handler expects only 2 parameters and their value can either be 0 or 1.
This code originates from the old dm-emc.c file. Appropriate changes have
been made to make it work in the new design.
Reported-by: Eddie Williams <Eddie.Williams@steeleye.com>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Tested-by: Eddie Williams <Eddie.Williams@steeleye.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
When we moved the device handler functionality from dm layer to SCSI layer
we dropped the parameter functionality.
This path adds an interface to scsi dh layer to set device handler
parameters.
Basically, multipath layer need to create a string with all the parameters
and call scsi_dh_set_params() after it called scsi_dh_attach() on a
device.
If a device handler provides such an interface it will handle the parameters
as it expects them.
Reported-by: Eddie Williams <Eddie.Williams@steeleye.com>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Tested-by: Eddie Williams <Eddie.Williams@steeleye.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
alua_activate only sends a STPG if only explicit is suppored.
As a result, for EMC targets that support both we end up doing
a implicit failover when X commands are finally sent to
the other SP.
This patch does a AND on the h->tpgs, so we do a explicit failover
right away.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Problem reported: http://marc.info/?l=dm-devel&m=124585978305866&w=2
scsi_dh does not do a refernce count for attach/detach, and this affects
the way it is supposed to work with multipath when a device is not
in the dev_list of the hardware handler.
This patch adds a reference count that counts each attach.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
During device discovery read capacity fails with 0x020401 and sets the
device size to 0. As a reason any I/O submitted to this path gets
killed at sd_prep_fn with BLKPREP_KILL. This patch is to retry for
0x020401. NEED_RETRY in scsi_decide_disposition does not give
sufficient time for the device to become ready.
Signed-off-by: Vijay Chauhan <vijay.chauhan@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch adds retry for NOT_READY check condition - Quiesce in
progress (02/A1/02)
Signed-off-by: Vijay Chauhan<vijay.chauhan@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch is to add retry for mode select if mode select command is
returned with sense NO_SENSE, UNIT_ATTENTION, ABORTED_COMMAND,
NOT_READY(02/04/01). This patch reorganise the sense keys from if-else
to switch-case format for better maintainability.
Signed-off-by: Vijay Chauhan <vijay.chauhan@lsi.com>
Acked-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
During device discovery read capacity fails with 0x068b02 and sets the
device size to 0. As a reason any I/O submitted to this path gets
killed at sd_prep_fn with BLKPREP_KILL. This patch is to retry for
0x068b02
Signed-off-by: Vijay Chauhan <vijay.chauhan@lsi.com>
Acked-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When we switch controllers the Intel Multi-Flex reports
REPORTED_LUNS_DATA_HAS_CHANGED. This patch just has us
retry the command.
Signed-off-by: Ilgu Hong <ilgu.hong@promise.com>
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This adds the Intel Multi-Flex device to scsi_dh_alua's
scsi_dh_devlist, so the module attaches to these devs.
Signed-off-by: Ilgu Hong <ilgu.hong@promise.com>
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The buf[i] is a byte but we are only asking 4 bits off the
group_id. This patch has us take off a byte.
Signed-off-by: Ilgu Hong <ilgu.hong@promise.com>
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When the mode select sent to the controller fails with the retryable
error, it is better to retry the mode_select from the hardware handler
itself, instead of propagating the failure to dm-multipath.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When the controller ownership is changed (from passive to active),
check_ownership() doesn't set the state of the device to ACTIVE.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Reported-by: "Moger, Babu" <Babu.Moger@lsi.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The test-unit-ready portion of this patch was causing boots to fail on
my test machine (as in http://lkml.org/lkml/2008/12/5/161). With this
patch in place, the system is booting reliably.
Mike Anderson found the same problem in the hp_hw_start_stop code,
and I applied the same solution in cdrom_read_cdda_bpc.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Cc: Mike Anderson <andmike@linux.vnet.ibm.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
Set the path state to be passive when we learn that the controller does
not own the path to the LUN.
This will avoid sending even a single i/o thru the passive path at the
probe time.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
We do not need to set REQ_NOMERGE because when the module calls
blk_execute_rq -> blk_execute_rq_nowait, blk_execute_rq_nowait sets
it for us. This brings all the modules in sync for those bits.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Multipath is best at handling transport errors. If it gets a device
error then there is not much the multipath layer can do. It will just
access the same device but from a different path.
This patch breaks up failfast into device, transport and driver errors.
The multipath layers (md and dm mutlipath) only ask the lower levels to
fast fail transport errors. The user of failfast, read ahead, will ask
to fast fail on all errors.
Note that blk_noretry_request will return true if any failfast bit
is set. This allows drivers that do not support the multipath failfast
bits to continue to fail on any failfast error like before. Drivers
like scsi that are able to fail fast specific errors can check
for the specific fail fast type. In the next patch I will convert
scsi.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Cc: Jens Axboe <jens.axboe@oracle.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
On Mon, 2008-09-22 at 14:56 -0700, akpm@linux-foundation.org wrote:
> From: Andrew Morton <akpm@linux-foundation.org>
>
> s390:
>
> drivers/scsi/device_handler/scsi_dh_emc.c: In function 'parse_sp_info_reply':
> drivers/scsi/device_handler/scsi_dh_emc.c:179: warning: comparison is always false due to limited range of data type
>
> because chars are unsigned, I assume.
Fix by making csdev->buffer explicitly an unsigned char and dropping
the < 0 test.
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Add Dell Powervault storage arrays into device list of rdac device
handler.
Signed-off-by: Yanqing Liu <yanqing_liu@dell.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
blk_get_request initializes rq->cmd (rq_init does) so the users don't
need to do that.
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Change scsi_dh check_sense functions to return ADD_TO_MLQUEUE
to allow retries to occur without restriction of blk_noretry_request
check.
Signed-off-by: Mike Anderson <andmike@linux.vnet.ibm.com>
Acked-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
* git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6: (22 commits)
[SCSI] ibmvfc: Driver version 1.0.2
[SCSI] ibmvfc: Add details to async event log
[SCSI] ibmvfc: Sanitize response lengths
[SCSI] ibmvfc: Fix for lost async events
[SCSI] ibmvfc: Fixup host state during reinit
[SCSI] ibmvfc: Fix another hang on module removal
[SCSI] ibmvscsi: Fixup desired DMA value for shared memory partitions
[SCSI] megaraid_sas: remove sysfs dbg_lvl world writeable permissions
[SCSI] qla2xxx: Update version number to 8.02.01-k7.
[SCSI] qla2xxx: Explicitly tear-down vports during PCI remove_one().
[SCSI] qla2xxx: Reference proper ha during SBR handling.
[SCSI] qla2xxx: Set npiv_supported flag for FCoE HBAs.
[SCSI] qla2xxx: Don't leak SG-DMA mappings while aborting commands.
[SCSI] qla2xxx: Correct vport-state management issues during ISP-ABORT.
[SCSI] qla2xxx: Correct synchronization of software/firmware fcport states.
[SCSI] scsi_dh: Initialize lun_state in check_ownership()
[SCSI] scsi_dh: Do not use scsilun in rdac hardware handler
[SCSI] megaraid_sas: version and Documentation Update
[SCSI] megaraid_sas: add new controllers (0x78 0x79)
[SCSI] megaraid_sas: add the shutdown DCMD cmd to driver shutdown routine
...
lun_state need to be initialized inside check_ownership().
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
RDAC storage controller doesn't seem to use the scsilun format. It uses
only the last byte for LUN.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch makes several needlessly global struct scsi_dh_devlist's
static.
Signed-off-by: Adrian Bunk <bunk@kernel.org>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[jejb: fixed up a ton of missed conversions.
All of you are on notice this has happened, driver trees will now
need to be rebased]
Signed-off-by: Harvey Harrison <harvey.harrison@gmail.com>
Cc: SCSI List <linux-scsi@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Create a cache of devices that are seen in a system. This will avoid
the unnecessary traversal of the device list in the scsi_dh when there
are multiple luns of a same type.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
multipath keeps a separate device table which may be
more current than the built-in one.
So we should make sure to always call ->attach whenever
a multipath map with hardware handler is instantiated.
And we should call ->detach on removal, too.
[sekharan: update as per comments from agk]
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch updates the RDAC device handler to
refuse to attach to devices not supporting the
RDAC vpd pages.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch updates the hp_sw device handler to properly
check the return codes etc.
And adds the 'correct' machine definitions.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch converts the EMC device handler to use a proper
state machine. We now also parse the extended INQUIRY
information to determine if long trespass commands are
supported. And we're now using the long trespass command
correctly. And finally there's now an check at init time
to refuse to attach to devices not supporting EMC-specific
VPD pages.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Implement a 'dh_state' sdev attribute for dynamic device handler
manipulation. A read on the attribute will return the name of
the currently attached device handler or 'detached' if no handler
is attached.
The attribute allows the following strings to be written:
- The name of the device handler to be attached if the state is
'detached'.
- 'activate' to trigger path activation if a device handler
is attached.
- 'detach' to detach the currently attached device handler.
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Instead of having each and every driver implement its own
device table scanning code we should rather implement a common
routine and scan the device tables there.
This allows us also to implement a general notifier chain
callback for all device handler instead for one per handler.
[sekharan: Fix rejections caused by conflicting bug fix]
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Before accessing the device data structure in hardware handlers,
make sure it is a indeed a sdev device.
Yinghai Lu <yhlu.kernel@gmail.com> found the bug on Jul 16, 2008,
and later tested/verified the following fix.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This adds support for EMC Clariions. This patch has the features that
currently exists in mainline and advanced features from Ed's patches.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch provides the device handler to support the older hp boxes
which cannot be upgraded.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This patch provides the device handler to support the LSI RDAC SCSI
based storage devices.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Some of the storage devices (that can be accessed through multiple paths),
do need some special handling for
1. Activating the passive path of the storage access.
2. Decode and handle the special sense codes returned by the devices.
3. Handle the I/Os being sent to the passive path, especially
during the device probe time.
when accessed through multiple paths.
As of today this special device handling is done at the dm-multipath
layer using dm-handlers. That works well for (1); for (2) to be handled
at dm layer, scsi sense information need to be exported from SCSI to dm-layer,
which is not very attractive; (3) cannot be done at all at the dm layer.
Device handler has been moved to SCSI mainly to handle (2) and (3) properly.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: Mike Anderson <andmike@linux.vnet.ibm.com>
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
