The conn_send_pdu API is evil in that it returns a pointer to an
iscsi_task, but that task might have been freed already so you can't touch
it. This patch splits the task allocation and transmission, so functions
like iscsi_send_nopout() can access the task before its sent and do
whatever bookkeeping is needed before it is sent.
Link: https://lore.kernel.org/r/20220616224557.115234-10-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
We need the back lock when freeing a task, so we hold it when calling
__iscsi_put_task() from the completion path to make it easier and to avoid
having to retake it in that path. For iscsi_put_task() we just grabbed it
while also doing the decrement on the refcount but it's only really needed
if the refcount is zero and we free the task. This modifies
iscsi_put_task() to just take the lock when needed then has the xmit path
use it. Normally we will then not take the back lock from the xmit path. It
will only be rare cases where the network is so fast that we get a response
right after we send the header/data.
Link: https://lore.kernel.org/r/20220616224557.115234-9-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
We currently require that the back_lock is held when calling the functions
that manipulate the iscsi_task refcount. The only reason for this is to
handle races where we are handling SCSI-ml EH callbacks and the cmd is
completing at the same time the normal completion path is running, and we
can't return from the EH callback until the driver has stopped accessing
the cmd. Holding the back_lock while also accessing the task->state made it
simple to check that a cmd is completing and also get/put a refcount at the
same time, and at the time we were not as concerned about performance.
The problem is that we don't want to take the back_lock from the xmit path
for normal I/O since it causes contention with the completion path if the
user has chosen to try and split those paths on different CPUs (in this
case abusing the CPUs and ignoring caching improves perf for some uses).
Begins to remove the back_lock requirement for iscsi_get/put_task by
removing the requirement for the get path. Instead of always holding the
back_lock we detect if something has done the last put and is about to call
iscsi_free_task(). A subsequent commit will then allow iSCSI code to do the
last put on a task and only grab the back_lock if the refcount is now zero
and it's going to call iscsi_free_task().
Link: https://lore.kernel.org/r/20220616224557.115234-8-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Commit 5923d64b7a ("scsi: libiscsi: Drop taskqueuelock") added an extra
task->state because for commit 6f8830f5bb ("scsi: libiscsi: add lock
around task lists to fix list corruption regression") we didn't know why we
ended up with cmds on the list and thought it might have been a bad target
sending a response while we were still sending the cmd. We were never able
to get a target to send us a response early, because it turns out the bug
was just a race in libiscsi/libiscsi_tcp where:
1. iscsi_tcp_r2t_rsp() queues a r2t to tcp_task->r2tqueue.
2. iscsi_tcp_task_xmit() runs iscsi_tcp_get_curr_r2t() and sees we have a
r2t. It dequeues it and iscsi_tcp_task_xmit() starts to process it.
3. iscsi_tcp_r2t_rsp() runs iscsi_requeue_task() and puts the task on the
requeue list.
4. iscsi_tcp_task_xmit() sends the data for r2t. This is the final chunk
of data, so the cmd is done.
5. target sends the response.
6. On a different CPU from #3, iscsi_complete_task() processes the
response. Since there was no common lock for the list, the lists/tasks
pointers are not fully in sync, so could end up with list corruption.
Since it was just a race on our side, remove the extra check and fix up the
comments.
Link: https://lore.kernel.org/r/20220616224557.115234-7-michael.christie@oracle.com
Reviewed-by: Wu Bo <wubo40@huawei.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Add helpers to allow the drivers to run their recv paths from libiscsi's
workqueue.
Link: https://lore.kernel.org/r/20220616224557.115234-3-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Rename iscsi_conn_queue_work() to iscsi_conn_queue_xmit() to reflect that
it handles queueing of xmits only.
Link: https://lore.kernel.org/r/20220616224557.115234-2-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Reviewed-by: Wu Bo <wubo40@huawei.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
When the system is shutting down, iscsid is not running so we will not get
a response to the ISCSI_ERR_INVALID_HOST error event. The system shutdown
will then hang waiting on userspace to remove the session.
This has libiscsi force the destruction of the session from the kernel when
iscsi_host_remove() is called from a driver's shutdown callout.
This fixes a regression added in qedi boot with commit d1f2ce7763 ("scsi:
qedi: Fix host removal with running sessions") which made qedi use the
common session removal function that waits on userspace instead of rolling
its own kernel based removal.
Link: https://lore.kernel.org/r/20220616222738.5722-7-michael.christie@oracle.com
Fixes: d1f2ce7763 ("scsi: qedi: Fix host removal with running sessions")
Tested-by: Nilesh Javali <njavali@marvell.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Reviewed-by: Nilesh Javali <njavali@marvell.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
iscsi_data_xmit() (TX worker) is iterating over the queue of new SCSI
commands concurrently with the queue being replenished. Only after the
queue is emptied will we start sending pending DataOut PDUs. That leads to
DataOut timeout on the target side and to connection reinstatement.
Give priority to pending DataOut commands over new commands.
Link: https://lore.kernel.org/r/20220607131953.11584-1-d.bogdanov@yadro.com
Reviewed-by: Konstantin Shelekhin <k.shelekhin@yadro.com>
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Dmitry Bogdanov <d.bogdanov@yadro.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
If a offload driver doesn't use the xmit workqueue, then when we are doing
ep_disconnect libiscsi can still inject PDUs to the driver. This adds a
check for if the connection is bound before trying to inject PDUs.
Link: https://lore.kernel.org/r/20220408001314.5014-9-michael.christie@oracle.com
Tested-by: Manish Rangankar <mrangankar@marvell.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Reviewed-by: Chris Leech <cleech@redhat.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Move the tx and rx suspend fields into one flags field.
Link: https://lore.kernel.org/r/20220408001314.5014-8-michael.christie@oracle.com
Tested-by: Manish Rangankar <mrangankar@marvell.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Reviewed-by: Chris Leech <cleech@redhat.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
iscsi_cls_conn is alloced by kzalloc(), the whole iscsi_cls_conn is zero
filled already including the dd_data. So it is unnecessary to call memset
again.
Link: https://lore.kernel.org/r/20220317150116.194140-1-haowenchao@huawei.com
Reviewed-by: Wu Bo <wubo40@huawei.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Wenchao Hao <haowenchao@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Commit 1b8d0300a3 ("scsi: libiscsi: Fix UAF in
iscsi_conn_get_param()/iscsi_conn_teardown()") fixed an UAF in
iscsi_conn_get_param() and introduced 2 tmp_xxx varibles.
We can gracefully fix this UAF with the help of device_del(). Calling
iscsi_remove_conn() at the beginning of iscsi_conn_teardown would make
userspace unable to see iscsi_cls_conn. This way we we can free memory
safely.
Remove iscsi_destroy_conn() since it is no longer used.
Link: https://lore.kernel.org/r/20220310015759.3296841-4-haowenchao@huawei.com
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Wenchao Hao <haowenchao@huawei.com>
Signed-off-by: Wu Bo <wubo40@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
iscsi_create_conn() exposed iscsi_cls_conn to sysfs prior to initialization
of iscsi_conn's dd_data. When userspace tried to access an attribute such
as the connect address, a NULL pointer dereference was observed.
Do not add iscsi_cls_conn to sysfs until it has been initialized. Remove
iscsi_create_conn() since it is no longer used.
Link: https://lore.kernel.org/r/20220310015759.3296841-3-haowenchao@huawei.com
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Wenchao Hao <haowenchao@huawei.com>
Signed-off-by: Wu Bo <wubo40@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
When the workqueue code was created it didn't allow variable args so we
have been using a temp buffer. Drop that.
Link: https://lore.kernel.org/r/20220226230435.38733-7-michael.christie@oracle.com
Reviewed-by: Chris Leech <cleech@redhat.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Instead of storing the iSCSI task pointer and the session age in the SCSI
pointer, use command-private variables. This patch prepares for removal of
the SCSI pointer from struct scsi_cmnd.
The list of iSCSI drivers has been obtained as follows:
$ git grep -lw iscsi_host_alloc
drivers/infiniband/ulp/iser/iscsi_iser.c
drivers/scsi/be2iscsi/be_main.c
drivers/scsi/bnx2i/bnx2i_iscsi.c
drivers/scsi/cxgbi/libcxgbi.c
drivers/scsi/iscsi_tcp.c
drivers/scsi/libiscsi.c
drivers/scsi/qedi/qedi_main.c
drivers/scsi/qla4xxx/ql4_os.c
include/scsi/libiscsi.h
Note: it is not clear to me how the qla4xxx driver can work without this
patch since it uses the scsi_cmnd::SCp.ptr member for two different
purposes:
- The qla4xxx driver uses this member to store a struct srb pointer.
- libiscsi uses this member to store a struct iscsi_task pointer.
Reviewed-by: Lee Duncan <lduncan@suse.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Himanshu Madhani <himanshu.madhani@oracle.com>
Cc: Chris Leech <cleech@redhat.com>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Nilesh Javali <njavali@marvell.com>
Cc: Manish Rangankar <mrangankar@marvell.com>
Cc: Karen Xie <kxie@chelsio.com>
Cc: Ketan Mukadam <ketan.mukadam@broadcom.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
iscsi
Link: https://lore.kernel.org/r/20220218195117.25689-26-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|- iscsi_if_destroy_conn |-dev_attr_show
|-iscsi_conn_teardown
|-spin_lock_bh |-iscsi_sw_tcp_conn_get_param
|-kfree(conn->persistent_address) |-iscsi_conn_get_param
|-kfree(conn->local_ipaddr)
==>|-read persistent_address
==>|-read local_ipaddr
|-spin_unlock_bh
When iscsi_conn_teardown() and iscsi_conn_get_param() happen in parallel, a
UAF may be triggered.
Link: https://lore.kernel.org/r/046ec8a0-ce95-d3fc-3235-666a7c65b224@huawei.com
Reported-by: Lu Tixiong <lutianxiong@huawei.com>
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Lixiaokeng <lixiaokeng@huawei.com>
Signed-off-by: Linfeilong <linfeilong@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Conditional statements are faster than indirect calls. Hence call
scsi_done() directly.
Link: https://lore.kernel.org/r/20211007202923.2174984-45-bvanassche@acm.org
Reviewed-by: Lee Duncan <lduncanb@suse.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Merge the 5.15/scsi-fixes branch into the staging tree to resolve UFS
conflict reported by sfr.
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Commit d39df15851 ("scsi: iscsi: Have abort handler get ref to conn")
added iscsi_get_conn()/iscsi_put_conn() calls during abort handling but
then also changed the handling of the case where we detect an already
completed task where we now end up doing a goto to the common put/cleanup
code. This results in a iscsi_task use after free, because the common
cleanup code will do a put on the iscsi_task.
This reverts the goto and moves the iscsi_get_conn() to after we've checked
if the iscsi_task is valid.
Link: https://lore.kernel.org/r/20211004210608.9962-1-michael.christie@oracle.com
Fixes: d39df15851 ("scsi: iscsi: Have abort handler get ref to conn")
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Commit ec29d0ac29 ("scsi: iscsi: Fix conn use after free during resets")
moved member ehwait from 'conn' to 'session', but left the initialization
of ehwait in iscsi_conn_setup().
Although a session can only have 1 conn currently, it is better to
initialize ehwait in iscsi_session_setup() in case we implement handling
multiple conns in the future.
Link: https://lore.kernel.org/r/20210911135159.20543-1-dinghui@sangfor.com.cn
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Ding Hui <dinghui@sangfor.com.cn>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Include Hannes' SCSI command result rework in the staging branch.
[mkp: remove DRIVER_SENSE from mpi3mr]
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This doesn't fix any bugs, but it makes more sense to free the pool after
we have removed the session. At that time we know nothing is touching any
of the session fields, because all devices have been removed and scans are
stopped.
Link: https://lore.kernel.org/r/20210525181821.7617-19-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
For aborts, qedi needs to cleanup the FW then send the TMF from a worker
thread. While it's doing these the cmd could complete normally and the TMF
could time out. libiscsi would then complete the iscsi_task which will call
into the driver to cleanup the driver level resources while it still might
be accessing them for the cleanup/abort.
This has iscsi_eh_abort keep the iscsi_task ref if the TMF times out, so
qedi does not have to worry about if the task is being freed while in use
and does not need to get its own ref.
Link: https://lore.kernel.org/r/20210525181821.7617-18-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
We have a ref to the task being aborted, so SCp.ptr will never be NULL. We
need to use iscsi_task_is_completed to check for the completed state.
Link: https://lore.kernel.org/r/20210525181821.7617-16-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
If we haven't done a unbind target call we can race where
iscsi_conn_teardown wakes up the EH thread and then frees the conn while
those threads are still accessing the conn ehwait.
We can only do one TMF per session so this just moves the TMF fields from
the conn to the session. We can then rely on the
iscsi_session_teardown->iscsi_remove_session->__iscsi_unbind_session call
to remove the target and it's devices, and know after that point there is
no device or scsi-ml callout trying to access the session.
Link: https://lore.kernel.org/r/20210525181821.7617-14-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The comment in iscsi_eh_session_reset is wrong and we don't wait for the
EH to complete before tearing down the conn. This has us get a ref to the
conn when we are not holding the eh_mutex/frwd_lock so it does not get
freed from under us.
Link: https://lore.kernel.org/r/20210525181821.7617-13-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
If SCSI midlayer is aborting a task when we are tearing down the conn we
could free the conn while the abort thread is accessing the conn. This has
the abort handler get a ref to the conn so it won't be freed from under it.
Note: this is not needed for device/target reset because we are holding the
eh_mutex when accessing the conn.
Link: https://lore.kernel.org/r/20210525181821.7617-12-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
There are a couple places where we could free the iscsi_cls_conn while it's
still in use. This adds some helpers to get/put a refcount on the struct
and converts an exiting user. Subsequent commits will then use the helpers
to fix 2 bugs in the eh code.
Link: https://lore.kernel.org/r/20210525181821.7617-11-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
During ep_disconnect we have been doing iscsi_suspend_tx/queue to block new
I/O but every driver except cxgbi and iscsi_tcp can still get I/O from
__iscsi_conn_send_pdu() if we haven't called iscsi_conn_failure() before
ep_disconnect. This could happen if we were terminating the session, and
the logout timed out before it was even sent to libiscsi.
Fix the issue by adding a helper which reverses the bind_conn call that
allows new I/O to be queued. Drivers implementing ep_disconnect can use this
to make sure new I/O is not queued to them when handling the disconnect.
Link: https://lore.kernel.org/r/20210525181821.7617-3-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Introduce scsi_build_sense() as a wrapper around scsi_build_sense_buffer()
to format the buffer and set the correct SCSI status.
Link: https://lore.kernel.org/r/20210427083046.31620-8-hare@suse.de
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This libsas fix is for a problem that occurs when trying to change the
cache type of an ATA device and the libiscsi one is a regression fix
from this merge window.
Signed-off-by: James E.J. Bottomley <jejb@linux.ibm.com>
-----BEGIN PGP SIGNATURE-----
iJwEABMIAEQWIQTnYEDbdso9F2cI+arnQslM7pishQUCYHuPRSYcamFtZXMuYm90
dG9tbGV5QGhhbnNlbnBhcnRuZXJzaGlwLmNvbQAKCRDnQslM7pisha52AQCXc8n0
6VAjfc+8aCqjX2Hpw4YCGeW5RYoNj1WXhiDv/AD+L4FVBMdQ4DE9ukH12YW7YBRS
qP03aNSHLCl8wfVon8Q=
=Btn4
-----END PGP SIGNATURE-----
Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI fixes from James Bottomley:
"Two fixes: the libsas fix is for a problem that occurs when trying to
change the cache type of an ATA device and the libiscsi one is a
regression fix from this merge window"
* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
scsi: libsas: Reset num_scatter if libata marks qc as NODATA
scsi: iscsi: Fix iSCSI cls conn state
In commit 9e67600ed6 ("scsi: iscsi: Fix race condition between login and
sync thread") I missed that libiscsi was now setting the iSCSI class state,
and that patch ended up resetting the state during conn stoppage and using
the wrong state value during ep_disconnect. This patch moves the setting of
the class state to the class module and then fixes the two issues above.
Link: https://lore.kernel.org/r/20210406171746.5016-1-michael.christie@oracle.com
Fixes: 9e67600ed6 ("scsi: iscsi: Fix race condition between login and sync thread")
Cc: Gulam Mohamed <gulam.mohamed@oracle.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
As the iSCSI parameters are exported back through sysfs, it should be
enforcing that they never are more than PAGE_SIZE (which should be more
than enough) before accepting updates through netlink.
Change all iSCSI sysfs attributes to use sysfs_emit().
Cc: stable@vger.kernel.org
Reported-by: Adam Nichols <adam@grimm-co.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Chris Leech <cleech@redhat.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
If we lose the session then relogin, but the new cmdsn window has shrunk
(due to something like an admin changing a setting) we will have the old
exp/max_cmdsn values and will never be able to update them. For example,
max_cmdsn would be 64, but if on the target the user set the window to be
smaller then the target could try to return the max_cmdsn as 32. We will
see that new max_cmdsn in the rsp but because it's lower than the old
max_cmdsn when the window was larger we will not update it.
So this patch has us reset the window values during session cleanup so they
can be updated after a new login.
Link: https://lore.kernel.org/r/20210207044608.27585-8-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This patch just breaks out the code that calculates the number of SCSI cmds
that will be used for a SCSI session. It also adds a check that we don't go
over the host's can_queue value.
Link: https://lore.kernel.org/r/20210207044608.27585-6-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
We allocate the iSCSI host workq in iscsi_host_alloc() so iscsi_host_free()
should do the destruction. Drivers can then do their error/goto handling
and call iscsi_host_free() to clean up what has been allocated in
iscsi_host_alloc().
Link: https://lore.kernel.org/r/20210207044608.27585-5-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The following bug was reported and debugged by wubo40@huawei.com:
When testing kernel 4.18 version, NULL pointer dereference problem occurs
in iscsi_eh_cmd_timed_out() function.
I think this bug in the upstream is still exists.
The analysis reasons are as follows:
1) For some reason, I/O command did not complete within the timeout
period. The block layer timer works, call scsi_times_out() to handle I/O
timeout logic. At the same time the command just completes.
2) scsi_times_out() call iscsi_eh_cmd_timed_out() to process timeout logic.
Although there is an NULL judgment for the task, the task has not been
released yet now.
3) iscsi_complete_task() calls __iscsi_put_task(). The task reference count
reaches zero, the conditions for free task is met, then
iscsi_free_task() frees the task, and sets sc->SCp.ptr = NULL. After
iscsi_eh_cmd_timed_out() passes the task judgment check, there can still
be NULL dereference scenarios.
CPU0 CPU3
|- scsi_times_out() |-
iscsi_complete_task()
| |
|- iscsi_eh_cmd_timed_out() |-
__iscsi_put_task()
| |
|- task=sc->SCp.ptr, task is not NUL, check passed |-
iscsi_free_task(task)
| |
| |-> sc->SCp.ptr
= NULL
| |
|- task is NULL now, NULL pointer dereference |
| |
\|/ \|/
Calltrace:
[380751.840862] BUG: unable to handle kernel NULL pointer dereference at
0000000000000138
[380751.843709] PGD 0 P4D 0
[380751.844770] Oops: 0000 [#1] SMP PTI
[380751.846283] CPU: 0 PID: 403 Comm: kworker/0:1H Kdump: loaded
Tainted: G
[380751.851467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996)
[380751.856521] Workqueue: kblockd blk_mq_timeout_work
[380751.858527] RIP: 0010:iscsi_eh_cmd_timed_out+0x15e/0x2e0 [libiscsi]
[380751.861129] Code: 83 ea 01 48 8d 74 d0 08 48 8b 10 48 8b 4a 50 48 85
c9 74 2c 48 39 d5 74
[380751.868811] RSP: 0018:ffffc1e280a5fd58 EFLAGS: 00010246
[380751.870978] RAX: ffff9fd1e84e15e0 RBX: ffff9fd1e84e6dd0 RCX:
0000000116acc580
[380751.873791] RDX: ffff9fd1f97a9400 RSI: ffff9fd1e84e1800 RDI:
ffff9fd1e4d6d420
[380751.876059] RBP: ffff9fd1e4d49000 R08: 0000000116acc580 R09:
0000000116acc580
[380751.878284] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9fd1e6e931e8
[380751.880500] R13: ffff9fd1e84e6ee0 R14: 0000000000000010 R15:
0000000000000003
[380751.882687] FS: 0000000000000000(0000) GS:ffff9fd1fac00000(0000)
knlGS:0000000000000000
[380751.885236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[380751.887059] CR2: 0000000000000138 CR3: 000000011860a001 CR4:
00000000003606f0
[380751.889308] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[380751.891523] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[380751.893738] Call Trace:
[380751.894639] scsi_times_out+0x60/0x1c0
[380751.895861] blk_mq_check_expired+0x144/0x200
[380751.897302] ? __switch_to_asm+0x35/0x70
[380751.898551] blk_mq_queue_tag_busy_iter+0x195/0x2e0
[380751.900091] ? __blk_mq_requeue_request+0x100/0x100
[380751.901611] ? __switch_to_asm+0x41/0x70
[380751.902853] ? __blk_mq_requeue_request+0x100/0x100
[380751.904398] blk_mq_timeout_work+0x54/0x130
[380751.905740] process_one_work+0x195/0x390
[380751.907228] worker_thread+0x30/0x390
[380751.908713] ? process_one_work+0x390/0x390
[380751.910350] kthread+0x10d/0x130
[380751.911470] ? kthread_flush_work_fn+0x10/0x10
[380751.913007] ret_from_fork+0x35/0x40
crash> dis -l iscsi_eh_cmd_timed_out+0x15e
xxxxx/drivers/scsi/libiscsi.c: 2062
1970 enum blk_eh_timer_return iscsi_eh_cmd_timed_out(struct scsi_cmnd
*sc)
{
...
1984 spin_lock_bh(&session->frwd_lock);
1985 task = (struct iscsi_task *)sc->SCp.ptr;
1986 if (!task) {
1987 /*
1988 * Raced with completion. Blk layer has taken
ownership
1989 * so let timeout code complete it now.
1990 */
1991 rc = BLK_EH_DONE;
1992 goto done;
1993 }
...
2052 for (i = 0; i < conn->session->cmds_max; i++) {
2053 running_task = conn->session->cmds[i];
2054 if (!running_task->sc || running_task == task ||
2055 running_task->state != ISCSI_TASK_RUNNING)
2056 continue;
2057
2058 /*
2059 * Only check if cmds started before this one have
made
2060 * progress, or this could never fail
2061 */
2062 if (time_after(running_task->sc->jiffies_at_alloc,
2063 task->sc->jiffies_at_alloc)) <---
2064 continue;
2065
...
}
carsh> struct scsi_cmnd ffff9fd1e6e931e8
struct scsi_cmnd {
...
SCp = {
ptr = 0x0, <--- iscsi_task
this_residual = 0,
...
},
}
To prevent this, we take a ref to the cmd under the back (completion) lock
so if the completion side were to call iscsi_complete_task() on the task
while the timer/eh paths are not holding the back_lock it will not be freed
from under us.
Note that this requires the previous patch, "scsi: libiscsi: Drop
taskqueuelock" because bnx2i sleeps in its cleanup_task callout if the cmd
is aborted. If the EH/timer and completion path are racing we don't know
which path will do the last put. The previous patch moved the operations we
needed to do under the forward lock to cleanup_queued_task. Once that has
run we can drop the forward lock for the cmd and bnx2i no longer has to
worry about if the EH, timer or completion path did the ast put and if the
forward lock is held or not since it won't be.
Link: https://lore.kernel.org/r/20210207044608.27585-4-michael.christie@oracle.com
Reported-by: Wu Bo <wubo40@huawei.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
The purpose of the taskqueuelock was to handle the issue where a bad target
decides to send a R2T and before its data has been sent decides to send a
cmd response to complete the cmd. The following patches fix up the
frwd/back locks so they are taken from the queue/xmit (frwd) and completion
(back) paths again. To get there this patch removes the taskqueuelock which
for iSCSI xmit wq based drivers was taken in the queue, xmit and completion
paths.
Instead of the lock, we just make sure we have a ref to the task when we
queue a R2T, and then we always remove the task from the requeue list in
the xmit path or the forced cleanup paths.
Link: https://lore.kernel.org/r/20210207044608.27585-3-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
If iscsi_prep_scsi_cmd_pdu() fails we try to add it back to the cmdqueue,
but we leave it partially setup. We don't have functions that can undo the
pdu and init task setup. We only have cleanup_task which can clean up both
parts. So this has us just fail the cmd and go through the standard cleanup
routine and then have the SCSI midlayer retry it like is done when it fails
in the queuecommand path.
Link: https://lore.kernel.org/r/20210207044608.27585-2-michael.christie@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This series consists of the usual driver updates (ufs, qla2xxx,
smartpqi, target, zfcp, fnic, mpt3sas, ibmvfc) plus a load of
cleanups, a major power management rework and a load of assorted minor
updates. There are a few core updates (formatting fixes being the big
one) but nothing major this cycle.
Signed-off-by: James E.J. Bottomley <jejb@linux.ibm.com>
-----BEGIN PGP SIGNATURE-----
iJwEABMIAEQWIQTnYEDbdso9F2cI+arnQslM7pishQUCX9o0KSYcamFtZXMuYm90
dG9tbGV5QGhhbnNlbnBhcnRuZXJzaGlwLmNvbQAKCRDnQslM7pishbOZAP9D5NTN
J7dJUo2MIMy84YBu+d9ag7yLlNiRWVY2yw5vHwD/Z7JjAVLwz/tzmyjU9//o2J6w
hwhOv6Uto89gLCWSEz8=
=KUPT
-----END PGP SIGNATURE-----
Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI updates from James Bottomley:
"This consists of the usual driver updates (ufs, qla2xxx, smartpqi,
target, zfcp, fnic, mpt3sas, ibmvfc) plus a load of cleanups, a major
power management rework and a load of assorted minor updates.
There are a few core updates (formatting fixes being the big one) but
nothing major this cycle"
* tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi: (279 commits)
scsi: mpt3sas: Update driver version to 36.100.00.00
scsi: mpt3sas: Handle trigger page after firmware update
scsi: mpt3sas: Add persistent MPI trigger page
scsi: mpt3sas: Add persistent SCSI sense trigger page
scsi: mpt3sas: Add persistent Event trigger page
scsi: mpt3sas: Add persistent Master trigger page
scsi: mpt3sas: Add persistent trigger pages support
scsi: mpt3sas: Sync time periodically between driver and firmware
scsi: qla2xxx: Update version to 10.02.00.104-k
scsi: qla2xxx: Fix device loss on 4G and older HBAs
scsi: qla2xxx: If fcport is undergoing deletion complete I/O with retry
scsi: qla2xxx: Fix the call trace for flush workqueue
scsi: qla2xxx: Fix flash update in 28XX adapters on big endian machines
scsi: qla2xxx: Handle aborts correctly for port undergoing deletion
scsi: qla2xxx: Fix N2N and NVMe connect retry failure
scsi: qla2xxx: Fix FW initialization error on big endian machines
scsi: qla2xxx: Fix crash during driver load on big endian machines
scsi: qla2xxx: Fix compilation issue in PPC systems
scsi: qla2xxx: Don't check for fw_started while posting NVMe command
scsi: qla2xxx: Tear down session if FW say it is down
...
iSCSI NOPs are sometimes "lost", mistakenly sent to the user-land iscsid
daemon instead of handled in the kernel, as they should be, resulting in a
message from the daemon like:
iscsid: Got nop in, but kernel supports nop handling.
This can occur because of the new forward- and back-locks, and the fact
that an iSCSI NOP response can occur before processing of the NOP send is
complete. This can result in "conn->ping_task" being NULL in
iscsi_nop_out_rsp(), when the pointer is actually in the process of being
set.
To work around this, we add a new state to the "ping_task" pointer. In
addition to NULL (not assigned) and a pointer (assigned), we add the state
"being set", which is signaled with an INVALID pointer (using "-1").
Link: https://lore.kernel.org/r/20201106193317.16993-1-leeman.duncan@gmail.com
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Commit 3ce419662d ("scsi: iscsi: Register sysfs for iscsi workqueue")
enabled 'cpumask' support for iSCSI workqueues. However, it is unnecessary
to set max_active = 2 since 'cpumask' can still be modified when max_active
is 1.
This patch sets max_active to 1 so as to keep the same behaviour as before.
Link: https://lore.kernel.org/r/20200701030745.16897-1-bob.liu@oracle.com
Reviewed-by: Mike Christie <michael.christie@oracle.com>
Signed-off-by: Bob Liu <bob.liu@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
This patch enables setting cpu affinity through "cpumask" for iscsi
workqueues (iscsi_q_xx and iscsi_eh), so as to get performance isolation.
The max number of active worker was changed form 1 to 2, because "cpumask"
of ordered workqueue isn't allowed to change.
Link: https://lore.kernel.org/r/20200505011908.15538-1-bob.liu@oracle.com
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Bob Liu <bob.liu@oracle.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Fix an error count for active session if the total_cmds is invalid on the
function iscsi_session_setup(). Decrement the number of active sessions
before the funcion return.
Link: https://lore.kernel.org/r/EDBAAA0BBBA2AC4E9C8B6B81DEEE1D6916A28542@DGGEML525-MBS.china.huawei.com
Reviewed-by: Lee Duncan <lduncan@suuse.com>
Signed-off-by: Wu Bo <wubo40@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
If an iSCSI connection happens to fail while the daemon isn't running (due
to a crash or for another reason), the kernel failure report is not
received. When the daemon restarts, there is insufficient kernel state in
sysfs for it to know that this happened. open-iscsi tries to reopen every
connection, but on different initiators, we'd like to know which
connections have failed.
There is session->state, but that has a different lifetime than an iSCSI
connection, so it doesn't directly reflect the connection state.
[mkp: typos]
Link: https://lore.kernel.org/r/20200317233422.532961-1-krisman@collabora.com
Cc: Khazhismel Kumykov <khazhy@google.com>
Suggested-by: Junho Ryu <jayr@google.com>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Gabriel Krisman Bertazi <krisman@collabora.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Some time ago the block layer was modified such that timeout handlers are
called from thread context instead of interrupt context. Make it safe to
run the iSCSI timeout handler in thread context. This patch fixes the
following lockdep complaint:
================================
WARNING: inconsistent lock state
5.5.1-dbg+ #11 Not tainted
--------------------------------
inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage.
kworker/7:1H/206 [HC0[0]:SC0[0]:HE1:SE1] takes:
ffff88802d9827e8 (&(&session->frwd_lock)->rlock){+.?.}, at: iscsi_eh_cmd_timed_out+0xa6/0x6d0 [libiscsi]
{IN-SOFTIRQ-W} state was registered at:
lock_acquire+0x106/0x240
_raw_spin_lock+0x38/0x50
iscsi_check_transport_timeouts+0x3e/0x210 [libiscsi]
call_timer_fn+0x132/0x470
__run_timers.part.0+0x39f/0x5b0
run_timer_softirq+0x63/0xc0
__do_softirq+0x12d/0x5fd
irq_exit+0xb3/0x110
smp_apic_timer_interrupt+0x131/0x3d0
apic_timer_interrupt+0xf/0x20
default_idle+0x31/0x230
arch_cpu_idle+0x13/0x20
default_idle_call+0x53/0x60
do_idle+0x38a/0x3f0
cpu_startup_entry+0x24/0x30
start_secondary+0x222/0x290
secondary_startup_64+0xa4/0xb0
irq event stamp: 1383705
hardirqs last enabled at (1383705): [<ffffffff81aace5c>] _raw_spin_unlock_irq+0x2c/0x50
hardirqs last disabled at (1383704): [<ffffffff81aacb98>] _raw_spin_lock_irq+0x18/0x50
softirqs last enabled at (1383690): [<ffffffffa0e2efea>] iscsi_queuecommand+0x76a/0xa20 [libiscsi]
softirqs last disabled at (1383682): [<ffffffffa0e2e998>] iscsi_queuecommand+0x118/0xa20 [libiscsi]
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&(&session->frwd_lock)->rlock);
<Interrupt>
lock(&(&session->frwd_lock)->rlock);
*** DEADLOCK ***
2 locks held by kworker/7:1H/206:
#0: ffff8880d57bf928 ((wq_completion)kblockd){+.+.}, at: process_one_work+0x472/0xab0
#1: ffff88802b9c7de8 ((work_completion)(&q->timeout_work)){+.+.}, at: process_one_work+0x476/0xab0
stack backtrace:
CPU: 7 PID: 206 Comm: kworker/7:1H Not tainted 5.5.1-dbg+ #11
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
Workqueue: kblockd blk_mq_timeout_work
Call Trace:
dump_stack+0xa5/0xe6
print_usage_bug.cold+0x232/0x23b
mark_lock+0x8dc/0xa70
__lock_acquire+0xcea/0x2af0
lock_acquire+0x106/0x240
_raw_spin_lock+0x38/0x50
iscsi_eh_cmd_timed_out+0xa6/0x6d0 [libiscsi]
scsi_times_out+0xf4/0x440 [scsi_mod]
scsi_timeout+0x1d/0x20 [scsi_mod]
blk_mq_check_expired+0x365/0x3a0
bt_iter+0xd6/0xf0
blk_mq_queue_tag_busy_iter+0x3de/0x650
blk_mq_timeout_work+0x1af/0x380
process_one_work+0x56d/0xab0
worker_thread+0x7a/0x5d0
kthread+0x1bc/0x210
ret_from_fork+0x24/0x30
Fixes: 287922eb0b ("block: defer timeouts to a workqueue")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Keith Busch <keith.busch@intel.com>
Cc: Lee Duncan <lduncan@suse.com>
Cc: Chris Leech <cleech@redhat.com>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/20191209173457.187370-1-bvanassche@acm.org
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Based on 1 normalized pattern(s):
this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license as published by
the free software foundation either version 2 of the license or at
your option any later version this program is distributed in the
hope that it will be useful but without any warranty without even
the implied warranty of merchantability or fitness for a particular
purpose see the gnu general public license for more details you
should have received a copy of the gnu general public license along
with this program if not write to the free software foundation inc
59 temple place suite 330 boston ma 02111 1307 usa
extracted by the scancode license scanner the SPDX license identifier
GPL-2.0-or-later
has been chosen to replace the boilerplate/reference in 1334 file(s).
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Allison Randal <allison@lohutok.net>
Reviewed-by: Richard Fontana <rfontana@redhat.com>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190527070033.113240726@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This is the final round of mostly small fixes and performance
improvements to our initial submit. The main regression fix is the
ia64 simscsi build failure which was missed in the serial number
elimination conversion.
Signed-off-by: James E.J. Bottomley <jejb@linux.ibm.com>
-----BEGIN PGP SIGNATURE-----
iJwEABMIAEQWIQTnYEDbdso9F2cI+arnQslM7pishQUCXIxBayYcamFtZXMuYm90
dG9tbGV5QGhhbnNlbnBhcnRuZXJzaGlwLmNvbQAKCRDnQslM7pisherpAP4rxLpX
bcUnQnEsvoxys/JyoK08Qfv1JebZo1B2MAZ62wD/VZ7LpOuzVLhsM2KhLFGRrs1/
7D2K4tgtO2dQsFix7H0=
=pcHl
-----END PGP SIGNATURE-----
Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull more SCSI updates from James Bottomley:
"This is the final round of mostly small fixes and performance
improvements to our initial submit.
The main regression fix is the ia64 simscsi build failure which was
missed in the serial number elimination conversion"
* tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi: (24 commits)
scsi: ia64: simscsi: use request tag instead of serial_number
scsi: aacraid: Fix performance issue on logical drives
scsi: lpfc: Fix error codes in lpfc_sli4_pci_mem_setup()
scsi: libiscsi: Hold back_lock when calling iscsi_complete_task
scsi: hisi_sas: Change SERDES_CFG init value to increase reliability of HiLink
scsi: hisi_sas: Send HARD RESET to clear the previous affiliation of STP target port
scsi: hisi_sas: Set PHY linkrate when disconnected
scsi: hisi_sas: print PHY RX errors count for later revision of v3 hw
scsi: hisi_sas: Fix a timeout race of driver internal and SMP IO
scsi: hisi_sas: Change return variable type in phy_up_v3_hw()
scsi: qla2xxx: check for kstrtol() failure
scsi: lpfc: fix 32-bit format string warning
scsi: lpfc: fix unused variable warning
scsi: target: tcmu: Switch to bitmap_zalloc()
scsi: libiscsi: fall back to sendmsg for slab pages
scsi: qla2xxx: avoid printf format warning
scsi: lpfc: resolve static checker warning in lpfc_sli4_hba_unset
scsi: lpfc: Correct __lpfc_sli_issue_iocb_s4 lockdep check
scsi: ufs: hisi: fix ufs_hba_variant_ops passing
scsi: qla2xxx: Fix panic in qla_dfs_tgt_counters_show
...
