0f90522591
Recent months, our customer reported several kernel crashes all
preceding with following message:
NETDEV WATCHDOG: eth2 (enic): transmit queue 0 timed out
Error message of one of those crashes:
BUG: unable to handle kernel paging request at ffffffffa007e090
After analyzing severl vmcores, I found that most of crashes are
caused by memory corruption. And all the corrupted memory areas
are overwritten by data of network packets. Moreover, I also found
that the tx queues were enabled over watchdog reset.
After going through the source code, I found that in enic_stop(),
the tx queues stopped by netif_tx_disable() could be woken up over
a small time window between netif_tx_disable() and the
napi_disable() by the following code path:
napi_poll->
enic_poll_msix_wq->
vnic_cq_service->
enic_wq_service->
netif_wake_subqueue(enic->netdev, q_number)->
test_and_clear_bit(__QUEUE_STATE_DRV_XOFF, &txq->state)
In turn, upper netowrk stack could queue skb to ENIC NIC though
enic_hard_start_xmit(). And this might introduce some race condition.
Our customer comfirmed that this kind of kernel crash doesn't occur over
90 days since they applied this patch.
Signed-off-by: Firo Yang <firo.yang@suse.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
|
||
|---|---|---|
| .. | ||
| cq_desc.h | ||
| cq_enet_desc.h | ||
| enic_api.c | ||
| enic_api.h | ||
| enic_clsf.c | ||
| enic_clsf.h | ||
| enic_dev.c | ||
| enic_dev.h | ||
| enic_ethtool.c | ||
| enic_main.c | ||
| enic_pp.c | ||
| enic_pp.h | ||
| enic_res.c | ||
| enic_res.h | ||
| enic.h | ||
| Kconfig | ||
| Makefile | ||
| rq_enet_desc.h | ||
| vnic_cq.c | ||
| vnic_cq.h | ||
| vnic_dev.c | ||
| vnic_dev.h | ||
| vnic_devcmd.h | ||
| vnic_enet.h | ||
| vnic_intr.c | ||
| vnic_intr.h | ||
| vnic_nic.h | ||
| vnic_resource.h | ||
| vnic_rq.c | ||
| vnic_rq.h | ||
| vnic_rss.h | ||
| vnic_stats.h | ||
| vnic_vic.c | ||
| vnic_vic.h | ||
| vnic_wq.c | ||
| vnic_wq.h | ||
| wq_enet_desc.h | ||