mirror of
https://github.com/torvalds/linux.git
synced 2024-11-30 16:11:38 +00:00
fa349e396e
When AF_XDP is used on on a veth interface the RX ring is updated in two
steps. veth_xdp_rcv() removes packet descriptors from the FILL ring
fills them and places them in the RX ring updating the cached_prod
pointer. Later xdp_do_flush() syncs the RX ring prod pointer with the
cached_prod pointer allowing user-space to see the recently filled in
descriptors. The rings are intended to be SPSC, however the existing
order in veth_poll allows the xdp_do_flush() to run concurrently with
another CPU creating a race condition that allows user-space to see old
or uninitialized descriptors in the RX ring. This bug has been observed
in production systems.
To summarize, we are expecting this ordering:
CPU 0 __xsk_rcv_zc()
CPU 0 __xsk_map_flush()
CPU 2 __xsk_rcv_zc()
CPU 2 __xsk_map_flush()
But we are seeing this order:
CPU 0 __xsk_rcv_zc()
CPU 2 __xsk_rcv_zc()
CPU 0 __xsk_map_flush()
CPU 2 __xsk_map_flush()
This occurs because we rely on NAPI to ensure that only one napi_poll
handler is running at a time for the given veth receive queue.
napi_schedule_prep() will prevent multiple instances from getting
scheduled. However calling napi_complete_done() signals that this
napi_poll is complete and allows subsequent calls to
napi_schedule_prep() and __napi_schedule() to succeed in scheduling a
concurrent napi_poll before the xdp_do_flush() has been called. For the
veth driver a concurrent call to napi_schedule_prep() and
__napi_schedule() can occur on a different CPU because the veth xmit
path can additionally schedule a napi_poll creating the race.
The fix as suggested by Magnus Karlsson, is to simply move the
xdp_do_flush() call before napi_complete_done(). This syncs the
producer ring pointers before another instance of napi_poll can be
scheduled on another CPU. It will also slightly improve performance by
moving the flush closer to when the descriptors were placed in the
RX ring.
Fixes:
|
||
|---|---|---|
| .. | ||
| accel | ||
| accessibility | ||
| acpi | ||
| amba | ||
| android | ||
| ata | ||
| atm | ||
| auxdisplay | ||
| base | ||
| bcma | ||
| block | ||
| bluetooth | ||
| bus | ||
| cdrom | ||
| char | ||
| clk | ||
| clocksource | ||
| comedi | ||
| connector | ||
| counter | ||
| cpufreq | ||
| cpuidle | ||
| crypto | ||
| cxl | ||
| dax | ||
| dca | ||
| devfreq | ||
| dio | ||
| dma | ||
| dma-buf | ||
| edac | ||
| eisa | ||
| extcon | ||
| firewire | ||
| firmware | ||
| fpga | ||
| fsi | ||
| gnss | ||
| gpio | ||
| gpu | ||
| greybus | ||
| hid | ||
| hsi | ||
| hte | ||
| hv | ||
| hwmon | ||
| hwspinlock | ||
| hwtracing | ||
| i2c | ||
| i3c | ||
| idle | ||
| iio | ||
| infiniband | ||
| input | ||
| interconnect | ||
| iommu | ||
| ipack | ||
| irqchip | ||
| isdn | ||
| leds | ||
| macintosh | ||
| mailbox | ||
| mcb | ||
| md | ||
| media | ||
| memory | ||
| memstick | ||
| message | ||
| mfd | ||
| misc | ||
| mmc | ||
| most | ||
| mtd | ||
| mux | ||
| net | ||
| nfc | ||
| ntb | ||
| nubus | ||
| nvdimm | ||
| nvme | ||
| nvmem | ||
| of | ||
| opp | ||
| parisc | ||
| parport | ||
| pci | ||
| pcmcia | ||
| peci | ||
| perf | ||
| phy | ||
| pinctrl | ||
| platform | ||
| pnp | ||
| power | ||
| powercap | ||
| pps | ||
| ps3 | ||
| ptp | ||
| pwm | ||
| rapidio | ||
| ras | ||
| regulator | ||
| remoteproc | ||
| reset | ||
| rpmsg | ||
| rtc | ||
| s390 | ||
| sbus | ||
| scsi | ||
| sh | ||
| siox | ||
| slimbus | ||
| soc | ||
| soundwire | ||
| spi | ||
| spmi | ||
| ssb | ||
| staging | ||
| target | ||
| tc | ||
| tee | ||
| thermal | ||
| thunderbolt | ||
| tty | ||
| ufs | ||
| uio | ||
| usb | ||
| vdpa | ||
| vfio | ||
| vhost | ||
| video | ||
| virt | ||
| virtio | ||
| vlynq | ||
| w1 | ||
| watchdog | ||
| xen | ||
| zorro | ||
| Kconfig | ||
| Makefile | ||