Currently, the driver calls memset() to set all relevant context memory
used by the chip to the initial value. This can take many milliseconds
with the potentially large number of context pages allocated for the
chip.
To make this faster, we only need to initialize the "context kind" field
of each block of context memory. This patch sets up the infrastructure
to do that with the bnxt_mem_init structure. In the next patch, we'll
add the logic to obtain the offset of the "context kind" from the
firmware. This patch is not changing the current behavior of calling
memset() to initialize all relevant context memory.
Reviewed-by: Pavan Chebbi <pavan.chebbi@broadcom.com>
Reviewed-by: Edwin Peer <edwin.peer@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
During some fatal firmware error conditions, the PCI config space
register 0x2e which normally contains the subsystem ID will become
0xffff. This register will revert back to the normal value after
the chip has completed core reset. If we detect this condition,
we can poll this config register immediately for the value to revert.
Because we use config read cycles to poll this register, there is no
possibility of Master Abort if we happen to read it during core reset.
This speeds up recovery significantly as we don't have to wait for the
conservative min_time before polling MMIO to see if the firmware has
come out of reset. As soon as this register changes value we can
proceed to re-initialize the device.
Reviewed-by: Edwin Peer <edwin.peer@broadcom.com>
Reviewed-by: Vasundhara Volam <vasundhara-v.volam@broadcom.com>
Reviewed-by: Andy Gospodarek <gospo@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Newer devices may have local context memory instead of relying on the
host for backing store. In these cases, HWRM_FUNC_BACKING_STORE_QCAPS
will return a zero entry size to indicate contexts for which the host
should not allocate backing store.
Selectively allocate context memory based on device capabilities and
only enable backing store for the appropriate contexts.
Signed-off-by: Edwin Peer <edwin.peer@broadcom.com>
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The main changes are the echo request/response from firmware for error
detection and the NO_FCS feature to transmit frames without FCS.
Signed-off-by: Michael Chan <michael.chan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The PCA95xx GPIO expander requires GPIO controller properties to operate
properly.
Signed-off-by: Krzysztof Kozlowski <krzk@kernel.org>
Acked-by: Joel Stanley <joel@jms.id.au>
Link: https://lore.kernel.org/r/20210212162640.66677-1-krzk@kernel.org'
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Add support for the global clock controller found on SC7280
based devices. This should allow most non-multimedia device
drivers to probe and control their clocks.
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Link: https://lore.kernel.org/r/1612981579-17391-3-git-send-email-tdas@codeaurora.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
Add device tree bindings for graphics clock controller for
Qualcomm Technology Inc's SDM630 and SDM660 SoCs.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-10-angelogioacchino.delregno@somainline.org
Reviewed-by: Rob Herring <robh@kernel.org>
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
In the previous commit ("clk: qcom: rcg2: Stop hardcoding gfx3d pingpong
parent numbers") the gfx3d ping-pong ops (clk_gfx3d_ops) were
generalized in order to be able to reuse the same ops for more than just
one clock for one SoC: follow the change here in the MSM8996 MMCC.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-7-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
The function clk_gfx3d_determine_rate is selecting different PLLs
to manage the GFX3D clock source in a special way: this one needs
to be ping-pong'ed on different PLLs to ensure stability during
frequency switching (set a PLL rate, let it stabilize, switch the
RCG to the new PLL) and fast frequency transitions.
This technique is currently being used in the MSM8996 SoC and the
function was assuming that the parents were always at a specific
index in the parents list, which is TRUE, if we use this only on
the MSM8996 MMCC.
Unfortunately, MSM8996 is not the only SoC that needs to ping-pong
the graphics RCG, so choices are:
1. Make new special ops just to hardcode *again* other indexes,
creating code duplication for (imo) no reason; or
2. Generalize this function, so that it becomes usable for a range
of SoCs with slightly different ping-pong configuration.
In this commit, the second road was taken: define a new "special"
struct clk_rcg2_gfx3d, containing the ordered list of parents to
ping-pong the graphics clock on, and the "regular" rcg2 clock
structure in order to generalize the clk_gfx3d_determine_rate
function and make it working for other SoCs.
As for the function itself it is left with the assumption that we
need to ping-pong over three parents. The reasons for this are:
1. The initial model was MSM8996, which has 3 parents for the
graphics clock pingpong;
2. The other example that was taken into consideration is the
SDM630/636/660 SoC gpu clock controller, which is ping-ponging
over two dynamic clocked and one fixed clock PLL.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-6-angelogioacchino.delregno@somainline.org
[sboyd@kernel.org: Grow some local variables, drop do_div() usage in
favor of plain division, we're not dealing with a u64 here]
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
Add a driver for the multimedia clock controller found on SDM660
based devices. This should allow most multimedia device drivers
to probe and control their clocks.
Signed-off-by: Martin Botka <martin.botka@somainline.org>
Co-developed-by: Konrad Dybcio <konrad.dybcio@somainline.org>
Signed-off-by: Konrad Dybcio <konrad.dybcio@somainline.org>
[angelogioacchino.delregno@somainline.org: Cleaned up SDM630 clock fixups]
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-4-angelogioacchino.delregno@somainline.org
[sboyd@kernel.org: Silence NULL pointer sparse warnings]
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
This clock is critical for any access to the GPU: gating it will
crash the system when the GPU has been initialized (so, you cannot
gate it unless you deinit the Adreno completely).
So, to achieve a working state with GPU on, set the CLK_IS_CRITICAL
flag to this clock.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-3-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
Similarly to MSM8998, any access to the MMSS depends on this clock.
Gating it will crash the system when RPMCC inits mmssnoc_axi_rpm_clk.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-2-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
The GPU PLL0 is not a fixed PLL and the rate can be set on it:
this is necessary especially on boards which bootloader is setting
a very low rate on this PLL before booting Linux, which would be
unsuitable for postdividing to reach the maximum allowed Adreno GPU
frequency of 710MHz (or, actually, even 670MHz..) on this SoC.
To allow setting rates on the GPU PLL0, also define VCO boundaries
and set the CLK_SET_RATE_PARENT flag to the GPU PLL0 postdivider.
With this change, the Adreno GPU is now able to scale through all
the available frequencies.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210114221059.483390-12-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
The GPU GX GDSC has GPU_GX_BCR reset and gfx3d_clk CXC, as stated
on downstream kernels (and as verified upstream, because otherwise
random lockups happen).
Also, add PWRSTS_RET and NO_RET_PERIPH: also as found downstream,
and also as verified here, to avoid GPU related lockups it is
necessary to force retain mem, but *not* peripheral when enabling
this GDSC (and, of course, the inverse on disablement).
With this change, the GPU finally works flawlessly on my four
different MSM8998 devices from two different manufacturers.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210114221059.483390-11-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
In some rare occasions, we want to only set the RETAIN_MEM bit, but
not the RETAIN_PERIPH one: this is seen on at least SDM630/636/660's
GPU-GX GDSC, where unsetting and setting back the RETAIN_PERIPH bit
will generate chaos and panics during GPU suspend time (mainly, the
chaos is unaligned access).
For this reason, introduce a new NO_RET_PERIPH flag to the GDSC
driver to address this corner case.
Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org>
Link: https://lore.kernel.org/r/20210113183817.447866-8-angelogioacchino.delregno@somainline.org
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
This adds a basic driver for the MPLL block found in MStar/SigmaStar
ARMv7 SoCs.
Currently this driver is only good for calculating the rates of it's
outputs and the actual configuration must be done before the kernel
boots. Usually this is done even before u-boot starts.
This driver targets the MPLL block found in the MSC313/MSC313E but
there is no documentation this chip so the register descriptions for
the another MStar chip the MST786 were used as they seem to match.
Signed-off-by: Daniel Palmer <daniel@0x0f.com>
Link: https://lore.kernel.org/r/20210211052206.2955988-5-daniel@0x0f.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
Add a devm helper for clk_hw_register_fixed_factor() so that drivers that internally
register fixed factor clocks for things like dividers don't need to manually unregister
them on remove or if probe fails.
Signed-off-by: Daniel Palmer <daniel@0x0f.com>
Link: https://lore.kernel.org/r/20210211052206.2955988-4-daniel@0x0f.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
Simple header to document the relationship between the MPLL outputs
and which divider they come from.
Output 0 is missing because it should not be consumed.
Signed-off-by: Daniel Palmer <daniel@0x0f.com>
Acked-by: Rob Herring <robh@kernel.org>
Link: https://lore.kernel.org/r/20210211052206.2955988-2-daniel@0x0f.com
Signed-off-by: Stephen Boyd <sboyd@kernel.org>
Pull LED fix from Pavel Machek:
"One-liner fixing a build problem"
* 'for-rc8-5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/pavel/linux-leds:
leds: rt8515: add V4L2_FLASH_LED_CLASS dependency
- Fix CONFIG_TRIM_UNUSED_KSYMS build for ppc64
- Use pkg-config for scripts/sign-file.c CFLAGS
-----BEGIN PGP SIGNATURE-----
iQJJBAABCgAzFiEEbmPs18K1szRHjPqEPYsBB53g2wYFAmApVnIVHG1hc2FoaXJv
eUBrZXJuZWwub3JnAAoJED2LAQed4NsG/cQP/iEd73lUabEPZzyBzOSFVS1nsMIf
xU+rtASlqqd2n+yDx9tv5cIgGdDXBPtf360tQ8W6RPXgjpaMLS+FJQfsvSpmZNwx
UbSwNvPzLw6I5Bs63V4bqkFnnL+14grPonbqUdRHoa/0h1C1vCpGlP5MZMCfoL9k
wjla90h/TYysLyS1C5w71P1tr9yAF0UvG0u1d0h49AM+BaRkyxXTvokhKbfD47iz
SP7b9Y7VarfVq1YOPVh37v2nueWaG41YtPjBLVM42oa2gXdrsasb3LVIgahsgc3N
QsiKMmz+3pE8GPG0smKbuFLXEi2JoXVnVWysSlJjJNIjRr2R6TB5pG6H2ZmmAqkV
jWjRPvPky9LbZC/psuwEvPjnDi+Q0byYKy/L1xeVMhnPcITK19jF6KNUAZ6d8ABI
YX5ZkLh2EHOmVvLxD4vsvOURSQERI7UtnMZ7dwhp3VsXvN2mSsZOlnTQKWJ1L+WD
JB8dP6dJZtiUI2q46hyA8NCsHaoJ7o+NnaanMZjdyzEvCDwUfAeY3rk/3p4u42jg
izEUYv0KR05qEbXOzJ4mr6Omb9onePmDHj3J9l92yeTexWJfSJeQFk3I+aIFnQos
L2VHLTro9KOtnT4qjMnZRnqxUzPQ/zI0+JvPXPb1kYu91bfn3kQl2U5L3THvpDXc
EohIJNYbUe07vS2f
=Pp2I
-----END PGP SIGNATURE-----
Merge tag 'kbuild-fixes-v5.11-3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
Pull Kbuild fixes from Masahiro Yamada:
- Fix CONFIG_TRIM_UNUSED_KSYMS build for ppc64
- Use pkg-config for scripts/sign-file.c CFLAGS
* tag 'kbuild-fixes-v5.11-3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
scripts: set proper OpenSSL include dir also for sign-file
sparc: remove wrong comment from arch/sparc/include/asm/Kbuild
kbuild: fix CONFIG_TRIM_UNUSED_KSYMS build for ppc64
- A fix for the fix to disable CET instrumentation generation for kernel code.
We forgot 32-bit, which we seem to do very often nowadays.
- A Xen PV fix to irqdomain init ordering.
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEzv7L6UO9uDPlPSfHEsHwGGHeVUoFAmAo+tUACgkQEsHwGGHe
VUqR0hAAqdYjPIk3ecxNKqjdhd9ggV66hmkwnq7iWkWGi8p33q7y9hAA1dG2PBc1
dMuGs9jOqGs5U5FrYjFD8nROVQnvNGt5ISOK02ABMqtiGSUVfInaib1nlCcyLTlt
y9a1PMISyuexrhgGwRe1LjTSV6BQmH+jlBSpioJ4lCYugfhttOPpghRKY3M2HKQ3
LhTh0j3+ls2onvgr7DP3wLfz+77ccxK7gRRc0BczqnRjVdpIdXpmxOOOaXA0Yhyd
FKnXDZ4ApIFwGB6gUvv7SlKKMULL7ev3LJQnRK4A7AUOJF37e6yMCDI7neGs6Qde
r3j7xwqNzge0B2vGaT6IOsKcytduxcB6/gpZVpExHhVBWgCl1AA7Ech+5kGdtWky
xMEYyGZZgpVvFND8iTYuoXvrkUOX3pAdXgsNLvcN5OxQrNGGfzpSzikwtZAsYqM4
Ckdv/XkMG9ALM46cNAR4kdecjOypP2zSt7k/olWbma4/MMLBjD+vAm1Vzjws7oMw
GSgENGMUa0zjLmi4gN5IZnMCbfBHkzvMlyFBop7cSt983nGLap8z/vx9J6R0jXHy
Nhlc129Pd5lRIptxFg0d6Fsn4IimOASFyfgxp1kGSR7nd9jnogXsDfMqC4TLCnwW
p1d+1JkUWIlrC2hqB+whQ23s81H9Myt4lp87GOT7s+FWmtAc7p0=
=9CtU
-----END PGP SIGNATURE-----
Merge tag 'x86_urgent_for_v5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fixes from Borislav Petkov:
"I kinda knew while typing 'I hope this is the last batch of x86/urgent
updates' last week, Murphy was reading too and uttered 'Hold my
beer!'.
So here's more fixes... Thanks Murphy.
Anyway, three more x86/urgent fixes for 5.11 final. We should be
finally ready (famous last words). :-)
- An SGX use after free fix
- A fix for the fix to disable CET instrumentation generation for
kernel code. We forgot 32-bit, which we seem to do very often
nowadays
- A Xen PV fix to irqdomain init ordering"
* tag 'x86_urgent_for_v5.11' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/pci: Create PCI/MSI irqdomain after x86_init.pci.arch_init()
x86/build: Disable CET instrumentation in the kernel for 32-bit too
x86/sgx: Maintain encl->refcount for each encl->mm_list entry
pcpu_build_alloc_info() is an __init function that makes a call to
cpumask_clear_cpu(). With CONFIG_GCOV_PROFILE_ALL enabled, the inline
heuristics are modified and such cpumask_clear_cpu() which is marked
inline doesn't get inlined. Because it works on mask in __initdata,
modpost throws a section mismatch error.
Arnd sent a patch with the flatten attribute as an alternative [2]. I've
added it to compiler_attributes.h.
modpost complaint:
WARNING: modpost: vmlinux.o(.text+0x735425): Section mismatch in reference from the function cpumask_clear_cpu() to the variable .init.data:pcpu_build_alloc_info.mask
The function cpumask_clear_cpu() references
the variable __initdata pcpu_build_alloc_info.mask.
This is often because cpumask_clear_cpu lacks a __initdata
annotation or the annotation of pcpu_build_alloc_info.mask is wrong.
clang output:
mm/percpu.c:2724:5: remark: cpumask_clear_cpu not inlined into pcpu_build_alloc_info because too costly to inline (cost=725, threshold=325) [-Rpass-missed=inline]
[1] https://lore.kernel.org/linux-mm/202012220454.9F6Bkz9q-lkp@intel.com/
[2] https://lore.kernel.org/lkml/CAK8P3a2ZWfNeXKSm8K_SUhhwkor17jFo3xApLXjzfPqX0eUDUA@mail.gmail.com/
Reported-by: kernel test robot <lkp@intel.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
To build group_map[] and group_cnt[], we find out which group
CPUs belong to by comparing the distance of the cpu. However,
this includes cases where comparisons are not required.
This patch uses a bitmap to record CPUs that is not classified in
the group. CPUs that we know which group they belong to should be
cleared from the bitmap. In result, we can reduce the number of
unnecessary comparisons.
Signed-off-by: Wonhyuk Yang <vvghjk1234@gmail.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
[Dennis: added cpumask_clear() call and #include cpumask.h.]
The leds-rt8515 driver can optionall use the v4l2 flash led class,
but it causes a link error when that class is in a loadable module
and the rt8515 driver itself is built-in:
ld.lld: error: undefined symbol: v4l2_flash_init
>>> referenced by leds-rt8515.c
>>> leds/flash/leds-rt8515.o:(rt8515_probe) in archive
drivers/built-in.a
Adding 'depends on V4L2_FLASH_LED_CLASS' in Kconfig would avoid that,
but it would make it impossible to use the driver without the
v4l2 support.
Add the same dependency that the other users of this class have
instead, which just prevents the broken configuration.
Fixes: e1c6edcbea ("leds: rt8515: Add Richtek RT8515 LED driver")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Pavel Machek <pavel@ucw.cz>
These are NOT exported to userspace.
The headers listed in arch/sparc/include/uapi/asm/Kbuild are exported.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
BOSS GP-10 with 0582:0185 requires the similar quirk to make the
implicit feedback working like other BOSS devices.
Reported-by: Keith Milner <kamilner@superlative.org>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/20210214154251.10750-1-tiwai@suse.de
Signed-off-by: Takashi Iwai <tiwai@suse.de>
Merely enabling CONFIG_COMPILE_TEST should not enable additional code.
To fix this, restrict the automatic enabling of IMX_INTMUX to ARCH_MXC,
and ask the user in case of compile-testing.
Fixes: 66968d7dfc ("irqchip: Add COMPILE_TEST support for IMX_INTMUX")
Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20210208145605.422943-1-geert+renesas@glider.be
Alexander Lobakin says:
====================
skbuff: introduce skbuff_heads bulking and reusing
Currently, all sorts of skb allocation always do allocate
skbuff_heads one by one via kmem_cache_alloc().
On the other hand, we have percpu napi_alloc_cache to store
skbuff_heads queued up for freeing and flush them by bulks.
We can use this cache not only for bulk-wiping, but also to obtain
heads for new skbs and avoid unconditional allocations, as well as
for bulk-allocating (like XDP's cpumap code and veth driver already
do).
As this might affect latencies, cache pressure and lots of hardware
and driver-dependent stuff, this new feature is mostly optional and
can be issued via:
- a new napi_build_skb() function (as a replacement for build_skb());
- existing {,__}napi_alloc_skb() and napi_get_frags() functions;
- __alloc_skb() with passing SKB_ALLOC_NAPI in flags.
iperf3 showed 35-70 Mbps bumps for both TCP and UDP while performing
VLAN NAT on 1.2 GHz MIPS board. The boost is likely to be bigger
on more powerful hosts and NICs with tens of Mpps.
Note on skbuff_heads from distant slabs or pfmemalloc'ed slabs:
- kmalloc()/kmem_cache_alloc() itself allows by default allocating
memory from the remote nodes to defragment their slabs. This is
controlled by sysctl, but according to this, skbuff_head from a
remote node is an OK case;
- The easiest way to check if the slab of skbuff_head is remote or
pfmemalloc'ed is:
if (!dev_page_is_reusable(virt_to_head_page(skb)))
/* drop it */;
...*but*, regarding that most slabs are built of compound pages,
virt_to_head_page() will hit unlikely-branch every single call.
This check costed at least 20 Mbps in test scenarios and seems
like it'd be better to _not_ do this.
Since v5 [4]:
- revert flags-to-bool conversion and simplify flags testing in
__alloc_skb() (Alexander Duyck).
Since v4 [3]:
- rebase on top of net-next and address kernel build robot issue;
- reorder checks a bit in __alloc_skb() to make new condition even
more harmless.
Since v3 [2]:
- make the feature mostly optional, so driver developers could
decide whether to use it or not (Paolo Abeni).
This reuses the old flag for __alloc_skb() and introduces
a new napi_build_skb();
- reduce bulk-allocation size from 32 to 16 elements (also Paolo).
This equals to the value of XDP's devmap and veth batch processing
(which were tested a lot) and should be sane enough;
- don't waste cycles on explicit in_serving_softirq() check.
Since v2 [1]:
- also cover {,__}alloc_skb() and {,__}build_skb() cases (became handy
after the changes that pass tiny skbs requests to kmalloc layer);
- cover the cache with KASAN instrumentation (suggested by Eric
Dumazet, help of Dmitry Vyukov);
- completely drop redundant __kfree_skb_flush() (also Eric);
- lots of code cleanups;
- expand the commit message with NUMA and pfmemalloc points (Jakub).
Since v1 [0]:
- use one unified cache instead of two separate to greatly simplify
the logics and reduce hotpath overhead (Edward Cree);
- new: recycle also GRO_MERGED_FREE skbs instead of immediate
freeing;
- correct performance numbers after optimizations and performing
lots of tests for different use cases.
[0] https://lore.kernel.org/netdev/20210111182655.12159-1-alobakin@pm.me
[1] https://lore.kernel.org/netdev/20210113133523.39205-1-alobakin@pm.me
[2] https://lore.kernel.org/netdev/20210209204533.327360-1-alobakin@pm.me
[3] https://lore.kernel.org/netdev/20210210162732.80467-1-alobakin@pm.me
[4] https://lore.kernel.org/netdev/20210211185220.9753-1-alobakin@pm.me
====================
Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
napi_frags_finish() and napi_skb_finish() can only be called inside
NAPI Rx context, so we can feed NAPI cache with skbuff_heads that
got NAPI_MERGED_FREE verdict instead of immediate freeing.
Replace __kfree_skb() with __kfree_skb_defer() in napi_skb_finish()
and move napi_skb_free_stolen_head() to skbuff.c, so it can drop skbs
to NAPI cache.
As many drivers call napi_alloc_skb()/napi_get_frags() on their
receive path, this becomes especially useful.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
{,__}napi_alloc_skb() is mostly used either for optional non-linear
receive methods (usually controlled via Ethtool private flags and off
by default) and/or for Rx copybreaks.
Use __napi_build_skb() here for obtaining skbuff_heads from NAPI cache
instead of inplace allocations. This includes both kmalloc and page
frag paths.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
Reuse the old and forgotten SKB_ALLOC_NAPI to add an option to get
an skbuff_head from the NAPI cache instead of inplace allocation
inside __alloc_skb().
This implies that the function is called from softirq or BH-off
context, not for allocating a clone or from a distant node.
Cc: Alexander Duyck <alexander.duyck@gmail.com> # Simplified flags check
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
NAPI cache structures will be used for allocating skbuff_heads,
so move their declarations a bit upper.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
This function isn't much needed as NAPI skb queue gets bulk-freed
anyway when there's no more room, and even may reduce the efficiency
of bulk operations.
It will be even less needed after reusing skb cache on allocation path,
so remove it and this way lighten network softirqs a bit.
Suggested-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
Just call __build_skb_around() instead of open-coding it.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
Use unlikely() annotations for skbuff_head and data similarly to the
two other allocation functions and remove totally redundant goto.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
__build_skb_around() can never fail and always returns passed skb.
Make it return void to simplify and optimize the code.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
Eversince the introduction of __kmalloc_reserve(), "ip" argument
hasn't been used. _RET_IP_ is embedded inside
kmalloc_node_track_caller().
Remove the redundant macro and rename the function after it.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
In preparation before reusing several functions in all three skb
allocation variants, move __alloc_skb() next to the
__netdev_alloc_skb() and __napi_alloc_skb().
No functional changes.
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
