Both callers of __delete_from_swap_cache have the swp_entry_t already,
so pass that in to make constructing the XA_STATE easier.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Combine __add_to_swap_cache and add_to_swap_cache into one function
since there is no more need to preload.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This is essentially xa_cmpxchg() with the locking handled above us,
and it doesn't have to handle replacing a NULL entry.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
We construct an XA_STATE and use it to delete the node with
xas_store() rather than adding a special function for this unique
use case. Includes a test that simulates this usage for the
test suite.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Instead of calling find_get_pages_range() and putting any reference,
use xas_find() to iterate over any entries in the range, skipping the
shadow/swap entries.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Now the page cache lookup is using the XArray, let's convert this
regression test from the radix tree API to the XArray so it's testing
roughly the same thing it was testing before.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Slight change of strategy here; if we have trouble getting hold of a
page for whatever reason (eg a compound page is split underneath us),
don't spin to stabilise the page, just continue the iteration, like we
would if we failed to trylock the page. Since this is a speculative
optimisation, it feels like we should allow the process to take an extra
fault if it turns out to need this page instead of spending time to pin
down a page it may not need.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The 'end' parameter of the xas_for_each iterator avoids a useless
iteration at the end of the range.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
There's no direct replacement for radix_tree_for_each_contig()
in the XArray API as it's an unusual thing to do. Instead,
open-code a loop using xas_next(). This removes the only user of
radix_tree_for_each_contig() so delete the iterator from the API and
the test suite code for it.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The 'end' parameter of the xas_for_each iterator avoids a useless
iteration at the end of the range.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Use the XArray APIs to add and replace pages in the page cache. This
removes two uses of the radix tree preload API and is significantly
shorter code. It also removes the last user of __radix_tree_create()
outside radix-tree.c itself, so make it static.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The page cache offers the ability to search for a miss in the previous or
next N locations. Rather than teach the XArray about the page cache's
definition of a miss, use xas_prev() and xas_next() to search the page
array. This should be more efficient as it does not have to start the
lookup from the top for each index.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Change i_pages from a radix_tree_root to an xarray, convert the
documentation into kernel-doc format and change the order of the elements
to pack them better on 64-bit systems.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Use the XA_TRACK_FREE ability to track which entries have a free bit,
similarly to how it uses the radix tree's IDR_FREE tag. This eliminates
the per-cpu ida_bitmap preload, and fixes the memory consumption
regression I introduced when making the IDR able to store any pointer.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Add the optional ability to track which entries in an XArray are free
and provide xa_alloc() to replace most of the functionality of the IDR.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This function reserves a slot in the XArray for users which need
to acquire multiple locks before storing their entry in the tree and
so cannot use a plain xa_store().
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This hopefully temporary function is useful for users who have not yet
been converted to multi-index entries.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This iterator iterates over each entry that is stored in the index or
indices specified by the xa_state. This is intended for use for a
conditional store of a multiindex entry, or to allow entries which are
about to be removed from the xarray to be disposed of properly.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xas_next and xas_prev functions move the xas index by one position,
and adjust the rest of the iterator state to match it. This is more
efficient than calling xas_set() as it keeps the iterator at the leaves
of the tree instead of walking the iterator from the root each time.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This function frees all the internal memory allocated to the xarray
and reinitialises it to be empty.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xa_extract function combines the functionality of
radix_tree_gang_lookup() and radix_tree_gang_lookup_tagged().
It extracts entries matching the specified filter into a normal array.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xa_for_each iterator allows the user to efficiently walk a range
of the array, executing the loop body once for each entry in that
range that matches the filter. This commit also includes xa_find()
and xa_find_after() which are helper functions for xa_for_each() but
may also be useful in their own right.
In the xas family of functions, we have xas_for_each(), xas_find(),
xas_next_entry(), xas_for_each_tagged(), xas_find_tagged(),
xas_next_tagged() and xas_pause().
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Like cmpxchg(), xa_cmpxchg will only store to the index if the current
entry matches the old entry. It returns the current entry, which is
usually more useful than the errno returned by radix_tree_insert().
For the users who really only want the errno, the xa_insert() wrapper
provides a more convenient calling convention.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
xa_store() differs from radix_tree_insert() in that it will overwrite an
existing element in the array rather than returning an error. This is
the behaviour which most users want, and those that want more complex
behaviour generally want to use the xas family of routines anyway.
For memory allocation, xa_store() will first attempt to request memory
from the slab allocator; if memory is not immediately available, it will
drop the xa_lock and allocate memory, keeping a pointer in the xa_state.
It does not use the per-CPU cache, although those will continue to exist
until all radix tree users are converted to the xarray.
This patch also includes xa_erase() and __xa_erase() for a streamlined
way to store NULL. Since there is no need to allocate memory in order
to store a NULL in the XArray, we do not need to trouble the user with
deciding what memory allocation flags to use.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
XArray marks are like the radix tree tags, only slightly more strongly
typed. They are renamed in order to distinguish them from tagged
pointers. This commit adds the basic get/set/clear operations.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The xa_load function brings with it a lot of infrastructure; xa_empty(),
xa_is_err(), and large chunks of the XArray advanced API that are used
to implement xa_load.
As the test-suite demonstrates, it is possible to use the XArray functions
on a radix tree. The radix tree functions depend on the GFP flags being
stored in the root of the tree, so it's not possible to use the radix
tree functions on an XArray.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
This is documentation on how to use the XArray, not details about its
internal implementation.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Acked-by: Josef Bacik <jbacik@fb.com>
This is a direct replacement for struct radix_tree_node. A couple of
struct members have changed name, so convert those. Use a #define so
that radix tree users continue to work without change.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
This is a direct replacement for struct radix_tree_root. Some of the
struct members have changed name; convert those, and use a #define so
that radix_tree users continue to work without change.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
Instead of storing a pointer to the slot containing the canonical entry,
store the offset of the slot. Produces slightly more efficient code
(~300 bytes) and simplifies the implementation.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
Introduce xarray value entries and tagged pointers to replace radix
tree exceptional entries. This is a slight change in encoding to allow
the use of an extra bit (we can now store BITS_PER_LONG - 1 bits in a
value entry). It is also a change in emphasis; exceptional entries are
intimidating and different. As the comment explains, you can choose
to store values or pointers in the xarray and they are both first-class
citizens.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Josef Bacik <jbacik@fb.com>
An upcoming change to the encoding of internal entries will set the bottom
two bits to 0b10. Unfortunately, m68k only aligns some data structures
to 2 bytes, so the IDR will interpret them as internal entries and things
will go badly wrong.
Change the radix tree so that it stops either when the node indicates
that it's the bottom of the tree (shift == 0) or when the entry is not an
internal entry. This means we cannot insert an arbitrary kernel pointer
as a multiorder entry, but the IDR does not permit multiorder entries.
Annoyingly, this means the IDR can no longer take advantage of the radix
tree's ability to store a single entry at offset 0 without allocating
memory. A pointer which is 2-byte aligned cannot be stored directly in
the root as it would be indistinguishable from a node, so we must allocate
a node in order to store a 2-byte pointer at index 0. The idr_replace()
function does not take a GFP flags argument, so cannot allocate memory.
If a user inserts a 4-byte aligned pointer at index 0 and then replaces
it with a 2-byte aligned pointer, we must be able to store it.
Arbitrary pointer values are still not permitted; pointers of the
form 2 + (i * 4) for values of i between 0 and 1023 are reserved for
the implementation. These are not valid kernel pointers as they would
point into the zero page.
This change does cause a runtime memory consumption regression for
the IDA. I will recover that later.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
Tested-by: Guenter Roeck <linux@roeck-us.net>
-----BEGIN PGP SIGNATURE-----
iQJEBAABCAAuFiEEwPw5LcreJtl1+l5K99NY+ylx4KYFAluv6bgQHGF4Ym9lQGtl
cm5lbC5kawAKCRD301j7KXHgprAFD/9YEm5/YGX8ypqepUISfr7MSspZCl/r0GVv
Oyu14jfTe18ji9mnFzA+g0y/KbNgdyOYstOp9l4V2Pxmt0Hq3KHjiqdbGbICUcDz
uPbwnXWW3Pem6l6JOmT86n14c5irkmB/XezlBEE2n7cVReCOwkycr8VNUkXax/Mu
Wuv1nAX2+uNBGSg0g4H2Y5Dk0fxmQcyKKkVfsz1xa9T2G5sB7gi8XU3+mqXT77hC
BN7aaB306g+gNwGuHp4V6r9eUmSilRHq53qYTKRD8Vtbe2VeVlsmnU8LjFGRuxcN
UZOuEO5MftIt32epi8hEQwWVxoZlaHv5qTjAHjiM77H7+kZGVK7Xv/ZrJHoRRQcI
vIrNKZX0wUtlsC/MmdCcYdqzxgyMJYNc7+Y13W2M/GgXamrOVcjaYWGaxDWfLlIN
jLkFrBK+9XRnvh5o0yKmoL/LXFJ4vXc3T9cvaYN/KTJUhYcfBEDuvfJTPKjbrWkc
iv6ORaLh9hbtUmIJO2yo0ZtLo9vxhegJK1NP6bICo0fJ9iiOrIQpxLiEegWA0ITb
85ot2Iepao5wqNnobSUBdlIKgIt1hECaNVCb3wUvIM7KYZilYP5nYsHg28x9+ZPq
CJZRzYmMBaH+yKo9FkO/+rxGPg6R9Gri5QZRTFAVUcW02I407A8+nlMwQMEBZxqb
80h8OO/ACw==
=8c0k
-----END PGP SIGNATURE-----
Merge tag 'for-linus-20180929' of git://git.kernel.dk/linux-block
Jens writes:
"Block fixes for 4.19-rc6
A set of fixes that should go into this release. This pull request
contains:
- A fix (hopefully) for the persistent grants for xen-blkfront. A
previous fix from this series wasn't complete, hence reverted, and
this one should hopefully be it. (Boris Ostrovsky)
- Fix for an elevator drain warning with SMR devices, which is
triggered when you switch schedulers (Damien)
- bcache deadlock fix (Guoju Fang)
- Fix for the block unplug tracepoint, which has had the
timer/explicit flag reverted since 4.11 (Ilya)
- Fix a regression in this series where the blk-mq timeout hook is
invoked with the RCU read lock held, hence preventing it from
blocking (Keith)
- NVMe pull from Christoph, with a single multipath fix (Susobhan Dey)"
* tag 'for-linus-20180929' of git://git.kernel.dk/linux-block:
xen/blkfront: correct purging of persistent grants
Revert "xen/blkfront: When purging persistent grants, keep them in the buffer"
blk-mq: I/O and timer unplugs are inverted in blktrace
bcache: add separate workqueue for journal_write to avoid deadlock
xen/blkfront: When purging persistent grants, keep them in the buffer
block: fix deadline elevator drain for zoned block devices
blk-mq: Allow blocking queue tag iter callbacks
nvme: properly propagate errors in nvme_mpath_init
Thomas writes:
"A single fix for the AMD memory encryption boot code so it does not
read random garbage instead of the cached encryption bit when a kexec
kernel is allocated above the 32bit address limit."
* 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/boot: Fix kexec booting failure in the SEV bit detection code
Thomas writes:
"Three small fixes for clocksource drivers:
- Proper error handling in the Atmel PIT driver
- Add CLOCK_SOURCE_SUSPEND_NONSTOP for TI SoCs so suspend works again
- Fix the next event function for Facebook Backpack-CMM BMC chips so
usleep(100) doesnt sleep several milliseconds"
* 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
clocksource/drivers/timer-atmel-pit: Properly handle error cases
clocksource/drivers/fttmr010: Fix set_next_event handler
clocksource/drivers/ti-32k: Add CLOCK_SOURCE_SUSPEND_NONSTOP flag for non-am43 SoCs
Thomas writes:
"A single fix for a missing sanity check when a pinned event is tried
to be read on the wrong CPU due to a legit event scheduling failure."
* 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
perf/core: Add sanity check to deal with pinned event failure
There is currently a warning when building the Kryo cpufreq driver into
the kernel image:
WARNING: vmlinux.o(.text+0x8aa424): Section mismatch in reference from
the function qcom_cpufreq_kryo_probe() to the function
.init.text:qcom_cpufreq_kryo_get_msm_id()
The function qcom_cpufreq_kryo_probe() references
the function __init qcom_cpufreq_kryo_get_msm_id().
This is often because qcom_cpufreq_kryo_probe lacks a __init
annotation or the annotation of qcom_cpufreq_kryo_get_msm_id is wrong.
Remove the '__init' annotation from qcom_cpufreq_kryo_get_msm_id
so that there is no more mismatch warning.
Additionally, Nick noticed that the remove function was marked as
'__init' when it should really be marked as '__exit'.
Fixes: 46e2856b8e (cpufreq: Add Kryo CPU scaling driver)
Fixes: 5ad7346b4a (cpufreq: kryo: Add module remove and exit)
Reported-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Cc: 4.18+ <stable@vger.kernel.org> # 4.18+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
-----BEGIN PGP SIGNATURE-----
iQI/BAABCgApFiEEgdbnc3r/njty3Iq9D55TZVIEUYMFAluuzLULHGhjaEBsc3Qu
ZGUACgkQD55TZVIEUYNQgA/+Ilbv0dBfBoQkdyR86f0BC2TTkaBTsMAmQiWGqnsn
q0exvy9NLDjbTu5k6hbZUvziMJGsF1MZtvyHLGiQT2q2m2BLadqxlGCe+tJ/09kS
CKus4PnElfJcnINW2Y4W/c0khzW0/ISiNJwigJbRwLx7h05SlOiMI51n1HU4K5oa
4fV90lrV+9CybtH5NNH3j2Ajsdf5LNN22bhQjYvqqHxcLDbXfp6HAhIBZUqPDXDM
8u7e+rnS+0cmvchF4JUxVtcKTXJgELTvJ7aF4Yvr8ZxYYbv259oVnvmD0syH4uiS
MP22eVCS3CPFsQcPWizL8s676kiM/EgM2E2c7Rm0Wd05EKAOTwUMFkQZGCgTIW7I
mbmCyMVJbSpUegMtD4eNn3bdJiyFgydo2VsWbd9yXznTDWhGHw8KIazRQS4FYopc
KxSkQneTSCPCtuFRMM8EpZUurPxQtGgudoAhh6KOnU7HFLodz+8dwWaXK6tfbep7
YrL968tdc8jJEAlu5eeMWgSECy/JPufBOelCNUCBufS5dSaO3F82Du7hlizqlhNr
vlAlAt2gmFyEfeeilz98hXuWSFqt1Jpg4I+vMS/ITJnl8GD4fDgK3vAZFc7FUZ2V
SV/+pvok9SKnCrwVGwPF2lc6NLswP7Xjf7GwOFv77gimNieLQ2P8OHsknw8VbJu6
G+I=
=QKIn
-----END PGP SIGNATURE-----
Merge tag 'dma-mapping-4.19-3' of git://git.infradead.org/users/hch/dma-mapping
Christoph writes:
"dma mapping fix for 4.19-rc6
fix a missing Kconfig symbol for commits introduced in 4.19-rc"
* tag 'dma-mapping-4.19-3' of git://git.infradead.org/users/hch/dma-mapping:
dma-mapping: add the missing ARCH_HAS_SYNC_DMA_FOR_CPU_ALL declaration
Dmitry writes:
"Input updates for v4.19-rc5
Just a few driver fixes"
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input:
Input: uinput - allow for max == min during input_absinfo validation
Input: elantech - enable middle button of touchpad on ThinkPad P72
Input: atakbd - fix Atari CapsLock behaviour
Input: atakbd - fix Atari keymap
Input: egalax_ts - add system wakeup support
Input: gpio-keys - fix a documentation index issue
Quite a few fixes for the Renesas drivers in here, plus a fix for the
Tegra driver and some documentation fixes for the recently added spi-mem
code. The Tegra fix is relatively large but fairly straightforward and
mechanical, it runs on probe so it's been reasonably well covered in
-next testing.
-----BEGIN PGP SIGNATURE-----
iQFHBAABCgAxFiEEreZoqmdXGLWf4p/qJNaLcl1Uh9AFAluuOnETHGJyb29uaWVA
a2VybmVsLm9yZwAKCRAk1otyXVSH0AhkB/9ahKdmxdxu0uuzzvS5qmdGD+98dTaV
M+7+Bq6CJfBz0YTc9dxTiynbJot15vo5mKDZkDaJrgOoLCWWDGWa/DjRif2qS2o9
QxjWlBPzfOhKb81/HMJ+MaIHYDtAxvCu4iZExUZPxZzypX8Q9U3+9a/u0+JnRug2
guQ/4vYdxTGbR/taFsKLzf28DRqQb74KuDRAHaN44UYBxA1begy5gFOc7cEQTRu0
WACGM8jNL2eWWLXDNnQQTfb2t1v7LbbMorazyCn2e5ujamE4+DSaYWJF5axZiaQN
ZcRVATw0SPiQlsQW/G014SSFJT0waybxWLOynWS9MSgjx84/GRYWvxRv
=3ATM
-----END PGP SIGNATURE-----
Merge tag 'spi-fix-v4.19-rc5' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
Mark writes:
"spi: Fixes for v4.19
Quite a few fixes for the Renesas drivers in here, plus a fix for the
Tegra driver and some documentation fixes for the recently added
spi-mem code. The Tegra fix is relatively large but fairly
straightforward and mechanical, it runs on probe so it's been
reasonably well covered in -next testing."
* tag 'spi-fix-v4.19-rc5' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi:
spi: spi-mem: Move the DMA-able constraint doc to the kerneldoc header
spi: spi-mem: Add missing description for data.nbytes field
spi: rspi: Fix interrupted DMA transfers
spi: rspi: Fix invalid SPI use during system suspend
spi: sh-msiof: Fix handling of write value for SISTR register
spi: sh-msiof: Fix invalid SPI use during system suspend
spi: gpio: Fix copy-and-paste error
spi: tegra20-slink: explicitly enable/disable clock
