2006-01-08 09:01:45 +00:00
|
|
|
/*
|
|
|
|
* SLOB Allocator: Simple List Of Blocks
|
|
|
|
*
|
|
|
|
* Matt Mackall <mpm@selenic.com> 12/30/03
|
|
|
|
*
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
* NUMA support by Paul Mundt, 2007.
|
|
|
|
*
|
2006-01-08 09:01:45 +00:00
|
|
|
* How SLOB works:
|
|
|
|
*
|
|
|
|
* The core of SLOB is a traditional K&R style heap allocator, with
|
|
|
|
* support for returning aligned objects. The granularity of this
|
2007-07-16 06:38:09 +00:00
|
|
|
* allocator is as little as 2 bytes, however typically most architectures
|
|
|
|
* will require 4 bytes on 32-bit and 8 bytes on 64-bit.
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
*
|
2008-02-05 06:29:37 +00:00
|
|
|
* The slob heap is a set of linked list of pages from alloc_pages(),
|
|
|
|
* and within each page, there is a singly-linked list of free blocks
|
|
|
|
* (slob_t). The heap is grown on demand. To reduce fragmentation,
|
|
|
|
* heap pages are segregated into three lists, with objects less than
|
|
|
|
* 256 bytes, objects less than 1024 bytes, and all other objects.
|
|
|
|
*
|
|
|
|
* Allocation from heap involves first searching for a page with
|
|
|
|
* sufficient free blocks (using a next-fit-like approach) followed by
|
|
|
|
* a first-fit scan of the page. Deallocation inserts objects back
|
|
|
|
* into the free list in address order, so this is effectively an
|
|
|
|
* address-ordered first fit.
|
2006-01-08 09:01:45 +00:00
|
|
|
*
|
|
|
|
* Above this is an implementation of kmalloc/kfree. Blocks returned
|
2007-07-16 06:38:09 +00:00
|
|
|
* from kmalloc are prepended with a 4-byte header with the kmalloc size.
|
2006-01-08 09:01:45 +00:00
|
|
|
* If kmalloc is asked for objects of PAGE_SIZE or larger, it calls
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
* alloc_pages() directly, allocating compound pages so the page order
|
2012-10-19 12:33:10 +00:00
|
|
|
* does not have to be separately tracked.
|
|
|
|
* These objects are detected in kfree() because PageSlab()
|
2007-07-16 06:38:08 +00:00
|
|
|
* is false for them.
|
2006-01-08 09:01:45 +00:00
|
|
|
*
|
|
|
|
* SLAB is emulated on top of SLOB by simply calling constructors and
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
* destructors for every SLAB allocation. Objects are returned with the
|
|
|
|
* 4-byte alignment unless the SLAB_HWCACHE_ALIGN flag is set, in which
|
|
|
|
* case the low-level allocator will fragment blocks to create the proper
|
|
|
|
* alignment. Again, objects of page-size or greater are allocated by
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
* calling alloc_pages(). As SLAB objects know their size, no separate
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
* size bookkeeping is necessary and there is essentially no allocation
|
2007-07-16 06:38:08 +00:00
|
|
|
* space overhead, and compound pages aren't needed for multi-page
|
|
|
|
* allocations.
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
*
|
|
|
|
* NUMA support in SLOB is fairly simplistic, pushing most of the real
|
|
|
|
* logic down to the page allocator, and simply doing the node accounting
|
|
|
|
* on the upper levels. In the event that a node id is explicitly
|
2009-06-16 22:31:54 +00:00
|
|
|
* provided, alloc_pages_exact_node() with the specified node id is used
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
* instead. The common case (or when the node id isn't explicitly provided)
|
|
|
|
* will default to the current node, as per numa_node_id().
|
|
|
|
*
|
|
|
|
* Node aware pages are still inserted in to the global freelist, and
|
|
|
|
* these are scanned for by matching against the node id encoded in the
|
|
|
|
* page flags. As a result, block allocations that can be satisfied from
|
|
|
|
* the freelist will only be done so on pages residing on the same node,
|
|
|
|
* in order to prevent random node placement.
|
2006-01-08 09:01:45 +00:00
|
|
|
*/
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
#include <linux/kernel.h>
|
2006-01-08 09:01:45 +00:00
|
|
|
#include <linux/slab.h>
|
2012-07-06 20:25:11 +00:00
|
|
|
|
2006-01-08 09:01:45 +00:00
|
|
|
#include <linux/mm.h>
|
2009-05-05 09:13:45 +00:00
|
|
|
#include <linux/swap.h> /* struct reclaim_state */
|
2006-01-08 09:01:45 +00:00
|
|
|
#include <linux/cache.h>
|
|
|
|
#include <linux/init.h>
|
2011-10-16 06:01:52 +00:00
|
|
|
#include <linux/export.h>
|
2007-05-17 05:10:49 +00:00
|
|
|
#include <linux/rcupdate.h>
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
#include <linux/list.h>
|
2009-06-11 12:23:17 +00:00
|
|
|
#include <linux/kmemleak.h>
|
2010-05-26 09:22:17 +00:00
|
|
|
|
|
|
|
#include <trace/events/kmem.h>
|
|
|
|
|
2011-07-26 23:09:06 +00:00
|
|
|
#include <linux/atomic.h>
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
|
2012-12-18 22:22:46 +00:00
|
|
|
#include "slab.h"
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* slob_block has a field 'units', which indicates size of block if +ve,
|
|
|
|
* or offset of next block if -ve (in SLOB_UNITs).
|
|
|
|
*
|
|
|
|
* Free blocks of size 1 unit simply contain the offset of the next block.
|
|
|
|
* Those with larger size contain their size in the first SLOB_UNIT of
|
|
|
|
* memory, and the offset of the next free block in the second SLOB_UNIT.
|
|
|
|
*/
|
2007-07-16 06:38:09 +00:00
|
|
|
#if PAGE_SIZE <= (32767 * 2)
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
typedef s16 slobidx_t;
|
|
|
|
#else
|
|
|
|
typedef s32 slobidx_t;
|
|
|
|
#endif
|
|
|
|
|
2006-01-08 09:01:45 +00:00
|
|
|
struct slob_block {
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
slobidx_t units;
|
2007-07-16 06:38:09 +00:00
|
|
|
};
|
2006-01-08 09:01:45 +00:00
|
|
|
typedef struct slob_block slob_t;
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
2008-02-05 06:29:37 +00:00
|
|
|
* All partially free slob pages go on these lists.
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
*/
|
2008-02-05 06:29:37 +00:00
|
|
|
#define SLOB_BREAK1 256
|
|
|
|
#define SLOB_BREAK2 1024
|
|
|
|
static LIST_HEAD(free_slob_small);
|
|
|
|
static LIST_HEAD(free_slob_medium);
|
|
|
|
static LIST_HEAD(free_slob_large);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* slob_page_free: true for pages on free_slob_pages list.
|
|
|
|
*/
|
2012-06-13 15:24:52 +00:00
|
|
|
static inline int slob_page_free(struct page *sp)
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
{
|
2012-06-13 15:24:52 +00:00
|
|
|
return PageSlobFree(sp);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
}
|
|
|
|
|
2012-06-13 15:24:52 +00:00
|
|
|
static void set_slob_page_free(struct page *sp, struct list_head *list)
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
{
|
2014-04-08 20:44:27 +00:00
|
|
|
list_add(&sp->lru, list);
|
2012-06-13 15:24:52 +00:00
|
|
|
__SetPageSlobFree(sp);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
}
|
|
|
|
|
2012-06-13 15:24:52 +00:00
|
|
|
static inline void clear_slob_page_free(struct page *sp)
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
{
|
2014-04-08 20:44:27 +00:00
|
|
|
list_del(&sp->lru);
|
2012-06-13 15:24:52 +00:00
|
|
|
__ClearPageSlobFree(sp);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
}
|
|
|
|
|
2006-01-08 09:01:45 +00:00
|
|
|
#define SLOB_UNIT sizeof(slob_t)
|
2012-12-20 19:11:39 +00:00
|
|
|
#define SLOB_UNITS(size) DIV_ROUND_UP(size, SLOB_UNIT)
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2007-05-17 05:10:49 +00:00
|
|
|
/*
|
|
|
|
* struct slob_rcu is inserted at the tail of allocated slob blocks, which
|
|
|
|
* were created with a SLAB_DESTROY_BY_RCU slab. slob_rcu is used to free
|
|
|
|
* the block using call_rcu.
|
|
|
|
*/
|
|
|
|
struct slob_rcu {
|
|
|
|
struct rcu_head head;
|
|
|
|
int size;
|
|
|
|
};
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* slob_lock protects all slob allocator structures.
|
|
|
|
*/
|
2006-01-08 09:01:45 +00:00
|
|
|
static DEFINE_SPINLOCK(slob_lock);
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* Encode the given size and next info into a free slob block s.
|
|
|
|
*/
|
|
|
|
static void set_slob(slob_t *s, slobidx_t size, slob_t *next)
|
|
|
|
{
|
|
|
|
slob_t *base = (slob_t *)((unsigned long)s & PAGE_MASK);
|
|
|
|
slobidx_t offset = next - base;
|
2006-12-30 00:48:28 +00:00
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
if (size > 1) {
|
|
|
|
s[0].units = size;
|
|
|
|
s[1].units = offset;
|
|
|
|
} else
|
|
|
|
s[0].units = -offset;
|
|
|
|
}
|
2006-01-08 09:01:45 +00:00
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* Return the size of a slob block.
|
|
|
|
*/
|
|
|
|
static slobidx_t slob_units(slob_t *s)
|
|
|
|
{
|
|
|
|
if (s->units > 0)
|
|
|
|
return s->units;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return the next free slob block pointer after this one.
|
|
|
|
*/
|
|
|
|
static slob_t *slob_next(slob_t *s)
|
|
|
|
{
|
|
|
|
slob_t *base = (slob_t *)((unsigned long)s & PAGE_MASK);
|
|
|
|
slobidx_t next;
|
|
|
|
|
|
|
|
if (s[0].units < 0)
|
|
|
|
next = -s[0].units;
|
|
|
|
else
|
|
|
|
next = s[1].units;
|
|
|
|
return base+next;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Returns true if s is the last free block in its page.
|
|
|
|
*/
|
|
|
|
static int slob_last(slob_t *s)
|
|
|
|
{
|
|
|
|
return !((unsigned long)slob_next(s) & ~PAGE_MASK);
|
|
|
|
}
|
|
|
|
|
2009-01-18 18:00:38 +00:00
|
|
|
static void *slob_new_pages(gfp_t gfp, int order, int node)
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
{
|
|
|
|
void *page;
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
2012-09-08 20:47:51 +00:00
|
|
|
if (node != NUMA_NO_NODE)
|
2009-06-16 22:31:54 +00:00
|
|
|
page = alloc_pages_exact_node(node, gfp, order);
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
else
|
|
|
|
#endif
|
|
|
|
page = alloc_pages(gfp, order);
|
|
|
|
|
|
|
|
if (!page)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
return page_address(page);
|
|
|
|
}
|
|
|
|
|
2009-01-18 18:00:38 +00:00
|
|
|
static void slob_free_pages(void *b, int order)
|
|
|
|
{
|
2009-05-05 09:13:45 +00:00
|
|
|
if (current->reclaim_state)
|
|
|
|
current->reclaim_state->reclaimed_slab += 1 << order;
|
2009-01-18 18:00:38 +00:00
|
|
|
free_pages((unsigned long)b, order);
|
|
|
|
}
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* Allocate a slob block within a given slob_page sp.
|
|
|
|
*/
|
2012-06-13 15:24:52 +00:00
|
|
|
static void *slob_page_alloc(struct page *sp, size_t size, int align)
|
2006-01-08 09:01:45 +00:00
|
|
|
{
|
2009-01-18 18:00:38 +00:00
|
|
|
slob_t *prev, *cur, *aligned = NULL;
|
2006-01-08 09:01:45 +00:00
|
|
|
int delta = 0, units = SLOB_UNITS(size);
|
|
|
|
|
2012-06-13 15:24:52 +00:00
|
|
|
for (prev = NULL, cur = sp->freelist; ; prev = cur, cur = slob_next(cur)) {
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
slobidx_t avail = slob_units(cur);
|
|
|
|
|
2006-01-08 09:01:45 +00:00
|
|
|
if (align) {
|
|
|
|
aligned = (slob_t *)ALIGN((unsigned long)cur, align);
|
|
|
|
delta = aligned - cur;
|
|
|
|
}
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
if (avail >= units + delta) { /* room enough? */
|
|
|
|
slob_t *next;
|
|
|
|
|
2006-01-08 09:01:45 +00:00
|
|
|
if (delta) { /* need to fragment head to align? */
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
next = slob_next(cur);
|
|
|
|
set_slob(aligned, avail - delta, next);
|
|
|
|
set_slob(cur, delta, aligned);
|
2006-01-08 09:01:45 +00:00
|
|
|
prev = cur;
|
|
|
|
cur = aligned;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
avail = slob_units(cur);
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
next = slob_next(cur);
|
|
|
|
if (avail == units) { /* exact fit? unlink. */
|
|
|
|
if (prev)
|
|
|
|
set_slob(prev, slob_units(prev), next);
|
|
|
|
else
|
2012-06-13 15:24:52 +00:00
|
|
|
sp->freelist = next;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
} else { /* fragment */
|
|
|
|
if (prev)
|
|
|
|
set_slob(prev, slob_units(prev), cur + units);
|
|
|
|
else
|
2012-06-13 15:24:52 +00:00
|
|
|
sp->freelist = cur + units;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
set_slob(cur + units, avail - units, next);
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
sp->units -= units;
|
|
|
|
if (!sp->units)
|
|
|
|
clear_slob_page_free(sp);
|
2006-01-08 09:01:45 +00:00
|
|
|
return cur;
|
|
|
|
}
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
if (slob_last(cur))
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
}
|
2006-01-08 09:01:45 +00:00
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* slob_alloc: entry point into the slob allocator.
|
|
|
|
*/
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
{
|
2012-06-13 15:24:52 +00:00
|
|
|
struct page *sp;
|
2007-07-21 11:37:40 +00:00
|
|
|
struct list_head *prev;
|
2008-02-05 06:29:37 +00:00
|
|
|
struct list_head *slob_list;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
slob_t *b = NULL;
|
|
|
|
unsigned long flags;
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2008-02-05 06:29:37 +00:00
|
|
|
if (size < SLOB_BREAK1)
|
|
|
|
slob_list = &free_slob_small;
|
|
|
|
else if (size < SLOB_BREAK2)
|
|
|
|
slob_list = &free_slob_medium;
|
|
|
|
else
|
|
|
|
slob_list = &free_slob_large;
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
spin_lock_irqsave(&slob_lock, flags);
|
|
|
|
/* Iterate through each partially free page, try to find room */
|
2014-04-08 20:44:27 +00:00
|
|
|
list_for_each_entry(sp, slob_list, lru) {
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
/*
|
|
|
|
* If there's a node specification, search for a partial
|
|
|
|
* page with a matching node id in the freelist.
|
|
|
|
*/
|
2012-09-08 20:47:51 +00:00
|
|
|
if (node != NUMA_NO_NODE && page_to_nid(sp) != node)
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
continue;
|
|
|
|
#endif
|
2007-07-21 11:37:40 +00:00
|
|
|
/* Enough room on this page? */
|
|
|
|
if (sp->units < SLOB_UNITS(size))
|
|
|
|
continue;
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
|
2007-07-21 11:37:40 +00:00
|
|
|
/* Attempt to alloc */
|
2014-04-08 20:44:27 +00:00
|
|
|
prev = sp->lru.prev;
|
2007-07-21 11:37:40 +00:00
|
|
|
b = slob_page_alloc(sp, size, align);
|
|
|
|
if (!b)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* Improve fragment distribution and reduce our average
|
|
|
|
* search time by starting our next search here. (see
|
|
|
|
* Knuth vol 1, sec 2.5, pg 449) */
|
2008-02-05 06:29:37 +00:00
|
|
|
if (prev != slob_list->prev &&
|
|
|
|
slob_list->next != prev->next)
|
|
|
|
list_move_tail(slob_list, prev->next);
|
2007-07-21 11:37:40 +00:00
|
|
|
break;
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
spin_unlock_irqrestore(&slob_lock, flags);
|
|
|
|
|
|
|
|
/* Not enough space: must allocate a new page */
|
|
|
|
if (!b) {
|
2009-01-18 18:00:38 +00:00
|
|
|
b = slob_new_pages(gfp & ~__GFP_ZERO, 0, node);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
if (!b)
|
2009-01-18 18:00:38 +00:00
|
|
|
return NULL;
|
2012-06-13 15:24:54 +00:00
|
|
|
sp = virt_to_page(b);
|
|
|
|
__SetPageSlab(sp);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
|
|
|
|
spin_lock_irqsave(&slob_lock, flags);
|
|
|
|
sp->units = SLOB_UNITS(PAGE_SIZE);
|
2012-06-13 15:24:52 +00:00
|
|
|
sp->freelist = b;
|
2014-04-08 20:44:27 +00:00
|
|
|
INIT_LIST_HEAD(&sp->lru);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
set_slob(b, SLOB_UNITS(PAGE_SIZE), b + SLOB_UNITS(PAGE_SIZE));
|
2008-02-05 06:29:37 +00:00
|
|
|
set_slob_page_free(sp, slob_list);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
b = slob_page_alloc(sp, size, align);
|
|
|
|
BUG_ON(!b);
|
|
|
|
spin_unlock_irqrestore(&slob_lock, flags);
|
|
|
|
}
|
2007-07-17 11:03:23 +00:00
|
|
|
if (unlikely((gfp & __GFP_ZERO) && b))
|
|
|
|
memset(b, 0, size);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
return b;
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* slob_free: entry point into the slob allocator.
|
|
|
|
*/
|
2006-01-08 09:01:45 +00:00
|
|
|
static void slob_free(void *block, int size)
|
|
|
|
{
|
2012-06-13 15:24:52 +00:00
|
|
|
struct page *sp;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
slob_t *prev, *next, *b = (slob_t *)block;
|
|
|
|
slobidx_t units;
|
2006-01-08 09:01:45 +00:00
|
|
|
unsigned long flags;
|
2010-07-10 10:05:33 +00:00
|
|
|
struct list_head *slob_list;
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2007-10-16 08:24:44 +00:00
|
|
|
if (unlikely(ZERO_OR_NULL_PTR(block)))
|
2006-01-08 09:01:45 +00:00
|
|
|
return;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
BUG_ON(!size);
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2012-06-13 15:24:54 +00:00
|
|
|
sp = virt_to_page(block);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
units = SLOB_UNITS(size);
|
2006-01-08 09:01:45 +00:00
|
|
|
|
|
|
|
spin_lock_irqsave(&slob_lock, flags);
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
if (sp->units + units == SLOB_UNITS(PAGE_SIZE)) {
|
|
|
|
/* Go directly to page allocator. Do not pass slob allocator */
|
|
|
|
if (slob_page_free(sp))
|
|
|
|
clear_slob_page_free(sp);
|
2009-03-16 10:00:28 +00:00
|
|
|
spin_unlock_irqrestore(&slob_lock, flags);
|
2012-06-13 15:24:54 +00:00
|
|
|
__ClearPageSlab(sp);
|
2013-02-23 00:34:59 +00:00
|
|
|
page_mapcount_reset(sp);
|
2009-05-05 09:13:45 +00:00
|
|
|
slob_free_pages(b, 0);
|
2009-03-16 10:00:28 +00:00
|
|
|
return;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
}
|
2006-01-08 09:01:45 +00:00
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
if (!slob_page_free(sp)) {
|
|
|
|
/* This slob page is about to become partially free. Easy! */
|
|
|
|
sp->units = units;
|
2012-06-13 15:24:52 +00:00
|
|
|
sp->freelist = b;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
set_slob(b, units,
|
|
|
|
(void *)((unsigned long)(b +
|
|
|
|
SLOB_UNITS(PAGE_SIZE)) & PAGE_MASK));
|
2010-07-10 10:05:33 +00:00
|
|
|
if (size < SLOB_BREAK1)
|
|
|
|
slob_list = &free_slob_small;
|
|
|
|
else if (size < SLOB_BREAK2)
|
|
|
|
slob_list = &free_slob_medium;
|
|
|
|
else
|
|
|
|
slob_list = &free_slob_large;
|
|
|
|
set_slob_page_free(sp, slob_list);
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Otherwise the page is already partially free, so find reinsertion
|
|
|
|
* point.
|
|
|
|
*/
|
|
|
|
sp->units += units;
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2012-06-13 15:24:52 +00:00
|
|
|
if (b < (slob_t *)sp->freelist) {
|
|
|
|
if (b + units == sp->freelist) {
|
|
|
|
units += slob_units(sp->freelist);
|
|
|
|
sp->freelist = slob_next(sp->freelist);
|
2008-02-05 06:29:37 +00:00
|
|
|
}
|
2012-06-13 15:24:52 +00:00
|
|
|
set_slob(b, units, sp->freelist);
|
|
|
|
sp->freelist = b;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
} else {
|
2012-06-13 15:24:52 +00:00
|
|
|
prev = sp->freelist;
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
next = slob_next(prev);
|
|
|
|
while (b > next) {
|
|
|
|
prev = next;
|
|
|
|
next = slob_next(prev);
|
|
|
|
}
|
2006-01-08 09:01:45 +00:00
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
if (!slob_last(prev) && b + units == next) {
|
|
|
|
units += slob_units(next);
|
|
|
|
set_slob(b, units, slob_next(next));
|
|
|
|
} else
|
|
|
|
set_slob(b, units, next);
|
|
|
|
|
|
|
|
if (prev + slob_units(prev) == b) {
|
|
|
|
units = slob_units(b) + slob_units(prev);
|
|
|
|
set_slob(prev, units, slob_next(b));
|
|
|
|
} else
|
|
|
|
set_slob(prev, slob_units(prev), b);
|
|
|
|
}
|
|
|
|
out:
|
2006-01-08 09:01:45 +00:00
|
|
|
spin_unlock_irqrestore(&slob_lock, flags);
|
|
|
|
}
|
|
|
|
|
slob: rework freelist handling
Improve slob by turning the freelist into a list of pages using struct page
fields, then each page has a singly linked freelist of slob blocks via a
pointer in the struct page.
- The first benefit is that the slob freelists can be indexed by a smaller
type (2 bytes, if the PAGE_SIZE is reasonable).
- Next is that freeing is much quicker because it does not have to traverse
the entire freelist. Allocation can be slightly faster too, because we can
skip almost-full freelist pages completely.
- Slob pages are then freed immediately when they become empty, rather than
having a periodic timer try to free them. This gives efficiency and memory
consumption improvement.
Then, we don't encode seperate size and next fields into each slob block,
rather we use the sign bit to distinguish between "size" or "next". Then
size 1 blocks contain a "next" offset, and others contain the "size" in
the first unit and "next" in the second unit.
- This allows minimum slob allocation alignment to go from 8 bytes to 2
bytes on 32-bit and 12 bytes to 2 bytes on 64-bit. In practice, it is
best to align them to word size, however some architectures (eg. cris)
could gain space savings from turning off this extra alignment.
Then, make kmalloc use its own slob_block at the front of the allocation
in order to encode allocation size, rather than rely on not overwriting
slob's existing header block.
- This reduces kmalloc allocation overhead similarly to alignment reductions.
- Decouples kmalloc layer from the slob allocator.
Then, add a page flag specific to slob pages.
- This means kfree of a page aligned slob block doesn't have to traverse
the bigblock list.
I would get benchmarks, but my test box's network doesn't come up with
slob before this patch. I think something is timing out. Anyway, things
are faster after the patch.
Code size goes up about 1K, however dynamic memory usage _should_ be
lower even on relatively small memory systems.
Future todo item is to restore the cyclic free list search, rather than
to always begin at the start.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:07 +00:00
|
|
|
/*
|
|
|
|
* End of slob allocator proper. Begin kmem_cache_alloc and kmalloc frontend.
|
|
|
|
*/
|
|
|
|
|
2012-09-08 20:47:53 +00:00
|
|
|
static __always_inline void *
|
|
|
|
__do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
|
2006-01-08 09:01:45 +00:00
|
|
|
{
|
2007-07-17 11:03:22 +00:00
|
|
|
unsigned int *m;
|
2012-10-05 14:55:20 +00:00
|
|
|
int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
|
2008-08-10 17:14:07 +00:00
|
|
|
void *ret;
|
2007-07-16 06:38:09 +00:00
|
|
|
|
2011-06-07 11:18:45 +00:00
|
|
|
gfp &= gfp_allowed_mask;
|
|
|
|
|
2009-03-15 05:03:11 +00:00
|
|
|
lockdep_trace_alloc(gfp);
|
lockdep: annotate reclaim context (__GFP_NOFS)
Here is another version, with the incremental patch rolled up, and
added reclaim context annotation to kswapd, and allocation tracing
to slab allocators (which may only ever reach the page allocator
in rare cases, so it is good to put annotations here too).
Haven't tested this version as such, but it should be getting closer
to merge worthy ;)
--
After noticing some code in mm/filemap.c accidentally perform a __GFP_FS
allocation when it should not have been, I thought it might be a good idea to
try to catch this kind of thing with lockdep.
I coded up a little idea that seems to work. Unfortunately the system has to
actually be in __GFP_FS page reclaim, then take the lock, before it will mark
it. But at least that might still be some orders of magnitude more common
(and more debuggable) than an actual deadlock condition, so we have some
improvement I hope (the concept is no less complete than discovery of a lock's
interrupt contexts).
I guess we could even do the same thing with __GFP_IO (normal reclaim), and
even GFP_NOIO locks too... but filesystems will have the most locks and fiddly
code paths, so let's start there and see how it goes.
It *seems* to work. I did a quick test.
=================================
[ INFO: inconsistent lock state ]
2.6.28-rc6-00007-ged31348-dirty #26
---------------------------------
inconsistent {in-reclaim-W} -> {ov-reclaim-W} usage.
modprobe/8526 [HC0[0]:SC0[0]:HE1:SE1] takes:
(testlock){--..}, at: [<ffffffffa0020055>] brd_init+0x55/0x216 [brd]
{in-reclaim-W} state was registered at:
[<ffffffff80267bdb>] __lock_acquire+0x75b/0x1a60
[<ffffffff80268f71>] lock_acquire+0x91/0xc0
[<ffffffff8070f0e1>] mutex_lock_nested+0xb1/0x310
[<ffffffffa002002b>] brd_init+0x2b/0x216 [brd]
[<ffffffff8020903b>] _stext+0x3b/0x170
[<ffffffff80272ebf>] sys_init_module+0xaf/0x1e0
[<ffffffff8020c3fb>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
irq event stamp: 3929
hardirqs last enabled at (3929): [<ffffffff8070f2b5>] mutex_lock_nested+0x285/0x310
hardirqs last disabled at (3928): [<ffffffff8070f089>] mutex_lock_nested+0x59/0x310
softirqs last enabled at (3732): [<ffffffff8061f623>] sk_filter+0x83/0xe0
softirqs last disabled at (3730): [<ffffffff8061f5b6>] sk_filter+0x16/0xe0
other info that might help us debug this:
1 lock held by modprobe/8526:
#0: (testlock){--..}, at: [<ffffffffa0020055>] brd_init+0x55/0x216 [brd]
stack backtrace:
Pid: 8526, comm: modprobe Not tainted 2.6.28-rc6-00007-ged31348-dirty #26
Call Trace:
[<ffffffff80265483>] print_usage_bug+0x193/0x1d0
[<ffffffff80266530>] mark_lock+0xaf0/0xca0
[<ffffffff80266735>] mark_held_locks+0x55/0xc0
[<ffffffffa0020000>] ? brd_init+0x0/0x216 [brd]
[<ffffffff802667ca>] trace_reclaim_fs+0x2a/0x60
[<ffffffff80285005>] __alloc_pages_internal+0x475/0x580
[<ffffffff8070f29e>] ? mutex_lock_nested+0x26e/0x310
[<ffffffffa0020000>] ? brd_init+0x0/0x216 [brd]
[<ffffffffa002006a>] brd_init+0x6a/0x216 [brd]
[<ffffffffa0020000>] ? brd_init+0x0/0x216 [brd]
[<ffffffff8020903b>] _stext+0x3b/0x170
[<ffffffff8070f8b9>] ? mutex_unlock+0x9/0x10
[<ffffffff8070f83d>] ? __mutex_unlock_slowpath+0x10d/0x180
[<ffffffff802669ec>] ? trace_hardirqs_on_caller+0x12c/0x190
[<ffffffff80272ebf>] sys_init_module+0xaf/0x1e0
[<ffffffff8020c3fb>] system_call_fastpath+0x16/0x1b
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-01-21 07:12:39 +00:00
|
|
|
|
2007-07-16 06:38:09 +00:00
|
|
|
if (size < PAGE_SIZE - align) {
|
2007-07-17 11:03:22 +00:00
|
|
|
if (!size)
|
|
|
|
return ZERO_SIZE_PTR;
|
|
|
|
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
m = slob_alloc(size + align, gfp, align, node);
|
2008-08-10 17:14:07 +00:00
|
|
|
|
2008-05-19 13:12:08 +00:00
|
|
|
if (!m)
|
|
|
|
return NULL;
|
|
|
|
*m = size;
|
2008-08-10 17:14:07 +00:00
|
|
|
ret = (void *)m + align;
|
|
|
|
|
2012-09-08 20:47:53 +00:00
|
|
|
trace_kmalloc_node(caller, ret,
|
2009-03-23 13:12:24 +00:00
|
|
|
size, size + align, gfp, node);
|
2007-07-16 06:38:08 +00:00
|
|
|
} else {
|
2008-08-10 17:14:07 +00:00
|
|
|
unsigned int order = get_order(size);
|
2007-07-16 06:38:08 +00:00
|
|
|
|
2010-08-22 23:16:06 +00:00
|
|
|
if (likely(order))
|
|
|
|
gfp |= __GFP_COMP;
|
|
|
|
ret = slob_new_pages(gfp, order, node);
|
2008-08-10 17:14:07 +00:00
|
|
|
|
2012-09-08 20:47:53 +00:00
|
|
|
trace_kmalloc_node(caller, ret,
|
2009-03-23 13:12:24 +00:00
|
|
|
size, PAGE_SIZE << order, gfp, node);
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
2008-08-10 17:14:07 +00:00
|
|
|
|
2009-06-11 12:23:17 +00:00
|
|
|
kmemleak_alloc(ret, size, 1, gfp);
|
2008-08-10 17:14:07 +00:00
|
|
|
return ret;
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
2012-09-08 20:47:53 +00:00
|
|
|
|
2013-09-04 16:35:34 +00:00
|
|
|
void *__kmalloc(size_t size, gfp_t gfp)
|
2012-09-08 20:47:53 +00:00
|
|
|
{
|
2013-09-04 16:35:34 +00:00
|
|
|
return __do_kmalloc_node(size, gfp, NUMA_NO_NODE, _RET_IP_);
|
2012-09-08 20:47:53 +00:00
|
|
|
}
|
2013-09-04 16:35:34 +00:00
|
|
|
EXPORT_SYMBOL(__kmalloc);
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2012-09-08 20:47:53 +00:00
|
|
|
void *__kmalloc_track_caller(size_t size, gfp_t gfp, unsigned long caller)
|
|
|
|
{
|
|
|
|
return __do_kmalloc_node(size, gfp, NUMA_NO_NODE, caller);
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
mm, slob: fix build breakage in __kmalloc_node_track_caller
On Sat, 8 Sep 2012, Ezequiel Garcia wrote:
> @@ -454,15 +455,35 @@ void *__kmalloc_node(size_t size, gfp_t gfp, int node)
> gfp |= __GFP_COMP;
> ret = slob_new_pages(gfp, order, node);
>
> - trace_kmalloc_node(_RET_IP_, ret,
> + trace_kmalloc_node(caller, ret,
> size, PAGE_SIZE << order, gfp, node);
> }
>
> kmemleak_alloc(ret, size, 1, gfp);
> return ret;
> }
> +
> +void *__kmalloc_node(size_t size, gfp_t gfp, int node)
> +{
> + return __do_kmalloc_node(size, gfp, node, _RET_IP_);
> +}
> EXPORT_SYMBOL(__kmalloc_node);
>
> +#ifdef CONFIG_TRACING
> +void *__kmalloc_track_caller(size_t size, gfp_t gfp, unsigned long caller)
> +{
> + return __do_kmalloc_node(size, gfp, NUMA_NO_NODE, caller);
> +}
> +
> +#ifdef CONFIG_NUMA
> +void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
> + int node, unsigned long caller)
> +{
> + return __do_kmalloc_node(size, gfp, node, caller);
> +}
> +#endif
This breaks Pekka's slab/next tree with this:
mm/slob.c: In function '__kmalloc_node_track_caller':
mm/slob.c:488: error: 'gfp' undeclared (first use in this function)
mm/slob.c:488: error: (Each undeclared identifier is reported only once
mm/slob.c:488: error: for each function it appears in.)
mm, slob: fix build breakage in __kmalloc_node_track_caller
"mm, slob: Add support for kmalloc_track_caller()" breaks the build
because gfp is undeclared. Fix it.
Acked-by: Ezequiel Garcia <elezegarcia@gmail.com>
Signed-off-by: David Rientjes <rientjes@google.com>
Signed-off-by: Pekka Enberg <penberg@kernel.org>
2012-09-25 19:53:51 +00:00
|
|
|
void *__kmalloc_node_track_caller(size_t size, gfp_t gfp,
|
2012-09-08 20:47:53 +00:00
|
|
|
int node, unsigned long caller)
|
|
|
|
{
|
|
|
|
return __do_kmalloc_node(size, gfp, node, caller);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2006-01-08 09:01:45 +00:00
|
|
|
void kfree(const void *block)
|
|
|
|
{
|
2012-06-13 15:24:52 +00:00
|
|
|
struct page *sp;
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2009-03-25 09:05:57 +00:00
|
|
|
trace_kfree(_RET_IP_, block);
|
|
|
|
|
2007-10-16 08:24:44 +00:00
|
|
|
if (unlikely(ZERO_OR_NULL_PTR(block)))
|
2006-01-08 09:01:45 +00:00
|
|
|
return;
|
2009-06-11 12:23:17 +00:00
|
|
|
kmemleak_free(block);
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2012-06-13 15:24:54 +00:00
|
|
|
sp = virt_to_page(block);
|
|
|
|
if (PageSlab(sp)) {
|
2012-10-05 14:55:20 +00:00
|
|
|
int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
|
2007-07-16 06:38:09 +00:00
|
|
|
unsigned int *m = (unsigned int *)(block - align);
|
|
|
|
slob_free(m, *m + align);
|
2007-07-16 06:38:08 +00:00
|
|
|
} else
|
2012-10-22 12:04:31 +00:00
|
|
|
__free_pages(sp, compound_order(sp));
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(kfree);
|
|
|
|
|
2007-07-16 06:38:08 +00:00
|
|
|
/* can't use ksize for kmem_cache_alloc memory, only kmalloc */
|
2007-05-06 21:48:40 +00:00
|
|
|
size_t ksize(const void *block)
|
2006-01-08 09:01:45 +00:00
|
|
|
{
|
2012-06-13 15:24:52 +00:00
|
|
|
struct page *sp;
|
2012-10-19 12:33:10 +00:00
|
|
|
int align;
|
|
|
|
unsigned int *m;
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2007-10-16 08:24:46 +00:00
|
|
|
BUG_ON(!block);
|
|
|
|
if (unlikely(block == ZERO_SIZE_PTR))
|
2006-01-08 09:01:45 +00:00
|
|
|
return 0;
|
|
|
|
|
2012-06-13 15:24:54 +00:00
|
|
|
sp = virt_to_page(block);
|
2012-10-19 12:33:10 +00:00
|
|
|
if (unlikely(!PageSlab(sp)))
|
|
|
|
return PAGE_SIZE << compound_order(sp);
|
|
|
|
|
2012-10-05 14:55:20 +00:00
|
|
|
align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
|
2012-10-19 12:33:10 +00:00
|
|
|
m = (unsigned int *)(block - align);
|
|
|
|
return SLOB_UNITS(*m) * SLOB_UNIT;
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
2009-02-10 13:21:44 +00:00
|
|
|
EXPORT_SYMBOL(ksize);
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2012-09-04 23:18:33 +00:00
|
|
|
int __kmem_cache_create(struct kmem_cache *c, unsigned long flags)
|
2006-01-08 09:01:45 +00:00
|
|
|
{
|
2012-09-05 00:20:34 +00:00
|
|
|
if (flags & SLAB_DESTROY_BY_RCU) {
|
|
|
|
/* leave room for rcu footer at the end of object */
|
|
|
|
c->size += sizeof(struct slob_rcu);
|
2012-07-06 20:25:10 +00:00
|
|
|
}
|
2012-09-05 00:20:34 +00:00
|
|
|
c->flags = flags;
|
|
|
|
return 0;
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
|
|
|
|
2013-09-04 16:35:34 +00:00
|
|
|
void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
|
2006-01-08 09:01:45 +00:00
|
|
|
{
|
|
|
|
void *b;
|
|
|
|
|
2011-06-07 11:18:45 +00:00
|
|
|
flags &= gfp_allowed_mask;
|
|
|
|
|
|
|
|
lockdep_trace_alloc(flags);
|
|
|
|
|
2008-08-10 17:14:07 +00:00
|
|
|
if (c->size < PAGE_SIZE) {
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
b = slob_alloc(c->size, flags, c->align, node);
|
2012-10-19 12:33:11 +00:00
|
|
|
trace_kmem_cache_alloc_node(_RET_IP_, b, c->object_size,
|
2009-03-23 13:12:24 +00:00
|
|
|
SLOB_UNITS(c->size) * SLOB_UNIT,
|
|
|
|
flags, node);
|
2008-08-10 17:14:07 +00:00
|
|
|
} else {
|
2009-01-18 18:00:38 +00:00
|
|
|
b = slob_new_pages(flags, get_order(c->size), node);
|
2012-10-19 12:33:11 +00:00
|
|
|
trace_kmem_cache_alloc_node(_RET_IP_, b, c->object_size,
|
2009-03-23 13:12:24 +00:00
|
|
|
PAGE_SIZE << get_order(c->size),
|
|
|
|
flags, node);
|
2008-08-10 17:14:07 +00:00
|
|
|
}
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2013-01-17 17:13:46 +00:00
|
|
|
if (b && c->ctor)
|
2008-07-26 02:45:34 +00:00
|
|
|
c->ctor(b);
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2009-06-11 12:23:17 +00:00
|
|
|
kmemleak_alloc_recursive(b, c->size, 1, c->flags, flags);
|
2006-01-08 09:01:45 +00:00
|
|
|
return b;
|
|
|
|
}
|
2013-09-04 16:35:34 +00:00
|
|
|
EXPORT_SYMBOL(slob_alloc_node);
|
|
|
|
|
|
|
|
void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
|
|
|
|
{
|
|
|
|
return slob_alloc_node(cachep, flags, NUMA_NO_NODE);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(kmem_cache_alloc);
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
void *__kmalloc_node(size_t size, gfp_t gfp, int node)
|
|
|
|
{
|
|
|
|
return __do_kmalloc_node(size, gfp, node, _RET_IP_);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__kmalloc_node);
|
|
|
|
|
|
|
|
void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t gfp, int node)
|
|
|
|
{
|
|
|
|
return slob_alloc_node(cachep, gfp, node);
|
|
|
|
}
|
slob: initial NUMA support
This adds preliminary NUMA support to SLOB, primarily aimed at systems with
small nodes (tested all the way down to a 128kB SRAM block), whether
asymmetric or otherwise.
We follow the same conventions as SLAB/SLUB, preferring current node
placement for new pages, or with explicit placement, if a node has been
specified. Presently on UP NUMA this has the side-effect of preferring
node#0 allocations (since numa_node_id() == 0, though this could be
reworked if we could hand off a pfn to determine node placement), so
single-CPU NUMA systems will want to place smaller nodes further out in
terms of node id. Once a page has been bound to a node (via explicit node
id typing), we only do block allocations from partial free pages that have
a matching node id in the page flags.
The current implementation does have some scalability problems, in that all
partial free pages are tracked in the global freelist (with contention due
to the single spinlock). However, these are things that are being reworked
for SMP scalability first, while things like per-node freelists can easily
be built on top of this sort of functionality once it's been added.
More background can be found in:
http://marc.info/?l=linux-mm&m=118117916022379&w=2
http://marc.info/?l=linux-mm&m=118170446306199&w=2
http://marc.info/?l=linux-mm&m=118187859420048&w=2
and subsequent threads.
Acked-by: Christoph Lameter <clameter@sgi.com>
Acked-by: Matt Mackall <mpm@selenic.com>
Signed-off-by: Paul Mundt <lethal@linux-sh.org>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-16 06:38:22 +00:00
|
|
|
EXPORT_SYMBOL(kmem_cache_alloc_node);
|
2013-09-04 16:35:34 +00:00
|
|
|
#endif
|
2006-01-08 09:01:45 +00:00
|
|
|
|
2007-05-17 05:10:49 +00:00
|
|
|
static void __kmem_cache_free(void *b, int size)
|
2006-01-08 09:01:45 +00:00
|
|
|
{
|
2007-05-17 05:10:49 +00:00
|
|
|
if (size < PAGE_SIZE)
|
|
|
|
slob_free(b, size);
|
2006-01-08 09:01:45 +00:00
|
|
|
else
|
2009-01-18 18:00:38 +00:00
|
|
|
slob_free_pages(b, get_order(size));
|
2007-05-17 05:10:49 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static void kmem_rcu_free(struct rcu_head *head)
|
|
|
|
{
|
|
|
|
struct slob_rcu *slob_rcu = (struct slob_rcu *)head;
|
|
|
|
void *b = (void *)slob_rcu - (slob_rcu->size - sizeof(struct slob_rcu));
|
|
|
|
|
|
|
|
__kmem_cache_free(b, slob_rcu->size);
|
|
|
|
}
|
|
|
|
|
|
|
|
void kmem_cache_free(struct kmem_cache *c, void *b)
|
|
|
|
{
|
2009-06-11 12:23:17 +00:00
|
|
|
kmemleak_free_recursive(b, c->flags);
|
2007-05-17 05:10:49 +00:00
|
|
|
if (unlikely(c->flags & SLAB_DESTROY_BY_RCU)) {
|
|
|
|
struct slob_rcu *slob_rcu;
|
|
|
|
slob_rcu = b + (c->size - sizeof(struct slob_rcu));
|
|
|
|
slob_rcu->size = c->size;
|
|
|
|
call_rcu(&slob_rcu->head, kmem_rcu_free);
|
|
|
|
} else {
|
|
|
|
__kmem_cache_free(b, c->size);
|
|
|
|
}
|
2008-08-10 17:14:07 +00:00
|
|
|
|
2009-03-23 13:12:24 +00:00
|
|
|
trace_kmem_cache_free(_RET_IP_, b);
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(kmem_cache_free);
|
|
|
|
|
2012-09-04 23:18:33 +00:00
|
|
|
int __kmem_cache_shutdown(struct kmem_cache *c)
|
|
|
|
{
|
|
|
|
/* No way to check for remaining objects */
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
slab: get_online_mems for kmem_cache_{create,destroy,shrink}
When we create a sl[au]b cache, we allocate kmem_cache_node structures
for each online NUMA node. To handle nodes taken online/offline, we
register memory hotplug notifier and allocate/free kmem_cache_node
corresponding to the node that changes its state for each kmem cache.
To synchronize between the two paths we hold the slab_mutex during both
the cache creationg/destruction path and while tuning per-node parts of
kmem caches in memory hotplug handler, but that's not quite right,
because it does not guarantee that a newly created cache will have all
kmem_cache_nodes initialized in case it races with memory hotplug. For
instance, in case of slub:
CPU0 CPU1
---- ----
kmem_cache_create: online_pages:
__kmem_cache_create: slab_memory_callback:
slab_mem_going_online_callback:
lock slab_mutex
for each slab_caches list entry
allocate kmem_cache node
unlock slab_mutex
lock slab_mutex
init_kmem_cache_nodes:
for_each_node_state(node, N_NORMAL_MEMORY)
allocate kmem_cache node
add kmem_cache to slab_caches list
unlock slab_mutex
online_pages (continued):
node_states_set_node
As a result we'll get a kmem cache with not all kmem_cache_nodes
allocated.
To avoid issues like that we should hold get/put_online_mems() during
the whole kmem cache creation/destruction/shrink paths, just like we
deal with cpu hotplug. This patch does the trick.
Note, that after it's applied, there is no need in taking the slab_mutex
for kmem_cache_shrink any more, so it is removed from there.
Signed-off-by: Vladimir Davydov <vdavydov@parallels.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: Tang Chen <tangchen@cn.fujitsu.com>
Cc: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Cc: Toshi Kani <toshi.kani@hp.com>
Cc: Xishi Qiu <qiuxishi@huawei.com>
Cc: Jiang Liu <liuj97@gmail.com>
Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-06-04 23:07:20 +00:00
|
|
|
int __kmem_cache_shrink(struct kmem_cache *d)
|
2006-12-13 08:34:23 +00:00
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-09-05 00:20:33 +00:00
|
|
|
struct kmem_cache kmem_cache_boot = {
|
|
|
|
.name = "kmem_cache",
|
|
|
|
.size = sizeof(struct kmem_cache),
|
|
|
|
.flags = SLAB_PANIC,
|
|
|
|
.align = ARCH_KMALLOC_MINALIGN,
|
|
|
|
};
|
|
|
|
|
2006-12-30 00:48:28 +00:00
|
|
|
void __init kmem_cache_init(void)
|
|
|
|
{
|
2012-09-05 00:20:33 +00:00
|
|
|
kmem_cache = &kmem_cache_boot;
|
2012-07-06 20:25:11 +00:00
|
|
|
slab_state = UP;
|
2006-01-08 09:01:45 +00:00
|
|
|
}
|
2009-08-06 08:36:25 +00:00
|
|
|
|
|
|
|
void __init kmem_cache_init_late(void)
|
|
|
|
{
|
2012-07-06 20:25:11 +00:00
|
|
|
slab_state = FULL;
|
2009-08-06 08:36:25 +00:00
|
|
|
}
|