linux/drivers/block
Weijie Yang d2d5e762c8 zram: replace global tb_lock with fine grain lock
Currently, we use a rwlock tb_lock to protect concurrent access to the
whole zram meta table.  However, according to the actual access model,
there is only a small chance for upper user to access the same
table[index], so the current lock granularity is too big.

The idea of optimization is to change the lock granularity from whole
meta table to per table entry (table -> table[index]), so that we can
protect concurrent access to the same table[index], meanwhile allow the
maximum concurrency.

With this in mind, several kinds of locks which could be used as a
per-entry lock were tested and compared:

Test environment:
x86-64 Intel Core2 Q8400, system memory 4GB, Ubuntu 12.04,
kernel v3.15.0-rc3 as base, zram with 4 max_comp_streams LZO.

iozone test:
iozone -t 4 -R -r 16K -s 200M -I +Z
(1GB zram with ext4 filesystem, take the average of 10 tests, KB/s)

      Test       base      CAS    spinlock    rwlock   bit_spinlock
-------------------------------------------------------------------
 Initial write  1381094   1425435   1422860   1423075   1421521
       Rewrite  1529479   1641199   1668762   1672855   1654910
          Read  8468009  11324979  11305569  11117273  10997202
       Re-read  8467476  11260914  11248059  11145336  10906486
  Reverse Read  6821393   8106334   8282174   8279195   8109186
   Stride read  7191093   8994306   9153982   8961224   9004434
   Random read  7156353   8957932   9167098   8980465   8940476
Mixed workload  4172747   5680814   5927825   5489578   5972253
  Random write  1483044   1605588   1594329   1600453   1596010
        Pwrite  1276644   1303108   1311612   1314228   1300960
         Pread  4324337   4632869   4618386   4457870   4500166

To enhance the possibility of access the same table[index] concurrently,
set zram a small disksize(10MB) and let threads run with large loop
count.

fio test:
fio --bs=32k --randrepeat=1 --randseed=100 --refill_buffers
--scramble_buffers=1 --direct=1 --loops=3000 --numjobs=4
--filename=/dev/zram0 --name=seq-write --rw=write --stonewall
--name=seq-read --rw=read --stonewall --name=seq-readwrite
--rw=rw --stonewall --name=rand-readwrite --rw=randrw --stonewall
(10MB zram raw block device, take the average of 10 tests, KB/s)

    Test     base     CAS    spinlock    rwlock  bit_spinlock
-------------------------------------------------------------
seq-write   933789   999357   1003298    995961   1001958
 seq-read  5634130  6577930   6380861   6243912   6230006
   seq-rw  1405687  1638117   1640256   1633903   1634459
  rand-rw  1386119  1614664   1617211   1609267   1612471

All the optimization methods show a higher performance than the base,
however, it is hard to say which method is the most appropriate.

On the other hand, zram is mostly used on small embedded system, so we
don't want to increase any memory footprint.

This patch pick the bit_spinlock method, pack object size and page_flag
into an unsigned long table.value, so as to not increase any memory
overhead on both 32-bit and 64-bit system.

On the third hand, even though different kinds of locks have different
performances, we can ignore this difference, because: if zram is used as
zram swapfile, the swap subsystem can prevent concurrent access to the
same swapslot; if zram is used as zram-blk for set up filesystem on it,
the upper filesystem and the page cache also prevent concurrent access
of the same block mostly.  So we can ignore the different performances
among locks.

Acked-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Reviewed-by: Davidlohr Bueso <davidlohr@hp.com>
Signed-off-by: Weijie Yang <weijie.yang@samsung.com>
Signed-off-by: Minchan Kim <minchan@kernel.org>
Cc: Jerome Marchand <jmarchan@redhat.com>
Cc: Nitin Gupta <ngupta@vflare.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-08-06 18:01:23 -07:00
..
aoe mm: close PageTail race 2014-03-04 07:55:47 -08:00
drbd drbd: fix regression 'out of mem, failed to invoke fence-peer helper' 2014-07-10 11:06:03 +02:00
mtip32xx mtip32xx: minor performance enhancements 2014-06-06 13:28:48 -06:00
paride block: remove struct request buffer member 2014-04-15 14:03:02 -06:00
rsxx block: Convert bio_for_each_segment() to bvec_iter 2013-11-23 22:33:49 -08:00
xen-blkback xen-blkback: defer freeing blkif to avoid blocking xenwatch 2014-05-28 14:17:32 -04:00
zram zram: replace global tb_lock with fine grain lock 2014-08-06 18:01:23 -07:00
amiflop.c block: remove struct request buffer member 2014-04-15 14:03:02 -06:00
ataflop.c Merge branch 'for-3.16/core' of git://git.kernel.dk/linux-block into next 2014-06-02 09:29:34 -07:00
brd.c brd: return -ENOSPC rather than -ENOMEM on page allocation failure 2014-06-04 16:54:02 -07:00
cciss_cmd.h
cciss_scsi.c cciss: switch to ->show_info() 2013-04-09 14:13:19 -04:00
cciss_scsi.h
cciss.c cciss: Use pci_enable_msix_exact() instead of pci_enable_msix() 2014-04-21 19:02:08 -06:00
cciss.h
cpqarray.c cpqarray: fix info leak in ida_locked_ioctl() 2013-09-24 17:00:26 -07:00
cpqarray.h
cryptoloop.c move linux/loop.h to drivers/block 2013-06-29 12:46:45 +04:00
DAC960.c DAC960: remove sleep_on usage 2014-03-13 14:56:38 -06:00
DAC960.h
floppy.c Merge branch 'for-jens' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/linux-block into for-linus 2014-06-18 10:30:22 -07:00
hd.c block: remove struct request buffer member 2014-04-15 14:03:02 -06:00
ida_cmd.h
ida_ioctl.h
Kconfig zram: promote zram from staging 2014-01-30 16:56:55 -08:00
loop.c Merge branch 'sched/urgent' into sched/core, to avoid conflicts 2014-05-07 13:15:46 +02:00
loop.h move linux/loop.h to drivers/block 2013-06-29 12:46:45 +04:00
Makefile zram: promote zram from staging 2014-01-30 16:56:55 -08:00
mg_disk.c block: remove struct request buffer member 2014-04-15 14:03:02 -06:00
nbd.c nbd: zero from and len fields in NBD_CMD_DISCONNECT. 2014-06-06 16:08:18 -07:00
null_blk.c null_blk: fix softirq completions for queue_mode == 1 2014-06-16 11:40:25 -06:00
nvme-core.c Merge git://git.infradead.org/users/willy/linux-nvme 2014-06-15 15:58:03 -10:00
nvme-scsi.c NVMe: Fix START_STOP_UNIT Scsi->NVMe translation. 2014-06-13 13:11:00 -04:00
osdblk.c block: replace strict_strtoul() with kstrtoul() 2013-09-11 15:56:56 -07:00
pktcdvd.c block: add blk_rq_set_block_pc() 2014-06-06 07:57:37 -06:00
ps3disk.c block: Kill bio_segments()/bi_vcnt usage 2013-11-23 22:33:51 -08:00
ps3vram.c block: Convert bio_for_each_segment() to bvec_iter 2013-11-23 22:33:49 -08:00
rbd_types.h
rbd.c rbd: handle parent_overlap on writes correctly 2014-06-23 12:55:37 +04:00
skd_main.c Merge branch 'for-3.16/core' into for-3.16/drivers 2014-05-28 09:50:26 -06:00
skd_s1120.h skd: fix formatting in skd_s1120.h 2013-11-08 09:10:30 -07:00
smart1,2.h
sunvdc.c sunvdc: Fix off-by-one in generic_request(). 2013-02-14 11:49:01 -08:00
swim3.c block: remove struct request buffer member 2014-04-15 14:03:02 -06:00
swim_asm.S
swim.c block: remove struct request buffer member 2014-04-15 14:03:02 -06:00
sx8.c drivers/block/sx8.c: remove unnecessary pci_set_drvdata() 2014-01-21 20:16:56 -08:00
umem.c block: Convert drivers to immutable biovecs 2013-11-23 22:33:51 -08:00
umem.h
virtio_blk.c Merge branch 'for-3.16/core' of git://git.kernel.dk/linux-block into next 2014-06-02 09:29:34 -07:00
xen-blkfront.c Merge branch 'stable/for-jens-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into for-3.16/drivers 2014-05-28 12:37:04 -06:00
xsysace.c block: remove struct request buffer member 2014-04-15 14:03:02 -06:00
z2ram.c block: remove struct request buffer member 2014-04-15 14:03:02 -06:00