Andi Kleen 65dd2aa90a dio: optimize cache misses in the submission path ... Some investigation of a transaction processing workload showed that a major consumer of cycles in __blockdev_direct_IO is the cache miss while accessing the block size. This is because it has to walk the chain from block_dev to gendisk to queue. The block size is needed early on to check alignment and sizes. It's only done if the check for the inode block size fails. But the costly block device state is unconditionally fetched. - Reorganize the code to only fetch block dev state when actually needed. Then do a prefetch on the block dev early on in the direct IO path. This is worth it, because there is substantial code run before we actually touch the block dev now. - I also added some unlikelies to make it clear the compiler that block device fetch code is not normally executed. This gave a small, but measurable improvement on a large database benchmark (about 0.3%) [akpm@linux-foundation.org: coding-style fixes] [sfr@canb.auug.org.au: using prefetch requires including prefetch.h] Signed-off-by: Andi Kleen <ak@linux.intel.com> Cc: Jeff Moyer <jmoyer@redhat.com> Cc: Jens Axboe <axboe@kernel.dk> Cc: Christoph Hellwig <hch@lst.de> Signed-off-by: Stephen Rothwell <sfr@canb.auug.org.au> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2012-01-12 20:13:12 -08:00
..
9p	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ericvh/v9fs	2012-01-10 15:09:01 -08:00
adfs	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
affs	affs: propagate umode_t	2012-01-03 22:55:04 -05:00
afs	switch ->create() to umode_t	2012-01-03 22:54:53 -05:00
autofs4	autofs4: deal with autofs4_write/autofs4_write races	2012-01-11 00:20:12 -05:00
befs	vfs: fix the stupidity with i_dentry in inode destructors	2012-01-03 22:52:40 -05:00
bfs	switch ->create() to umode_t	2012-01-03 22:54:53 -05:00
btrfs	mm: compaction: introduce sync-light migration for use by compaction	2012-01-12 20:13:09 -08:00
cachefiles	fs: move code out of buffer.c	2012-01-03 22:54:07 -05:00
ceph	ceph: d_alloc_root() may fail	2012-01-09 16:36:12 -05:00
cifs	Merge branch 'for-linus2' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2012-01-08 12:19:57 -08:00
coda	coda: switch coda_cnode_make() to sane API as well, clean coda_lookup()	2012-01-10 11:13:16 -05:00
configfs	configfs: convert to umode_t	2012-01-03 22:54:57 -05:00
cramfs	Merge branches 'vfsmount-guts', 'umode_t' and 'partitions' into Z	2012-01-06 23:15:54 -05:00
debugfs	Merge branch 'for-linus2' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2012-01-08 12:19:57 -08:00
devpts	devpts: fix double-free on mount failure	2012-01-08 20:19:03 -05:00
dlm	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm	2012-01-10 14:55:55 -08:00
ecryptfs	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
efs	vfs: fix the stupidity with i_dentry in inode destructors	2012-01-03 22:52:40 -05:00
exofs	Merge branch 'for-linus' of git://git.open-osd.org/linux-open-osd	2012-01-09 12:51:01 -08:00
exportfs
ext2	Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs	2012-01-09 12:51:21 -08:00
ext3	Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs	2012-01-09 12:51:21 -08:00
ext4	Merge branch 'for_linus' into for_linus_merged	2012-01-10 11:54:07 -05:00
fat	Merge branch 'usb-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb	2012-01-09 12:09:47 -08:00
freevxfs	fs: propagate umode_t, misc bits	2012-01-03 22:55:10 -05:00
fscache	FS-Cache: Fix __fscache_uncache_all_inode_pages()'s outer loop	2011-07-21 10:59:16 -07:00
fuse	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
gfs2	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm	2012-01-10 14:55:55 -08:00
hfs	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
hfsplus	hfsplus: creation of hidden dir on mount can fail	2012-01-10 17:48:52 -05:00
hostfs	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
hpfs	switch ->mknod() to umode_t	2012-01-03 22:54:54 -05:00
hppfs	vfs: for usbfs, etc. internal vfsmounts ->mnt_sb->s_root == ->mnt_root	2012-01-03 22:52:41 -05:00
hugetlbfs	mm: compaction: introduce sync-light migration for use by compaction	2012-01-12 20:13:09 -08:00
isofs	isofs: inode leak on mount failure	2012-01-09 10:48:11 -05:00
jbd	Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs	2012-01-09 12:51:21 -08:00
jbd2	Merge branch 'for_linus' into for_linus_merged	2012-01-10 11:54:07 -05:00
jffs2	MTD pull for 3.3	2012-01-10 13:45:22 -08:00
jfs	Merge branch 'pm-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm	2012-01-08 13:10:57 -08:00
lockd	vfs: prefer ->dentry->d_sb to ->mnt->mnt_sb	2012-01-06 23:16:53 -05:00
logfs	MTD pull for 3.3	2012-01-10 13:45:22 -08:00
minix	Merge branch 'for-linus2' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2012-01-08 12:19:57 -08:00
ncpfs	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
nfs	mm: compaction: introduce sync-light migration for use by compaction	2012-01-12 20:13:09 -08:00
nfs_common
nfsd	Merge branch 'nfs-for-3.3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs	2012-01-10 14:57:40 -08:00
nilfs2	Merge branch 'pm-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm	2012-01-08 13:10:57 -08:00
nls	NLS: raname "maxlen" to "maxout" in UTF conversion routines	2011-11-26 19:58:47 -08:00
notify	vfs: move fsnotify junk to struct mount	2012-01-03 22:57:12 -05:00
ntfs	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
ocfs2	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm	2012-01-10 14:55:55 -08:00
omfs	omfs: propagate umode_t	2012-01-03 22:55:01 -05:00
openpromfs	vfs: fix the stupidity with i_dentry in inode destructors	2012-01-03 22:52:40 -05:00
proc	proc: fix null pointer deref in proc_pid_permission()	2012-01-12 20:13:02 -08:00
pstore	pstore: gracefully handle NULL pstore_info functions	2011-11-18 13:49:00 -08:00
qnx4	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial	2012-01-08 13:21:22 -08:00
quota	vfs: prefer ->dentry->d_sb to ->mnt->mnt_sb	2012-01-06 23:16:53 -05:00
ramfs	pohmelfs: propagate umode_t	2012-01-03 22:55:07 -05:00
reiserfs	reiserfs: don't lock root inode searching	2012-01-10 16:30:54 -08:00
romfs	MTD pull for 3.3	2012-01-10 13:45:22 -08:00
squashfs	vfs: fix the stupidity with i_dentry in inode destructors	2012-01-03 22:52:40 -05:00
sysfs	sysfs: propagate umode_t	2012-01-03 22:55:03 -05:00
sysv	vfs: prefer ->dentry->d_sb to ->mnt->mnt_sb	2012-01-06 23:16:53 -05:00
ubifs	Merge branch 'linux-next' of git://git.infradead.org/ubifs-2.6	2012-01-10 14:57:19 -08:00
udf	Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs	2012-01-09 12:51:21 -08:00
ufs	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
xfs	Merge branch 'for-linus' of git://oss.sgi.com/xfs/xfs	2012-01-09 12:50:15 -08:00
aio.c	aio: allocate kiocbs in batches	2011-11-02 16:07:03 -07:00
anon_inodes.c	vfs: dont chain pipe/anon/socket on superblock s_inodes list	2011-07-26 12:57:09 -04:00
attr.c	switch is_sxid() to umode_t	2012-01-03 22:55:11 -05:00
bad_inode.c	switch ->mknod() to umode_t	2012-01-03 22:54:54 -05:00
binfmt_aout.c
binfmt_elf_fdpic.c	consolidate BINPRM_FLAGS_ENFORCE_NONDUMP handling	2011-07-20 01:43:10 -04:00
binfmt_elf.c	fs: binfmt_elf: create Kconfig variable for PIE randomization	2012-01-10 16:30:51 -08:00
binfmt_em86.c
binfmt_flat.c
binfmt_misc.c	vfs: prefer ->dentry->d_sb to ->mnt->mnt_sb	2012-01-06 23:16:53 -05:00
binfmt_script.c
binfmt_som.c
bio-integrity.c	fs: add export.h to files using EXPORT_SYMBOL/THIS_MODULE macros	2011-10-31 19:30:31 -04:00
bio.c	bio: change some signed vars to unsigned	2011-11-16 09:21:50 +01:00
block_dev.c	vfs: cache request_queue in struct block_device	2012-01-12 20:13:12 -08:00
buffer.c	fs: move code out of buffer.c	2012-01-03 22:54:07 -05:00
char_dev.c	char_dev.c: fix up some whitespace errors	2011-12-13 11:18:17 -08:00
compat_binfmt_elf.c
compat_ioctl.c	vfs: fix up ENOIOCTLCMD error handling	2012-01-05 15:40:12 -08:00
compat.c	switch open and mkdir syscalls to umode_t	2012-01-03 22:55:19 -05:00
dcache.c	fix shrink_dcache_parent() livelock	2012-01-10 13:06:32 -05:00
dcookies.c
direct-io.c	dio: optimize cache misses in the submission path	2012-01-12 20:13:12 -08:00
drop_caches.c
eventfd.c
eventpoll.c	epoll: limit paths	2012-01-12 20:13:04 -08:00
exec.c	tracepoint: add tracepoints for debugging oom_score_adj	2012-01-10 16:30:44 -08:00
fcntl.c
fhandle.c	vfs: prefer ->dentry->d_sb to ->mnt->mnt_sb	2012-01-06 23:16:53 -05:00
fifo.c
file_table.c	vfs: prevent remount read-only if pending removes	2012-01-06 23:20:13 -05:00
file.c
filesystems.c	vfs: convert fs_supers to hlist	2012-01-03 22:52:39 -05:00
fs_struct.c
fs-writeback.c	Merge branch 'writeback-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/wfg/linux	2012-01-10 16:59:59 -08:00
generic_acl.c	switch posix_acl_equiv_mode() to umode_t *	2011-08-01 02:10:06 -04:00
inode.c	mm: account reaped page cache on inode cache pruning	2012-01-10 16:30:42 -08:00
internal.h	vfs: protect remounting superblock read-only	2012-01-06 23:20:12 -05:00
ioctl.c	vfs: fix up ENOIOCTLCMD error handling	2012-01-05 15:40:12 -08:00
ioprio.c	fs: add export.h to files using EXPORT_SYMBOL/THIS_MODULE macros	2011-10-31 19:30:31 -04:00
Kconfig	Merge branch 'for-linus' of git://git.open-osd.org/linux-open-osd	2012-01-09 12:51:01 -08:00
Kconfig.binfmt	fs: binfmt_elf: create Kconfig variable for PIE randomization	2012-01-10 16:30:51 -08:00
libfs.c	fs: move code out of buffer.c	2012-01-03 22:54:07 -05:00
locks.c	vfs: fix handling of lock allocation failure in lease-break case	2011-12-26 10:25:26 -08:00
Makefile	Merge branches 'vfsmount-guts', 'umode_t' and 'partitions' into Z	2012-01-06 23:15:54 -05:00
mbcache.c
mount.h	vfs: keep list of mounts for each superblock	2012-01-06 23:20:12 -05:00
mpage.c
namei.c	Merge branches 'vfsmount-guts', 'umode_t' and 'partitions' into Z	2012-01-06 23:15:54 -05:00
namespace.c	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial	2012-01-08 13:21:22 -08:00
no-block.c
open.c	switch security_path_chmod() to struct path *	2012-01-06 23:16:53 -05:00
pipe.c	pipe: fail cleanly when root tries F_SETPIPE_SZ with big size	2012-01-12 20:13:04 -08:00
pnode.c	vfs: switch pnode.h macros to struct mount *	2012-01-03 22:57:11 -05:00
pnode.h	vfs: switch pnode.h macros to struct mount *	2012-01-03 22:57:11 -05:00
posix_acl.c	vfs: pass all mask flags check_acl and posix_acl_permission	2011-10-28 14:58:54 +02:00
proc_namespace.c	vfs: switch ->show_options() to struct dentry *	2012-01-06 23:19:54 -05:00
read_write.c	Cross Memory Attach	2011-10-31 17:30:44 -07:00
read_write.h
readdir.c
select.c
seq_file.c	constify seq_file stuff	2012-01-03 22:52:40 -05:00
signalfd.c
splice.c	fs: move code out of buffer.c	2012-01-03 22:54:07 -05:00
stack.c	filesystems: add set_nlink()	2011-11-02 12:53:43 +01:00
stat.c	readlinkat: ensure we return ENOENT for the empty pathname for normal lookups	2011-11-02 12:53:42 +01:00
statfs.c	vfs: new helper - vfs_ustat()	2012-01-03 22:53:07 -05:00
super.c	vfs: prevent remount read-only if pending removes	2012-01-06 23:20:13 -05:00
sync.c	fs: move code out of buffer.c	2012-01-03 22:54:07 -05:00
timerfd.c
utimes.c
xattr_acl.c
xattr.c	vfs: mnt_drop_write_file()	2012-01-03 22:52:40 -05:00