linux/fs/xfs
Darrick J. Wong 7561cea5db xfs: prevent a UAF when log IO errors race with unmount
KASAN reported the following use after free bug when running
generic/475:

 XFS (dm-0): Mounting V5 Filesystem
 XFS (dm-0): Starting recovery (logdev: internal)
 XFS (dm-0): Ending recovery (logdev: internal)
 Buffer I/O error on dev dm-0, logical block 20639616, async page read
 Buffer I/O error on dev dm-0, logical block 20639617, async page read
 XFS (dm-0): log I/O error -5
 XFS (dm-0): Filesystem has been shut down due to log error (0x2).
 XFS (dm-0): Unmounting Filesystem
 XFS (dm-0): Please unmount the filesystem and rectify the problem(s).
 ==================================================================
 BUG: KASAN: use-after-free in do_raw_spin_lock+0x246/0x270
 Read of size 4 at addr ffff888109dd84c4 by task 3:1H/136

 CPU: 3 PID: 136 Comm: 3:1H Not tainted 5.19.0-rc4-xfsx #rc4 8e53ab5ad0fddeb31cee5e7063ff9c361915a9c4
 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
 Workqueue: xfs-log/dm-0 xlog_ioend_work [xfs]
 Call Trace:
  <TASK>
  dump_stack_lvl+0x34/0x44
  print_report.cold+0x2b8/0x661
  ? do_raw_spin_lock+0x246/0x270
  kasan_report+0xab/0x120
  ? do_raw_spin_lock+0x246/0x270
  do_raw_spin_lock+0x246/0x270
  ? rwlock_bug.part.0+0x90/0x90
  xlog_force_shutdown+0xf6/0x370 [xfs 4ad76ae0d6add7e8183a553e624c31e9ed567318]
  xlog_ioend_work+0x100/0x190 [xfs 4ad76ae0d6add7e8183a553e624c31e9ed567318]
  process_one_work+0x672/0x1040
  worker_thread+0x59b/0xec0
  ? __kthread_parkme+0xc6/0x1f0
  ? process_one_work+0x1040/0x1040
  ? process_one_work+0x1040/0x1040
  kthread+0x29e/0x340
  ? kthread_complete_and_exit+0x20/0x20
  ret_from_fork+0x1f/0x30
  </TASK>

 Allocated by task 154099:
  kasan_save_stack+0x1e/0x40
  __kasan_kmalloc+0x81/0xa0
  kmem_alloc+0x8d/0x2e0 [xfs]
  xlog_cil_init+0x1f/0x540 [xfs]
  xlog_alloc_log+0xd1e/0x1260 [xfs]
  xfs_log_mount+0xba/0x640 [xfs]
  xfs_mountfs+0xf2b/0x1d00 [xfs]
  xfs_fs_fill_super+0x10af/0x1910 [xfs]
  get_tree_bdev+0x383/0x670
  vfs_get_tree+0x7d/0x240
  path_mount+0xdb7/0x1890
  __x64_sys_mount+0x1fa/0x270
  do_syscall_64+0x2b/0x80
  entry_SYSCALL_64_after_hwframe+0x46/0xb0

 Freed by task 154151:
  kasan_save_stack+0x1e/0x40
  kasan_set_track+0x21/0x30
  kasan_set_free_info+0x20/0x30
  ____kasan_slab_free+0x110/0x190
  slab_free_freelist_hook+0xab/0x180
  kfree+0xbc/0x310
  xlog_dealloc_log+0x1b/0x2b0 [xfs]
  xfs_unmountfs+0x119/0x200 [xfs]
  xfs_fs_put_super+0x6e/0x2e0 [xfs]
  generic_shutdown_super+0x12b/0x3a0
  kill_block_super+0x95/0xd0
  deactivate_locked_super+0x80/0x130
  cleanup_mnt+0x329/0x4d0
  task_work_run+0xc5/0x160
  exit_to_user_mode_prepare+0xd4/0xe0
  syscall_exit_to_user_mode+0x1d/0x40
  entry_SYSCALL_64_after_hwframe+0x46/0xb0

This appears to be a race between the unmount process, which frees the
CIL and waits for in-flight iclog IO; and the iclog IO completion.  When
generic/475 runs, it starts fsstress in the background, waits a few
seconds, and substitutes a dm-error device to simulate a disk falling
out of a machine.  If the fsstress encounters EIO on a pure data write,
it will exit but the filesystem will still be online.

The next thing the test does is unmount the filesystem, which tries to
clean the log, free the CIL, and wait for iclog IO completion.  If an
iclog was being written when the dm-error switch occurred, it can race
with log unmounting as follows:

Thread 1				Thread 2

					xfs_log_unmount
					xfs_log_clean
					xfs_log_quiesce
xlog_ioend_work
<observe error>
xlog_force_shutdown
test_and_set_bit(XLOG_IOERROR)
					xfs_log_force
					<log is shut down, nop>
					xfs_log_umount_write
					<log is shut down, nop>
					xlog_dealloc_log
					xlog_cil_destroy
					<wait for iclogs>
spin_lock(&log->l_cilp->xc_push_lock)
<KABOOM>

Therefore, free the CIL after waiting for the iclogs to complete.  I
/think/ this race has existed for quite a few years now, though I don't
remember the ~2014 era logging code well enough to know if it was a real
threat then or if the actual race was exposed only more recently.

Fixes: ac983517ec ("xfs: don't sleep in xlog_cil_force_lsn on shutdown")
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
2022-07-01 09:09:52 -07:00
..
libxfs xfs: don't hold xattr leaf buffers across transaction rolls 2022-06-29 08:47:56 -07:00
scrub xfs: implement per-mount warnings for scrub and shrink usage 2022-05-27 10:31:34 +10:00
Kconfig
kmem.c mm: introduce memalloc_retry_wait() 2022-01-15 16:30:29 +02:00
kmem.h xfs: remove kmem_zone typedef 2021-10-22 16:00:31 -07:00
Makefile xfs: Set up infrastructure for log attribute replay 2022-05-04 12:41:02 +10:00
mrlock.h
xfs_acl.c xfs: move xfs_attr_use_log_assist usage out of libxfs 2022-05-27 10:34:04 +10:00
xfs_acl.h xfs: improve __xfs_set_acl 2022-04-26 13:34:42 +10:00
xfs_aops.c Page cache changes for 5.19 2022-05-24 19:55:07 -07:00
xfs_aops.h
xfs_attr_inactive.c xfs: rename _zone variables to _cache 2021-10-22 16:04:20 -07:00
xfs_attr_item.c xfs: don't hold xattr leaf buffers across transaction rolls 2022-06-29 08:47:56 -07:00
xfs_attr_item.h xfs: share xattr name and value buffers when logging xattr updates 2022-05-23 08:43:46 +10:00
xfs_attr_list.c xfs: Set up infrastructure for log attribute replay 2022-05-04 12:41:02 +10:00
xfs_bio_io.c Bug fixes for 5.18: 2022-04-01 19:30:44 -07:00
xfs_bmap_item.c xfs: whiteouts release intents that are not in the AIL 2022-05-04 11:46:47 +10:00
xfs_bmap_item.h xfs: rename _zone variables to _cache 2021-10-22 16:04:20 -07:00
xfs_bmap_util.c xfs: dont treat rt extents beyond EOF as eofblocks to be cleared 2022-06-29 08:47:56 -07:00
xfs_bmap_util.h xfs: kill the XFS_IOC_{ALLOC,FREE}SP* ioctls 2022-01-17 09:16:41 -08:00
xfs_buf_item_recover.c xfs: convert buf_cancel_table allocation to kmalloc_array 2022-05-27 10:27:19 +10:00
xfs_buf_item.c xfs: log items should have a xlog pointer, not a mount 2022-03-20 08:59:49 -07:00
xfs_buf_item.h xfs: convert buffer log item flags to unsigned. 2022-04-21 10:46:40 +10:00
xfs_buf.c xfs: convert buffer flags to unsigned. 2022-04-21 08:44:59 +10:00
xfs_buf.h xfs: convert buffer flags to unsigned. 2022-04-21 08:44:59 +10:00
xfs_dir2_readdir.c xfs: take the ILOCK when readdir inspects directory mapping data 2022-01-11 15:11:04 -08:00
xfs_discard.c block: decouple REQ_OP_SECURE_ERASE from REQ_OP_DISCARD 2022-04-17 19:49:59 -06:00
xfs_discard.h
xfs_dquot_item_recover.c xfs: replace xfs_sb_version checks with feature flag checks 2021-08-19 10:07:12 -07:00
xfs_dquot_item.c
xfs_dquot_item.h
xfs_dquot.c xfs: remove warning counters from struct xfs_dquot_res 2022-05-11 17:12:09 +10:00
xfs_dquot.h xfs: remove warning counters from struct xfs_dquot_res 2022-05-11 17:12:09 +10:00
xfs_error.c xfs: add leaf to node error tag 2022-05-11 17:01:23 +10:00
xfs_error.h xfs: convert ptag flags to unsigned. 2022-04-21 10:47:25 +10:00
xfs_export.c xfs: convert remaining mount flags to state flags 2021-08-19 10:07:13 -07:00
xfs_export.h
xfs_extent_busy.c
xfs_extent_busy.h
xfs_extfree_item.c xfs: whiteouts release intents that are not in the AIL 2022-05-04 11:46:47 +10:00
xfs_extfree_item.h xfs: rename _zone variables to _cache 2021-10-22 16:04:20 -07:00
xfs_file.c xfs: Changes for 5.19-rc1 [2nd set] 2022-06-01 17:23:53 -07:00
xfs_filestream.c xfs: fix soft lockup via spinning in filestream ag selection loop 2022-04-26 13:34:54 +10:00
xfs_filestream.h xfs: convert mount flags to features 2021-08-19 10:07:12 -07:00
xfs_fsmap.c xfs: pass explicit mount pointer to rtalloc query functions 2022-04-12 06:49:41 +10:00
xfs_fsmap.h
xfs_fsops.c xfs: implement per-mount warnings for scrub and shrink usage 2022-05-27 10:31:34 +10:00
xfs_fsops.h
xfs_globals.c xfs: Add larp debug option 2022-05-11 17:01:22 +10:00
xfs_health.c xfs: replace XFS_FORCED_SHUTDOWN with xfs_is_shutdown 2021-08-19 10:07:13 -07:00
xfs_icache.c xfs: introduce xfs_inodegc_push() 2022-06-23 13:34:38 -07:00
xfs_icache.h xfs: introduce xfs_inodegc_push() 2022-06-23 13:34:38 -07:00
xfs_icreate_item.c xfs: fix potential log item leak 2022-05-04 11:45:11 +10:00
xfs_icreate_item.h xfs: rename _zone variables to _cache 2021-10-22 16:04:20 -07:00
xfs_inode_item_recover.c xfs: hide log iovec alignment constraints 2022-05-04 11:45:50 +10:00
xfs_inode_item.c xfs: hide log iovec alignment constraints 2022-05-04 11:45:50 +10:00
xfs_inode_item.h xfs: aborting inodes on shutdown may need buffer lock 2022-03-29 18:21:59 -07:00
xfs_inode.c xfs: use invalidate_lock to check the state of mmap_lock 2022-06-26 14:43:28 -07:00
xfs_inode.h Merge tag 'large-extent-counters-v9' of https://github.com/chandanr/linux into xfs-5.19-for-next 2022-04-21 16:46:17 +10:00
xfs_ioctl32.c xfs: Set up infrastructure for log attribute replay 2022-05-04 12:41:02 +10:00
xfs_ioctl32.h xfs: remove unused xfs_ioctl32.h declarations 2022-01-18 10:18:36 -08:00
xfs_ioctl.c xfs: preserve DIFLAG2_NREXT64 when setting other inode attributes 2022-06-15 23:13:33 -07:00
xfs_ioctl.h xfs: kill the XFS_IOC_{ALLOC,FREE}SP* ioctls 2022-01-17 09:16:41 -08:00
xfs_iomap.c xfs: Conditionally upgrade existing inodes to use large extent counters 2022-04-13 07:02:44 +00:00
xfs_iomap.h iomap: add a IOMAP_DAX flag 2021-12-04 08:58:53 -08:00
xfs_iops.c xfs: move xfs_attr_use_log_assist usage out of libxfs 2022-05-27 10:34:04 +10:00
xfs_iops.h
xfs_itable.c xfs: Enable bulkstat ioctl to support 64-bit per-inode extent counters 2022-04-13 07:02:45 +00:00
xfs_itable.h xfs: Enable bulkstat ioctl to support 64-bit per-inode extent counters 2022-04-13 07:02:45 +00:00
xfs_iwalk.c xfs: avoid buffer deadlocks when walking fs inodes 2021-08-09 11:13:16 -07:00
xfs_iwalk.h xfs: Decouple XFS_IBULK flags from XFS_IWALK flags 2022-04-13 07:02:44 +00:00
xfs_linux.h xfs: drop async cache flushes from CIL commits. 2022-03-29 18:22:02 -07:00
xfs_log_cil.c xfs: Changes for 5.19-rc1 2022-05-25 19:34:40 -07:00
xfs_log_priv.h xfs: refactor buffer cancellation table allocation 2022-05-27 10:26:17 +10:00
xfs_log_recover.c xfs: convert buf_cancel_table allocation to kmalloc_array 2022-05-27 10:27:19 +10:00
xfs_log.c xfs: prevent a UAF when log IO errors race with unmount 2022-07-01 09:09:52 -07:00
xfs_log.h xfs: share xattr name and value buffers when logging xattr updates 2022-05-23 08:43:46 +10:00
xfs_message.c Merge branch 'guilt/xfs-unsigned-flags-5.18' into xfs-5.19-for-next 2022-04-21 16:45:03 +10:00
xfs_message.h xfs: implement per-mount warnings for scrub and shrink usage 2022-05-27 10:31:34 +10:00
xfs_mount.c xfs: don't log every time we clear the log incompat flags 2022-05-27 10:29:51 +10:00
xfs_mount.h xfs: bound maximum wait time for inodegc work 2022-06-23 13:34:38 -07:00
xfs_mru_cache.c xfs: rename _zone variables to _cache 2021-10-22 16:04:20 -07:00
xfs_mru_cache.h
xfs_ondisk.h xfs: Set up infrastructure for log attribute replay 2022-05-04 12:41:02 +10:00
xfs_pnfs.c xfs: use setattr_copy to set vfs inode attributes 2022-03-14 10:23:16 -07:00
xfs_pnfs.h
xfs_pwork.c
xfs_pwork.h
xfs_qm_bhv.c xfs: replace xfs_sb_version checks with feature flag checks 2021-08-19 10:07:12 -07:00
xfs_qm_syscalls.c xfs: introduce xfs_inodegc_push() 2022-06-23 13:34:38 -07:00
xfs_qm.c xfs: purge dquots after inode walk fails during quotacheck 2022-05-27 10:21:43 +10:00
xfs_qm.h xfs: remove quota warning limit from struct xfs_quota_limits 2022-05-11 17:12:09 +10:00
xfs_quota.h xfs: queue inactivation immediately when quota is nearing enforcement 2021-08-09 10:52:18 -07:00
xfs_quotaops.c xfs: don't set quota warning values 2022-05-11 17:12:09 +10:00
xfs_refcount_item.c xfs: whiteouts release intents that are not in the AIL 2022-05-04 11:46:47 +10:00
xfs_refcount_item.h xfs: rename _zone variables to _cache 2021-10-22 16:04:20 -07:00
xfs_reflink.c xfs: rewrite xfs_reflink_end_cow to use intents 2022-04-28 10:25:50 -07:00
xfs_reflink.h xfs: convert xfs_sb_version_has checks to use mount features 2021-08-19 10:07:14 -07:00
xfs_rmap_item.c xfs: whiteouts release intents that are not in the AIL 2022-05-04 11:46:47 +10:00
xfs_rmap_item.h xfs: rename _zone variables to _cache 2021-10-22 16:04:20 -07:00
xfs_rtalloc.c Merge tag 'large-extent-counters-v9' of https://github.com/chandanr/linux into xfs-5.19-for-next 2022-04-21 16:46:17 +10:00
xfs_rtalloc.h xfs: recalculate free rt extents after log recovery 2022-04-12 06:49:42 +10:00
xfs_stats.c
xfs_stats.h
xfs_super.c xfs: introduce xfs_inodegc_push() 2022-06-23 13:34:38 -07:00
xfs_super.h xfs: move xfs_attr_use_log_assist out of xfs_log.c 2022-05-27 10:33:29 +10:00
xfs_symlink.c xfs: Directory's data fork extent counter can never overflow 2022-04-13 07:02:07 +00:00
xfs_symlink.h
xfs_sysctl.c
xfs_sysctl.h xfs: Add larp debug option 2022-05-11 17:01:22 +10:00
xfs_sysfs.c xfs: Add larp debug option 2022-05-11 17:01:22 +10:00
xfs_sysfs.h
xfs_trace.c xfs: add trace point for fs shutdown 2021-08-18 18:46:00 -07:00
xfs_trace.h xfs: introduce xfs_inodegc_push() 2022-06-23 13:34:38 -07:00
xfs_trans_ail.c xfs: log shutdown triggers should only shut down the log 2022-03-29 18:22:01 -07:00
xfs_trans_buf.c xfs: introduce xfs_buf_daddr() 2021-08-19 10:07:14 -07:00
xfs_trans_dquot.c xfs: remove quota warning limit from struct xfs_quota_limits 2022-05-11 17:12:09 +10:00
xfs_trans_priv.h xfs: AIL should be log centric 2022-03-20 08:59:49 -07:00
xfs_trans.c xfs: report "max_resp" used for min log size computation 2022-04-28 10:25:23 -07:00
xfs_trans.h xfs: intent item whiteouts 2022-05-04 11:50:29 +10:00
xfs_xattr.c xfs: fix TOCTOU race involving the new logged xattrs control knob 2022-06-15 23:13:32 -07:00
xfs_xattr.h xfs: move xfs_attr_use_log_assist usage out of libxfs 2022-05-27 10:34:04 +10:00
xfs.h