forked from Minki/linux
80168676eb
I have been seeing occasional pauses in transaction throughput up to 30s long under heavy parallel workloads. The only notable thing was that the xfsaild was trying to be active during the pauses, but making no progress. It was running exactly 20 times a second (on the 50ms no-progress backoff), and the number of pushbuf events was constant across this time as well. IOWs, the xfsaild appeared to be stuck on buffers that it could not push out. Further investigation indicated that it was trying to push out inode buffers that were pinned and/or locked. The xfsbufd was also getting woken at the same frequency (by the xfsaild, no doubt) to push out delayed write buffers. The xfsbufd was not making any progress because all the buffers in the delwri queue were pinned. This scan- and-make-no-progress dance went one in the trace for some seconds, before the xfssyncd came along an issued a log force, and then things started going again. However, I noticed something strange about the log force - there were way too many IO's issued. 516 log buffers were written, to be exact. That added up to 129MB of log IO, which got me very interested because it's almost exactly 25% of the size of the log. He delayed logging code is suppose to aggregate the minimum of 25% of the log or 8MB worth of changes before flushing. That's what really puzzled me - why did a log force write 129MB instead of only 8MB? Essentially what has happened is that no CIL pushes had occurred since the previous tail push which cleared out 25% of the log space. That caused all the new transactions to block because there wasn't log space for them, but they kick the xfsaild to push the tail. However, the xfsaild was not making progress because there were buffers it could not lock and flush, and the xfsbufd could not flush them because they were pinned. As a result, both the xfsaild and the xfsbufd could not move the tail of the log forward without the CIL first committing. The cause of the problem was that the background CIL push, which should happen when 8MB of aggregated changes have been committed, is being held off by the concurrent transaction commit load. The background push does a down_write_trylock() which will fail if there is a concurrent transaction commit holding the push lock in read mode. With 8 CPUs all doing transactions as fast as they can, there was enough concurrent transaction commits to hold off the background push until tail-pushing could no longer free log space, and the halt would occur. It should be noted that there is no reason why it would halt at 25% of log space used by a single CIL checkpoint. This bug could definitely violate the "no transaction should be larger than half the log" requirement and hence result in corruption if the system crashed under heavy load. This sort of bug is exactly the reason why delayed logging was tagged as experimental.... The fix is to start blocking background pushes once the threshold has been exceeded. Rework the threshold calculations to keep the amount of log space a CIL checkpoint can use to below that of the AIL push threshold to avoid the problem completely. Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Alex Elder <aelder@sgi.com> Reviewed-by: Christoph Hellwig <hch@lst.de> |
||
|---|---|---|
| .. | ||
| linux-2.6 | ||
| quota | ||
| support | ||
| Kconfig | ||
| Makefile | ||
| xfs_acl.h | ||
| xfs_ag.h | ||
| xfs_alloc_btree.c | ||
| xfs_alloc_btree.h | ||
| xfs_alloc.c | ||
| xfs_alloc.h | ||
| xfs_arch.h | ||
| xfs_attr_leaf.c | ||
| xfs_attr_leaf.h | ||
| xfs_attr_sf.h | ||
| xfs_attr.c | ||
| xfs_attr.h | ||
| xfs_bit.c | ||
| xfs_bit.h | ||
| xfs_bmap_btree.c | ||
| xfs_bmap_btree.h | ||
| xfs_bmap.c | ||
| xfs_bmap.h | ||
| xfs_btree_trace.c | ||
| xfs_btree_trace.h | ||
| xfs_btree.c | ||
| xfs_btree.h | ||
| xfs_buf_item.c | ||
| xfs_buf_item.h | ||
| xfs_da_btree.c | ||
| xfs_da_btree.h | ||
| xfs_dfrag.c | ||
| xfs_dfrag.h | ||
| xfs_dinode.h | ||
| xfs_dir2_block.c | ||
| xfs_dir2_block.h | ||
| xfs_dir2_data.c | ||
| xfs_dir2_data.h | ||
| xfs_dir2_leaf.c | ||
| xfs_dir2_leaf.h | ||
| xfs_dir2_node.c | ||
| xfs_dir2_node.h | ||
| xfs_dir2_sf.c | ||
| xfs_dir2_sf.h | ||
| xfs_dir2.c | ||
| xfs_dir2.h | ||
| xfs_error.c | ||
| xfs_error.h | ||
| xfs_extfree_item.c | ||
| xfs_extfree_item.h | ||
| xfs_filestream.c | ||
| xfs_filestream.h | ||
| xfs_fs.h | ||
| xfs_fsops.c | ||
| xfs_fsops.h | ||
| xfs_ialloc_btree.c | ||
| xfs_ialloc_btree.h | ||
| xfs_ialloc.c | ||
| xfs_ialloc.h | ||
| xfs_iget.c | ||
| xfs_inode_item.c | ||
| xfs_inode_item.h | ||
| xfs_inode.c | ||
| xfs_inode.h | ||
| xfs_inum.h | ||
| xfs_iomap.c | ||
| xfs_iomap.h | ||
| xfs_itable.c | ||
| xfs_itable.h | ||
| xfs_log_cil.c | ||
| xfs_log_priv.h | ||
| xfs_log_recover.c | ||
| xfs_log_recover.h | ||
| xfs_log.c | ||
| xfs_log.h | ||
| xfs_mount.c | ||
| xfs_mount.h | ||
| xfs_mru_cache.c | ||
| xfs_mru_cache.h | ||
| xfs_quota.h | ||
| xfs_refcache.h | ||
| xfs_rename.c | ||
| xfs_rtalloc.c | ||
| xfs_rtalloc.h | ||
| xfs_rw.c | ||
| xfs_rw.h | ||
| xfs_sb.h | ||
| xfs_trans_ail.c | ||
| xfs_trans_buf.c | ||
| xfs_trans_extfree.c | ||
| xfs_trans_inode.c | ||
| xfs_trans_priv.h | ||
| xfs_trans_space.h | ||
| xfs_trans.c | ||
| xfs_trans.h | ||
| xfs_types.h | ||
| xfs_utils.c | ||
| xfs_utils.h | ||
| xfs_vnodeops.c | ||
| xfs_vnodeops.h | ||
| xfs.h | ||