2019-05-31 08:09:56 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-only
|
2006-01-16 16:50:04 +00:00
|
|
|
/*
|
|
|
|
* Copyright (C) Sistina Software, Inc. 1997-2003 All rights reserved.
|
2006-05-18 19:09:15 +00:00
|
|
|
* Copyright (C) 2004-2006 Red Hat, Inc. All rights reserved.
|
2006-01-16 16:50:04 +00:00
|
|
|
*/
|
|
|
|
|
2014-03-06 20:10:45 +00:00
|
|
|
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
|
|
|
|
|
2006-01-16 16:50:04 +00:00
|
|
|
#include <linux/spinlock.h>
|
|
|
|
#include <linux/completion.h>
|
|
|
|
#include <linux/buffer_head.h>
|
|
|
|
#include <linux/crc32.h>
|
2006-02-27 22:23:27 +00:00
|
|
|
#include <linux/gfs2_ondisk.h>
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
#include <linux/delay.h>
|
2016-12-24 19:46:01 +00:00
|
|
|
#include <linux/uaccess.h>
|
2006-01-16 16:50:04 +00:00
|
|
|
|
|
|
|
#include "gfs2.h"
|
2006-02-27 22:23:27 +00:00
|
|
|
#include "incore.h"
|
2006-01-16 16:50:04 +00:00
|
|
|
#include "glock.h"
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
#include "glops.h"
|
|
|
|
#include "log.h"
|
2019-02-18 15:37:25 +00:00
|
|
|
#include "lops.h"
|
|
|
|
#include "recovery.h"
|
2018-08-15 17:09:49 +00:00
|
|
|
#include "rgrp.h"
|
2019-02-18 15:37:25 +00:00
|
|
|
#include "super.h"
|
2006-02-27 22:23:27 +00:00
|
|
|
#include "util.h"
|
2006-01-16 16:50:04 +00:00
|
|
|
|
2006-12-07 04:33:20 +00:00
|
|
|
struct kmem_cache *gfs2_glock_cachep __read_mostly;
|
2009-12-08 12:12:13 +00:00
|
|
|
struct kmem_cache *gfs2_glock_aspace_cachep __read_mostly;
|
2006-12-07 04:33:20 +00:00
|
|
|
struct kmem_cache *gfs2_inode_cachep __read_mostly;
|
|
|
|
struct kmem_cache *gfs2_bufdata_cachep __read_mostly;
|
2008-01-28 23:20:26 +00:00
|
|
|
struct kmem_cache *gfs2_rgrpd_cachep __read_mostly;
|
2008-11-17 14:25:37 +00:00
|
|
|
struct kmem_cache *gfs2_quotad_cachep __read_mostly;
|
2015-10-26 15:40:28 +00:00
|
|
|
struct kmem_cache *gfs2_qadata_cachep __read_mostly;
|
2019-04-17 18:04:27 +00:00
|
|
|
struct kmem_cache *gfs2_trans_cachep __read_mostly;
|
2012-04-16 08:28:31 +00:00
|
|
|
mempool_t *gfs2_page_pool __read_mostly;
|
2006-01-16 16:50:04 +00:00
|
|
|
|
|
|
|
void gfs2_assert_i(struct gfs2_sbd *sdp)
|
|
|
|
{
|
2014-03-06 20:10:46 +00:00
|
|
|
fs_emerg(sdp, "fatal assertion failed\n");
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
2019-02-18 15:37:25 +00:00
|
|
|
/**
|
|
|
|
* check_journal_clean - Make sure a journal is clean for a spectator mount
|
|
|
|
* @sdp: The GFS2 superblock
|
|
|
|
* @jd: The journal descriptor
|
2021-03-30 16:44:29 +00:00
|
|
|
* @verbose: Show more prints in the log
|
2019-02-18 15:37:25 +00:00
|
|
|
*
|
|
|
|
* Returns: 0 if the journal is clean or locked, else an error
|
|
|
|
*/
|
2019-02-18 20:04:13 +00:00
|
|
|
int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd,
|
|
|
|
bool verbose)
|
2019-02-18 15:37:25 +00:00
|
|
|
{
|
|
|
|
int error;
|
|
|
|
struct gfs2_holder j_gh;
|
|
|
|
struct gfs2_log_header_host head;
|
|
|
|
struct gfs2_inode *ip;
|
|
|
|
|
|
|
|
ip = GFS2_I(jd->jd_inode);
|
|
|
|
error = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, LM_FLAG_NOEXP |
|
|
|
|
GL_EXACT | GL_NOCACHE, &j_gh);
|
|
|
|
if (error) {
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
fs_err(sdp, "Error %d locking journal for spectator "
|
|
|
|
"mount.\n", error);
|
2019-02-18 15:37:25 +00:00
|
|
|
return -EPERM;
|
|
|
|
}
|
|
|
|
error = gfs2_jdesc_check(jd);
|
|
|
|
if (error) {
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
fs_err(sdp, "Error checking journal for spectator "
|
|
|
|
"mount.\n");
|
2019-02-18 15:37:25 +00:00
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
error = gfs2_find_jhead(jd, &head, false);
|
|
|
|
if (error) {
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
fs_err(sdp, "Error parsing journal for spectator "
|
|
|
|
"mount.\n");
|
2019-02-18 15:37:25 +00:00
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
if (!(head.lh_flags & GFS2_LOG_HEAD_UNMOUNT)) {
|
|
|
|
error = -EPERM;
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
fs_err(sdp, "jid=%u: Journal is dirty, so the first "
|
|
|
|
"mounter must not be a spectator.\n",
|
|
|
|
jd->jd_jid);
|
2019-02-18 15:37:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
out_unlock:
|
|
|
|
gfs2_glock_dq_uninit(&j_gh);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2020-12-22 20:43:27 +00:00
|
|
|
/**
|
|
|
|
* gfs2_freeze_lock - hold the freeze glock
|
|
|
|
* @sdp: the superblock
|
|
|
|
* @freeze_gh: pointer to the requested holder
|
|
|
|
* @caller_flags: any additional flags needed by the caller
|
|
|
|
*/
|
|
|
|
int gfs2_freeze_lock(struct gfs2_sbd *sdp, struct gfs2_holder *freeze_gh,
|
|
|
|
int caller_flags)
|
|
|
|
{
|
|
|
|
int flags = LM_FLAG_NOEXP | GL_EXACT | caller_flags;
|
|
|
|
int error;
|
|
|
|
|
|
|
|
error = gfs2_glock_nq_init(sdp->sd_freeze_gl, LM_ST_SHARED, flags,
|
|
|
|
freeze_gh);
|
|
|
|
if (error && error != GLR_TRYFAILED)
|
|
|
|
fs_err(sdp, "can't lock the freeze lock: %d\n", error);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
|
|
|
void gfs2_freeze_unlock(struct gfs2_holder *freeze_gh)
|
|
|
|
{
|
|
|
|
if (gfs2_holder_initialized(freeze_gh))
|
|
|
|
gfs2_glock_dq_uninit(freeze_gh);
|
|
|
|
}
|
|
|
|
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
static void signal_our_withdraw(struct gfs2_sbd *sdp)
|
|
|
|
{
|
2021-01-18 20:18:59 +00:00
|
|
|
struct gfs2_glock *live_gl = sdp->sd_live_gh.gh_gl;
|
2021-03-12 12:58:54 +00:00
|
|
|
struct inode *inode;
|
|
|
|
struct gfs2_inode *ip;
|
|
|
|
struct gfs2_glock *i_gl;
|
|
|
|
u64 no_formal_ino;
|
2020-12-22 20:43:28 +00:00
|
|
|
int log_write_allowed = test_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
int ret = 0;
|
|
|
|
int tries;
|
|
|
|
|
2021-03-12 12:58:54 +00:00
|
|
|
if (test_bit(SDF_NORECOVERY, &sdp->sd_flags) || !sdp->sd_jdesc)
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
return;
|
|
|
|
|
2021-05-19 18:54:02 +00:00
|
|
|
gfs2_ail_drain(sdp); /* frees all transactions */
|
2021-03-12 12:58:54 +00:00
|
|
|
inode = sdp->sd_jdesc->jd_inode;
|
|
|
|
ip = GFS2_I(inode);
|
|
|
|
i_gl = ip->i_gl;
|
|
|
|
no_formal_ino = ip->i_no_formal_ino;
|
|
|
|
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
/* Prevent any glock dq until withdraw recovery is complete */
|
|
|
|
set_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags);
|
|
|
|
/*
|
|
|
|
* Don't tell dlm we're bailing until we have no more buffers in the
|
|
|
|
* wind. If journal had an IO error, the log code should just purge
|
|
|
|
* the outstanding buffers rather than submitting new IO. Making the
|
|
|
|
* file system read-only will flush the journal, etc.
|
|
|
|
*
|
|
|
|
* During a normal unmount, gfs2_make_fs_ro calls gfs2_log_shutdown
|
|
|
|
* which clears SDF_JOURNAL_LIVE. In a withdraw, we must not write
|
|
|
|
* any UNMOUNT log header, so we can't call gfs2_log_shutdown, and
|
|
|
|
* therefore we need to clear SDF_JOURNAL_LIVE manually.
|
|
|
|
*/
|
|
|
|
clear_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
|
2020-12-22 20:43:28 +00:00
|
|
|
if (!sb_rdonly(sdp->sd_vfs)) {
|
|
|
|
struct gfs2_holder freeze_gh;
|
|
|
|
|
|
|
|
gfs2_holder_mark_uninitialized(&freeze_gh);
|
|
|
|
if (sdp->sd_freeze_gl &&
|
|
|
|
!gfs2_glock_is_locked_by_me(sdp->sd_freeze_gl)) {
|
|
|
|
ret = gfs2_freeze_lock(sdp, &freeze_gh,
|
|
|
|
log_write_allowed ? 0 : LM_FLAG_TRY);
|
|
|
|
if (ret == GLR_TRYFAILED)
|
|
|
|
ret = 0;
|
|
|
|
}
|
|
|
|
if (!ret)
|
2021-03-04 14:28:57 +00:00
|
|
|
gfs2_make_fs_ro(sdp);
|
2020-12-22 20:43:28 +00:00
|
|
|
gfs2_freeze_unlock(&freeze_gh);
|
|
|
|
}
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
|
gfs2: Fix BUG during unmount after file system withdraw
Before this patch, when the logd daemon was forced to withdraw, it
would try to request its journal be recovered by another cluster node.
However, in single-user cases with lock_nolock, there are no other
nodes to recover the journal. Function signal_our_withdraw() was
recognizing the lock_nolock situation, but not until after it had
evicted its journal inode. Since the journal descriptor that points
to the inode was never removed from the master list, when the unmount
occurred, it did another iput on the evicted inode, which resulted in
a BUG_ON(inode->i_state & I_CLEAR).
This patch moves the check for this situation earlier in function
signal_our_withdraw(), which avoids the extra iput, so the unmount
may happen normally.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2020-04-24 17:15:21 +00:00
|
|
|
if (sdp->sd_lockstruct.ls_ops->lm_lock == NULL) { /* lock_nolock */
|
|
|
|
if (!ret)
|
|
|
|
ret = -EIO;
|
|
|
|
clear_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags);
|
|
|
|
goto skip_recovery;
|
|
|
|
}
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
/*
|
|
|
|
* Drop the glock for our journal so another node can recover it.
|
|
|
|
*/
|
|
|
|
if (gfs2_holder_initialized(&sdp->sd_journal_gh)) {
|
|
|
|
gfs2_glock_dq_wait(&sdp->sd_journal_gh);
|
|
|
|
gfs2_holder_uninit(&sdp->sd_journal_gh);
|
|
|
|
}
|
|
|
|
sdp->sd_jinode_gh.gh_flags |= GL_NOCACHE;
|
|
|
|
gfs2_glock_dq(&sdp->sd_jinode_gh);
|
|
|
|
if (test_bit(SDF_FS_FROZEN, &sdp->sd_flags)) {
|
|
|
|
/* Make sure gfs2_unfreeze works if partially-frozen */
|
2020-12-03 13:51:41 +00:00
|
|
|
flush_work(&sdp->sd_freeze_work);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
atomic_set(&sdp->sd_freeze_state, SFS_FROZEN);
|
|
|
|
thaw_super(sdp->sd_vfs);
|
|
|
|
} else {
|
2021-01-18 20:18:59 +00:00
|
|
|
wait_on_bit(&i_gl->gl_flags, GLF_DEMOTE,
|
|
|
|
TASK_UNINTERRUPTIBLE);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* holder_uninit to force glock_put, to force dlm to let go
|
|
|
|
*/
|
|
|
|
gfs2_holder_uninit(&sdp->sd_jinode_gh);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Note: We need to be careful here:
|
|
|
|
* Our iput of jd_inode will evict it. The evict will dequeue its
|
|
|
|
* glock, but the glock dq will wait for the withdraw unless we have
|
|
|
|
* exception code in glock_dq.
|
|
|
|
*/
|
|
|
|
iput(inode);
|
|
|
|
/*
|
|
|
|
* Wait until the journal inode's glock is freed. This allows try locks
|
|
|
|
* on other nodes to be successful, otherwise we remain the owner of
|
|
|
|
* the glock as far as dlm is concerned.
|
|
|
|
*/
|
2021-01-18 20:18:59 +00:00
|
|
|
if (i_gl->gl_ops->go_free) {
|
|
|
|
set_bit(GLF_FREEING, &i_gl->gl_flags);
|
|
|
|
wait_on_bit(&i_gl->gl_flags, GLF_FREEING, TASK_UNINTERRUPTIBLE);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Dequeue the "live" glock, but keep a reference so it's never freed.
|
|
|
|
*/
|
2021-01-18 20:18:59 +00:00
|
|
|
gfs2_glock_hold(live_gl);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
gfs2_glock_dq_wait(&sdp->sd_live_gh);
|
|
|
|
/*
|
|
|
|
* We enqueue the "live" glock in EX so that all other nodes
|
|
|
|
* get a demote request and act on it. We don't really want the
|
|
|
|
* lock in EX, so we send a "try" lock with 1CB to produce a callback.
|
|
|
|
*/
|
|
|
|
fs_warn(sdp, "Requesting recovery of jid %d.\n",
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
|
|
|
gfs2_holder_reinit(LM_ST_EXCLUSIVE, LM_FLAG_TRY_1CB | LM_FLAG_NOEXP,
|
|
|
|
&sdp->sd_live_gh);
|
|
|
|
msleep(GL_GLOCK_MAX_HOLD);
|
|
|
|
/*
|
|
|
|
* This will likely fail in a cluster, but succeed standalone:
|
|
|
|
*/
|
|
|
|
ret = gfs2_glock_nq(&sdp->sd_live_gh);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we actually got the "live" lock in EX mode, there are no other
|
|
|
|
* nodes available to replay our journal. So we try to replay it
|
|
|
|
* ourselves. We hold the "live" glock to prevent other mounters
|
|
|
|
* during recovery, then just dequeue it and reacquire it in our
|
|
|
|
* normal SH mode. Just in case the problem that caused us to
|
|
|
|
* withdraw prevents us from recovering our journal (e.g. io errors
|
|
|
|
* and such) we still check if the journal is clean before proceeding
|
|
|
|
* but we may wait forever until another mounter does the recovery.
|
|
|
|
*/
|
|
|
|
if (ret == 0) {
|
|
|
|
fs_warn(sdp, "No other mounters found. Trying to recover our "
|
|
|
|
"own journal jid %d.\n", sdp->sd_lockstruct.ls_jid);
|
|
|
|
if (gfs2_recover_journal(sdp->sd_jdesc, 1))
|
|
|
|
fs_warn(sdp, "Unable to recover our journal jid %d.\n",
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
|
|
|
gfs2_glock_dq_wait(&sdp->sd_live_gh);
|
|
|
|
gfs2_holder_reinit(LM_ST_SHARED, LM_FLAG_NOEXP | GL_EXACT,
|
|
|
|
&sdp->sd_live_gh);
|
|
|
|
gfs2_glock_nq(&sdp->sd_live_gh);
|
|
|
|
}
|
|
|
|
|
2021-01-18 20:18:59 +00:00
|
|
|
gfs2_glock_queue_put(live_gl); /* drop extra reference we acquired */
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
clear_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* At this point our journal is evicted, so we need to get a new inode
|
|
|
|
* for it. Once done, we need to call gfs2_find_jhead which
|
|
|
|
* calls gfs2_map_journal_extents to map it for us again.
|
|
|
|
*
|
|
|
|
* Note that we don't really want it to look up a FREE block. The
|
|
|
|
* GFS2_BLKST_FREE simply overrides a block check in gfs2_inode_lookup
|
|
|
|
* which would otherwise fail because it requires grabbing an rgrp
|
|
|
|
* glock, which would fail with -EIO because we're withdrawing.
|
|
|
|
*/
|
|
|
|
inode = gfs2_inode_lookup(sdp->sd_vfs, DT_UNKNOWN,
|
|
|
|
sdp->sd_jdesc->jd_no_addr, no_formal_ino,
|
|
|
|
GFS2_BLKST_FREE);
|
|
|
|
if (IS_ERR(inode)) {
|
|
|
|
fs_warn(sdp, "Reprocessing of jid %d failed with %ld.\n",
|
|
|
|
sdp->sd_lockstruct.ls_jid, PTR_ERR(inode));
|
|
|
|
goto skip_recovery;
|
|
|
|
}
|
|
|
|
sdp->sd_jdesc->jd_inode = inode;
|
2021-07-30 17:40:25 +00:00
|
|
|
d_mark_dontcache(inode);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Now wait until recovery is complete.
|
|
|
|
*/
|
|
|
|
for (tries = 0; tries < 10; tries++) {
|
2019-02-18 20:04:13 +00:00
|
|
|
ret = check_journal_clean(sdp, sdp->sd_jdesc, false);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
if (!ret)
|
|
|
|
break;
|
|
|
|
msleep(HZ);
|
|
|
|
fs_warn(sdp, "Waiting for journal recovery jid %d.\n",
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
|
|
|
}
|
|
|
|
skip_recovery:
|
|
|
|
if (!ret)
|
|
|
|
fs_warn(sdp, "Journal recovery complete for jid %d.\n",
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
|
|
|
else
|
2021-07-26 17:53:12 +00:00
|
|
|
fs_warn(sdp, "Journal recovery skipped for jid %d until next "
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
"mount.\n", sdp->sd_lockstruct.ls_jid);
|
|
|
|
fs_warn(sdp, "Glock dequeues delayed: %lu\n", sdp->sd_glock_dqs_held);
|
|
|
|
sdp->sd_glock_dqs_held = 0;
|
|
|
|
wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_RECOVERY);
|
|
|
|
}
|
|
|
|
|
2020-01-23 17:41:00 +00:00
|
|
|
void gfs2_lm(struct gfs2_sbd *sdp, const char *fmt, ...)
|
|
|
|
{
|
|
|
|
struct va_format vaf;
|
|
|
|
va_list args;
|
|
|
|
|
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW &&
|
|
|
|
test_bit(SDF_WITHDRAWN, &sdp->sd_flags))
|
|
|
|
return;
|
|
|
|
|
|
|
|
va_start(args, fmt);
|
|
|
|
vaf.fmt = fmt;
|
|
|
|
vaf.va = &args;
|
|
|
|
fs_err(sdp, "%pV", &vaf);
|
|
|
|
va_end(args);
|
|
|
|
}
|
|
|
|
|
|
|
|
int gfs2_withdraw(struct gfs2_sbd *sdp)
|
2008-01-30 15:34:04 +00:00
|
|
|
{
|
2009-01-12 10:43:39 +00:00
|
|
|
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
|
|
|
|
const struct lm_lockops *lm = ls->ls_ops;
|
2008-01-30 15:34:04 +00:00
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW &&
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
test_and_set_bit(SDF_WITHDRAWN, &sdp->sd_flags)) {
|
|
|
|
if (!test_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags))
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_IN_PROG,
|
|
|
|
TASK_UNINTERRUPTIBLE);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
set_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags);
|
2008-01-30 15:34:04 +00:00
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW) {
|
|
|
|
fs_err(sdp, "about to withdraw this file system\n");
|
|
|
|
BUG_ON(sdp->sd_args.ar_debug);
|
2008-01-30 15:34:04 +00:00
|
|
|
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
signal_our_withdraw(sdp);
|
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
kobject_uevent(&sdp->sd_kobj, KOBJ_OFFLINE);
|
2009-01-12 10:43:39 +00:00
|
|
|
|
2013-02-13 12:21:40 +00:00
|
|
|
if (!strcmp(sdp->sd_lockstruct.ls_ops->lm_proto_name, "lock_dlm"))
|
|
|
|
wait_for_completion(&sdp->sd_wdack);
|
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (lm->lm_unmount) {
|
|
|
|
fs_err(sdp, "telling LM to unmount\n");
|
|
|
|
lm->lm_unmount(sdp);
|
|
|
|
}
|
2016-03-23 18:29:59 +00:00
|
|
|
set_bit(SDF_SKIP_DLM_UNLOCK, &sdp->sd_flags);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
fs_err(sdp, "File system withdrawn\n");
|
2009-08-24 09:44:18 +00:00
|
|
|
dump_stack();
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
clear_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags);
|
|
|
|
smp_mb__after_atomic();
|
|
|
|
wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_IN_PROG);
|
2009-01-12 10:43:39 +00:00
|
|
|
}
|
2009-08-24 09:44:18 +00:00
|
|
|
|
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
|
2014-03-06 20:10:45 +00:00
|
|
|
panic("GFS2: fsid=%s: panic requested\n", sdp->sd_fsname);
|
2008-01-30 15:34:04 +00:00
|
|
|
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_assert_withdraw_i - Cause the machine to withdraw if @assertion is false
|
|
|
|
*/
|
|
|
|
|
2020-01-23 18:36:21 +00:00
|
|
|
void gfs2_assert_withdraw_i(struct gfs2_sbd *sdp, char *assertion,
|
2020-01-08 17:37:30 +00:00
|
|
|
const char *function, char *file, unsigned int line,
|
|
|
|
bool delayed)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2020-01-08 17:37:30 +00:00
|
|
|
if (gfs2_withdrawn(sdp))
|
|
|
|
return;
|
|
|
|
|
|
|
|
fs_err(sdp,
|
|
|
|
"fatal: assertion \"%s\" failed\n"
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
assertion, function, file, line);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If errors=panic was specified on mount, it won't help to delay the
|
|
|
|
* withdraw.
|
|
|
|
*/
|
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
|
|
|
|
delayed = false;
|
|
|
|
|
|
|
|
if (delayed)
|
|
|
|
gfs2_withdraw_delayed(sdp);
|
|
|
|
else
|
|
|
|
gfs2_withdraw(sdp);
|
2006-02-08 11:50:51 +00:00
|
|
|
dump_stack();
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_assert_warn_i - Print a message to the console if @assertion is false
|
|
|
|
*/
|
|
|
|
|
2020-01-23 18:36:21 +00:00
|
|
|
void gfs2_assert_warn_i(struct gfs2_sbd *sdp, char *assertion,
|
|
|
|
const char *function, char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
|
|
|
if (time_before(jiffies,
|
|
|
|
sdp->sd_last_warning +
|
|
|
|
gfs2_tune_get(sdp, gt_complain_secs) * HZ))
|
2020-01-23 18:36:21 +00:00
|
|
|
return;
|
2006-01-16 16:50:04 +00:00
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW)
|
2014-03-06 20:10:46 +00:00
|
|
|
fs_warn(sdp, "warning: assertion \"%s\" failed at function = %s, file = %s, line = %u\n",
|
|
|
|
assertion, function, file, line);
|
2006-01-16 16:50:04 +00:00
|
|
|
|
|
|
|
if (sdp->sd_args.ar_debug)
|
|
|
|
BUG();
|
2006-02-08 11:50:51 +00:00
|
|
|
else
|
|
|
|
dump_stack();
|
2006-01-16 16:50:04 +00:00
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
|
|
|
|
panic("GFS2: fsid=%s: warning: assertion \"%s\" failed\n"
|
|
|
|
"GFS2: fsid=%s: function = %s, file = %s, line = %u\n",
|
|
|
|
sdp->sd_fsname, assertion,
|
|
|
|
sdp->sd_fsname, function, file, line);
|
|
|
|
|
2006-01-16 16:50:04 +00:00
|
|
|
sdp->sd_last_warning = jiffies;
|
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_consist_i - Flag a filesystem consistency error and withdraw
|
|
|
|
*/
|
|
|
|
|
2020-01-23 18:31:06 +00:00
|
|
|
void gfs2_consist_i(struct gfs2_sbd *sdp, const char *function,
|
|
|
|
char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_lm(sdp,
|
|
|
|
"fatal: filesystem consistency error - function = %s, file = %s, line = %u\n",
|
|
|
|
function, file, line);
|
2020-01-23 18:31:06 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_consist_inode_i - Flag an inode consistency error and withdraw
|
|
|
|
*/
|
|
|
|
|
2020-01-23 18:31:06 +00:00
|
|
|
void gfs2_consist_inode_i(struct gfs2_inode *ip,
|
|
|
|
const char *function, char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2006-06-14 19:32:57 +00:00
|
|
|
struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode);
|
2020-01-23 17:41:00 +00:00
|
|
|
|
|
|
|
gfs2_lm(sdp,
|
|
|
|
"fatal: filesystem consistency error\n"
|
|
|
|
" inode = %llu %llu\n"
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
(unsigned long long)ip->i_no_formal_ino,
|
|
|
|
(unsigned long long)ip->i_no_addr,
|
|
|
|
function, file, line);
|
2020-01-23 18:31:06 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_consist_rgrpd_i - Flag a RG consistency error and withdraw
|
|
|
|
*/
|
|
|
|
|
2020-01-23 18:31:06 +00:00
|
|
|
void gfs2_consist_rgrpd_i(struct gfs2_rgrpd *rgd,
|
|
|
|
const char *function, char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
|
|
|
struct gfs2_sbd *sdp = rgd->rd_sbd;
|
2019-08-13 13:25:15 +00:00
|
|
|
char fs_id_buf[sizeof(sdp->sd_fsname) + 7];
|
2018-08-15 17:09:49 +00:00
|
|
|
|
2019-05-09 14:21:48 +00:00
|
|
|
sprintf(fs_id_buf, "fsid=%s: ", sdp->sd_fsname);
|
2020-10-07 11:30:58 +00:00
|
|
|
gfs2_rgrp_dump(NULL, rgd, fs_id_buf);
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_lm(sdp,
|
|
|
|
"fatal: filesystem consistency error\n"
|
|
|
|
" RG = %llu\n"
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
(unsigned long long)rgd->rd_addr,
|
|
|
|
function, file, line);
|
2020-01-23 18:31:06 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_meta_check_ii - Flag a magic number consistency error and withdraw
|
|
|
|
* Returns: -1 if this call withdrew the machine,
|
|
|
|
* -2 if it was already withdrawn
|
|
|
|
*/
|
|
|
|
|
|
|
|
int gfs2_meta_check_ii(struct gfs2_sbd *sdp, struct buffer_head *bh,
|
|
|
|
const char *type, const char *function, char *file,
|
|
|
|
unsigned int line)
|
|
|
|
{
|
|
|
|
int me;
|
2020-01-23 17:41:00 +00:00
|
|
|
|
|
|
|
gfs2_lm(sdp,
|
|
|
|
"fatal: invalid metadata block\n"
|
|
|
|
" bh = %llu (%s)\n"
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
(unsigned long long)bh->b_blocknr, type,
|
|
|
|
function, file, line);
|
|
|
|
me = gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
return (me) ? -1 : -2;
|
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_metatype_check_ii - Flag a metadata type consistency error and withdraw
|
|
|
|
* Returns: -1 if this call withdrew the machine,
|
|
|
|
* -2 if it was already withdrawn
|
|
|
|
*/
|
|
|
|
|
|
|
|
int gfs2_metatype_check_ii(struct gfs2_sbd *sdp, struct buffer_head *bh,
|
2006-09-04 16:49:07 +00:00
|
|
|
u16 type, u16 t, const char *function,
|
2006-01-16 16:50:04 +00:00
|
|
|
char *file, unsigned int line)
|
|
|
|
{
|
|
|
|
int me;
|
2020-01-23 17:41:00 +00:00
|
|
|
|
|
|
|
gfs2_lm(sdp,
|
|
|
|
"fatal: invalid metadata block\n"
|
|
|
|
" bh = %llu (type: exp=%u, found=%u)\n"
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
(unsigned long long)bh->b_blocknr, type, t,
|
|
|
|
function, file, line);
|
|
|
|
me = gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
return (me) ? -1 : -2;
|
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_io_error_i - Flag an I/O error and withdraw
|
|
|
|
* Returns: -1 if this call withdrew the machine,
|
|
|
|
* 0 if it was already withdrawn
|
|
|
|
*/
|
|
|
|
|
|
|
|
int gfs2_io_error_i(struct gfs2_sbd *sdp, const char *function, char *file,
|
|
|
|
unsigned int line)
|
|
|
|
{
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_lm(sdp,
|
|
|
|
"fatal: I/O error\n"
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
function, file, line);
|
|
|
|
return gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2018-06-07 10:56:46 +00:00
|
|
|
* gfs2_io_error_bh_i - Flag a buffer I/O error
|
|
|
|
* @withdraw: withdraw the filesystem
|
2006-01-16 16:50:04 +00:00
|
|
|
*/
|
|
|
|
|
2018-06-07 10:56:46 +00:00
|
|
|
void gfs2_io_error_bh_i(struct gfs2_sbd *sdp, struct buffer_head *bh,
|
|
|
|
const char *function, char *file, unsigned int line,
|
|
|
|
bool withdraw)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2019-02-12 20:43:55 +00:00
|
|
|
if (gfs2_withdrawn(sdp))
|
|
|
|
return;
|
|
|
|
|
|
|
|
fs_err(sdp, "fatal: I/O error\n"
|
|
|
|
" block = %llu\n"
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
(unsigned long long)bh->b_blocknr, function, file, line);
|
2018-06-07 10:56:46 +00:00
|
|
|
if (withdraw)
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|