2019-05-31 08:09:56 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-only
|
2006-01-16 16:50:04 +00:00
|
|
|
/*
|
|
|
|
|
* Copyright (C) Sistina Software, Inc. 1997-2003 All rights reserved.
|
2006-05-18 19:09:15 +00:00
|
|
|
* Copyright (C) 2004-2006 Red Hat, Inc. All rights reserved.
|
2006-01-16 16:50:04 +00:00
|
|
|
*/
|
|
|
|
|
|
2014-03-06 20:10:45 +00:00
|
|
|
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
|
|
|
|
|
|
2006-01-16 16:50:04 +00:00
|
|
|
#include <linux/spinlock.h>
|
|
|
|
|
#include <linux/completion.h>
|
|
|
|
|
#include <linux/buffer_head.h>
|
2023-08-28 14:39:20 +00:00
|
|
|
#include <linux/kthread.h>
|
2006-01-16 16:50:04 +00:00
|
|
|
#include <linux/crc32.h>
|
2006-02-27 22:23:27 +00:00
|
|
|
#include <linux/gfs2_ondisk.h>
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
#include <linux/delay.h>
|
2016-12-24 19:46:01 +00:00
|
|
|
#include <linux/uaccess.h>
|
2006-01-16 16:50:04 +00:00
|
|
|
|
|
|
|
|
#include "gfs2.h"
|
2006-02-27 22:23:27 +00:00
|
|
|
#include "incore.h"
|
2006-01-16 16:50:04 +00:00
|
|
|
#include "glock.h"
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
#include "glops.h"
|
|
|
|
|
#include "log.h"
|
2019-02-18 15:37:25 +00:00
|
|
|
#include "lops.h"
|
|
|
|
|
#include "recovery.h"
|
2018-08-15 17:09:49 +00:00
|
|
|
#include "rgrp.h"
|
2019-02-18 15:37:25 +00:00
|
|
|
#include "super.h"
|
2006-02-27 22:23:27 +00:00
|
|
|
#include "util.h"
|
2006-01-16 16:50:04 +00:00
|
|
|
|
2006-12-07 04:33:20 +00:00
|
|
|
struct kmem_cache *gfs2_glock_cachep __read_mostly;
|
2009-12-08 12:12:13 +00:00
|
|
|
struct kmem_cache *gfs2_glock_aspace_cachep __read_mostly;
|
2006-12-07 04:33:20 +00:00
|
|
|
struct kmem_cache *gfs2_inode_cachep __read_mostly;
|
|
|
|
|
struct kmem_cache *gfs2_bufdata_cachep __read_mostly;
|
2008-01-28 23:20:26 +00:00
|
|
|
struct kmem_cache *gfs2_rgrpd_cachep __read_mostly;
|
2008-11-17 14:25:37 +00:00
|
|
|
struct kmem_cache *gfs2_quotad_cachep __read_mostly;
|
2015-10-26 15:40:28 +00:00
|
|
|
struct kmem_cache *gfs2_qadata_cachep __read_mostly;
|
2019-04-17 18:04:27 +00:00
|
|
|
struct kmem_cache *gfs2_trans_cachep __read_mostly;
|
2012-04-16 08:28:31 +00:00
|
|
|
mempool_t *gfs2_page_pool __read_mostly;
|
2006-01-16 16:50:04 +00:00
|
|
|
|
|
|
|
|
void gfs2_assert_i(struct gfs2_sbd *sdp)
|
|
|
|
|
{
|
2014-03-06 20:10:46 +00:00
|
|
|
fs_emerg(sdp, "fatal assertion failed\n");
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
|
2019-02-18 15:37:25 +00:00
|
|
|
/**
|
|
|
|
|
* check_journal_clean - Make sure a journal is clean for a spectator mount
|
|
|
|
|
* @sdp: The GFS2 superblock
|
|
|
|
|
* @jd: The journal descriptor
|
2021-03-30 16:44:29 +00:00
|
|
|
* @verbose: Show more prints in the log
|
2019-02-18 15:37:25 +00:00
|
|
|
*
|
|
|
|
|
* Returns: 0 if the journal is clean or locked, else an error
|
|
|
|
|
*/
|
2019-02-18 20:04:13 +00:00
|
|
|
int check_journal_clean(struct gfs2_sbd *sdp, struct gfs2_jdesc *jd,
|
|
|
|
|
bool verbose)
|
2019-02-18 15:37:25 +00:00
|
|
|
{
|
|
|
|
|
int error;
|
|
|
|
|
struct gfs2_holder j_gh;
|
|
|
|
|
struct gfs2_log_header_host head;
|
|
|
|
|
struct gfs2_inode *ip;
|
|
|
|
|
|
|
|
|
|
ip = GFS2_I(jd->jd_inode);
|
|
|
|
|
error = gfs2_glock_nq_init(ip->i_gl, LM_ST_SHARED, LM_FLAG_NOEXP |
|
|
|
|
|
GL_EXACT | GL_NOCACHE, &j_gh);
|
|
|
|
|
if (error) {
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
|
fs_err(sdp, "Error %d locking journal for spectator "
|
|
|
|
|
"mount.\n", error);
|
2019-02-18 15:37:25 +00:00
|
|
|
return -EPERM;
|
|
|
|
|
}
|
|
|
|
|
error = gfs2_jdesc_check(jd);
|
|
|
|
|
if (error) {
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
|
fs_err(sdp, "Error checking journal for spectator "
|
|
|
|
|
"mount.\n");
|
2019-02-18 15:37:25 +00:00
|
|
|
goto out_unlock;
|
|
|
|
|
}
|
|
|
|
|
error = gfs2_find_jhead(jd, &head, false);
|
|
|
|
|
if (error) {
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
|
fs_err(sdp, "Error parsing journal for spectator "
|
|
|
|
|
"mount.\n");
|
2019-02-18 15:37:25 +00:00
|
|
|
goto out_unlock;
|
|
|
|
|
}
|
|
|
|
|
if (!(head.lh_flags & GFS2_LOG_HEAD_UNMOUNT)) {
|
|
|
|
|
error = -EPERM;
|
2019-02-18 20:04:13 +00:00
|
|
|
if (verbose)
|
|
|
|
|
fs_err(sdp, "jid=%u: Journal is dirty, so the first "
|
|
|
|
|
"mounter must not be a spectator.\n",
|
|
|
|
|
jd->jd_jid);
|
2019-02-18 15:37:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
out_unlock:
|
|
|
|
|
gfs2_glock_dq_uninit(&j_gh);
|
|
|
|
|
return error;
|
|
|
|
|
}
|
|
|
|
|
|
2020-12-22 20:43:27 +00:00
|
|
|
/**
|
2022-11-14 17:26:00 +00:00
|
|
|
* gfs2_freeze_lock_shared - hold the freeze glock
|
2020-12-22 20:43:27 +00:00
|
|
|
* @sdp: the superblock
|
|
|
|
|
*/
|
2022-11-28 01:30:35 +00:00
|
|
|
int gfs2_freeze_lock_shared(struct gfs2_sbd *sdp)
|
2020-12-22 20:43:27 +00:00
|
|
|
{
|
|
|
|
|
int error;
|
|
|
|
|
|
gfs2: Add quota_change type
Function do_qc has two main uses: (1) to re-sync the local quota changes
(qd) to the master quotas, and (2) normal quota changes. In the case of
normal quota changes, the change can be positive or negative, as the
quota usage goes up and down.
Before this patch function do_qc was distinguishing one from another by
whether the resulting value is or isn't zero: In the case of a re-sync
(called do_sync) the quota value is moved from the temporary value to a
master value, so the amount is added to one and subtracted from the
other. The problem is that since the values can be positive or negative
we can occasionally run into situations where we are not doing a re-sync
but the quota change just happens to cancel out the previous value.
In the case of a re-sync extra references and locks are taken, and so
do_qc needs to release them. In the case of a normal quota change, no
extra references and locks are taken, so it must not try to release
them.
The problem is: if the quota change is not a re-sync but the value just
happens to cancel out the original quota change, the resulting zero
value fools do_qc into thinking this is a re-sync and therefore it must
release the extra references. This results in problems, mainly having to
do with slot reference numbers going smaller than zero.
This patch introduces new constants, QC_SYNC and QC_CHANGE so do_qc can
really tell the difference. For QC_SYNC calls it must release the extra
references acquired by gfs2_quota_unlock's call to qd_check_sync. For
QC_CHANGE calls it does not have extra references to put.
Note that this allows quota changes back to a value of zero, and so I
removed an assert warning related to that.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2023-06-28 14:54:41 +00:00
|
|
|
error = gfs2_glock_nq_init(sdp->sd_freeze_gl, LM_ST_SHARED,
|
|
|
|
|
LM_FLAG_NOEXP | GL_EXACT,
|
2022-11-28 01:30:35 +00:00
|
|
|
&sdp->sd_freeze_gh);
|
gfs2: Add quota_change type
Function do_qc has two main uses: (1) to re-sync the local quota changes
(qd) to the master quotas, and (2) normal quota changes. In the case of
normal quota changes, the change can be positive or negative, as the
quota usage goes up and down.
Before this patch function do_qc was distinguishing one from another by
whether the resulting value is or isn't zero: In the case of a re-sync
(called do_sync) the quota value is moved from the temporary value to a
master value, so the amount is added to one and subtracted from the
other. The problem is that since the values can be positive or negative
we can occasionally run into situations where we are not doing a re-sync
but the quota change just happens to cancel out the previous value.
In the case of a re-sync extra references and locks are taken, and so
do_qc needs to release them. In the case of a normal quota change, no
extra references and locks are taken, so it must not try to release
them.
The problem is: if the quota change is not a re-sync but the value just
happens to cancel out the original quota change, the resulting zero
value fools do_qc into thinking this is a re-sync and therefore it must
release the extra references. This results in problems, mainly having to
do with slot reference numbers going smaller than zero.
This patch introduces new constants, QC_SYNC and QC_CHANGE so do_qc can
really tell the difference. For QC_SYNC calls it must release the extra
references acquired by gfs2_quota_unlock's call to qd_check_sync. For
QC_CHANGE calls it does not have extra references to put.
Note that this allows quota changes back to a value of zero, and so I
removed an assert warning related to that.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2023-06-28 14:54:41 +00:00
|
|
|
if (error)
|
2022-11-16 13:19:06 +00:00
|
|
|
fs_err(sdp, "can't lock the freeze glock: %d\n", error);
|
2020-12-22 20:43:27 +00:00
|
|
|
return error;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void gfs2_freeze_unlock(struct gfs2_holder *freeze_gh)
|
|
|
|
|
{
|
|
|
|
|
if (gfs2_holder_initialized(freeze_gh))
|
|
|
|
|
gfs2_glock_dq_uninit(freeze_gh);
|
|
|
|
|
}
|
|
|
|
|
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
static void signal_our_withdraw(struct gfs2_sbd *sdp)
|
|
|
|
|
{
|
2021-01-18 20:18:59 +00:00
|
|
|
struct gfs2_glock *live_gl = sdp->sd_live_gh.gh_gl;
|
2021-03-12 12:58:54 +00:00
|
|
|
struct inode *inode;
|
|
|
|
|
struct gfs2_inode *ip;
|
|
|
|
|
struct gfs2_glock *i_gl;
|
|
|
|
|
u64 no_formal_ino;
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
int ret = 0;
|
|
|
|
|
int tries;
|
|
|
|
|
|
2021-03-12 12:58:54 +00:00
|
|
|
if (test_bit(SDF_NORECOVERY, &sdp->sd_flags) || !sdp->sd_jdesc)
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
return;
|
|
|
|
|
|
2021-05-19 18:54:02 +00:00
|
|
|
gfs2_ail_drain(sdp); /* frees all transactions */
|
2021-03-12 12:58:54 +00:00
|
|
|
inode = sdp->sd_jdesc->jd_inode;
|
|
|
|
|
ip = GFS2_I(inode);
|
|
|
|
|
i_gl = ip->i_gl;
|
|
|
|
|
no_formal_ino = ip->i_no_formal_ino;
|
|
|
|
|
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
/* Prevent any glock dq until withdraw recovery is complete */
|
|
|
|
|
set_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags);
|
|
|
|
|
/*
|
|
|
|
|
* Don't tell dlm we're bailing until we have no more buffers in the
|
|
|
|
|
* wind. If journal had an IO error, the log code should just purge
|
|
|
|
|
* the outstanding buffers rather than submitting new IO. Making the
|
|
|
|
|
* file system read-only will flush the journal, etc.
|
|
|
|
|
*
|
|
|
|
|
* During a normal unmount, gfs2_make_fs_ro calls gfs2_log_shutdown
|
|
|
|
|
* which clears SDF_JOURNAL_LIVE. In a withdraw, we must not write
|
|
|
|
|
* any UNMOUNT log header, so we can't call gfs2_log_shutdown, and
|
|
|
|
|
* therefore we need to clear SDF_JOURNAL_LIVE manually.
|
|
|
|
|
*/
|
|
|
|
|
clear_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
|
2020-12-22 20:43:28 +00:00
|
|
|
if (!sb_rdonly(sdp->sd_vfs)) {
|
gfs2: Rework freeze / thaw logic
So far, at mount time, gfs2 would take the freeze glock in shared mode
and then immediately drop it again, turning it into a cached glock that
can be reclaimed at any time. To freeze the filesystem cluster-wide,
the node initiating the freeze would take the freeze glock in exclusive
mode, which would cause the freeze glock's freeze_go_sync() callback to
run on each node. There, gfs2 would freeze the filesystem and schedule
gfs2_freeze_func() to run. gfs2_freeze_func() would re-acquire the
freeze glock in shared mode, thaw the filesystem, and drop the freeze
glock again. The initiating node would keep the freeze glock held in
exclusive mode. To thaw the filesystem, the initiating node would drop
the freeze glock again, which would allow gfs2_freeze_func() to resume
on all nodes, leaving the filesystem in the thawed state.
It turns out that in freeze_go_sync(), we cannot reliably and safely
freeze the filesystem. This is primarily because the final unmount of a
filesystem takes a write lock on the s_umount rw semaphore before
calling into gfs2_put_super(), and freeze_go_sync() needs to call
freeze_super() which also takes a write lock on the same semaphore,
causing a deadlock. We could work around this by trying to take an
active reference on the super block first, which would prevent unmount
from running at the same time. But that can fail, and freeze_go_sync()
isn't actually allowed to fail.
To get around this, this patch changes the freeze glock locking scheme
as follows:
At mount time, each node takes the freeze glock in shared mode. To
freeze a filesystem, the initiating node first freezes the filesystem
locally and then drops and re-acquires the freeze glock in exclusive
mode. All other nodes notice that there is contention on the freeze
glock in their go_callback callbacks, and they schedule
gfs2_freeze_func() to run. There, they freeze the filesystem locally
and drop and re-acquire the freeze glock before re-thawing the
filesystem. This is happening outside of the glock state engine, so
there, we are allowed to fail.
From a cluster point of view, taking and immediately dropping a glock is
indistinguishable from taking the glock and only dropping it upon
contention, so this new scheme is compatible with the old one.
Thanks to Li Dong <lidong@vivo.com> for reporting a locking bug in
gfs2_freeze_func() in a previous version of this commit.
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2022-11-14 22:34:50 +00:00
|
|
|
bool locked = mutex_trylock(&sdp->sd_freeze_mutex);
|
|
|
|
|
|
gfs2: Fix asynchronous thread destruction
The kernel threads are currently stopped and destroyed synchronously by
gfs2_make_fs_ro() and gfs2_put_super(), and asynchronously by
signal_our_withdraw(), with no synchronization, so the synchronous and
asynchronous contexts can race with each other.
First, when creating the kernel threads, take an extra task struct
reference so that the task struct won't go away immediately when they
terminate. This allows those kthreads to terminate immediately when
they're done rather than hanging around as zombies until they are reaped
by kthread_stop(). When kthread_stop() is called on a terminated
kthread, it will return immediately.
Second, in signal_our_withdraw(), once the SDF_JOURNAL_LIVE flag has
been cleared, wake up the logd and quotad wait queues instead of
stopping the logd and quotad kthreads. The kthreads are then expected
to terminate automatically within short time, but if they cannot, they
will not block the withdraw.
For example, if a user process and one of the kthread decide to withdraw
at the same time, only one of them will perform the actual withdraw and
the other will wait for it to be done. If the kthread ends up being the
one to wait, the withdrawing user process won't be able to stop it.
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2023-08-28 15:14:32 +00:00
|
|
|
wake_up(&sdp->sd_logd_waitq);
|
|
|
|
|
wake_up(&sdp->sd_quota_wait);
|
2023-08-28 14:39:20 +00:00
|
|
|
|
|
|
|
|
wait_event_timeout(sdp->sd_log_waitq,
|
|
|
|
|
gfs2_log_is_empty(sdp),
|
|
|
|
|
HZ * 5);
|
|
|
|
|
|
|
|
|
|
sdp->sd_vfs->s_flags |= SB_RDONLY;
|
gfs2: Rework freeze / thaw logic
So far, at mount time, gfs2 would take the freeze glock in shared mode
and then immediately drop it again, turning it into a cached glock that
can be reclaimed at any time. To freeze the filesystem cluster-wide,
the node initiating the freeze would take the freeze glock in exclusive
mode, which would cause the freeze glock's freeze_go_sync() callback to
run on each node. There, gfs2 would freeze the filesystem and schedule
gfs2_freeze_func() to run. gfs2_freeze_func() would re-acquire the
freeze glock in shared mode, thaw the filesystem, and drop the freeze
glock again. The initiating node would keep the freeze glock held in
exclusive mode. To thaw the filesystem, the initiating node would drop
the freeze glock again, which would allow gfs2_freeze_func() to resume
on all nodes, leaving the filesystem in the thawed state.
It turns out that in freeze_go_sync(), we cannot reliably and safely
freeze the filesystem. This is primarily because the final unmount of a
filesystem takes a write lock on the s_umount rw semaphore before
calling into gfs2_put_super(), and freeze_go_sync() needs to call
freeze_super() which also takes a write lock on the same semaphore,
causing a deadlock. We could work around this by trying to take an
active reference on the super block first, which would prevent unmount
from running at the same time. But that can fail, and freeze_go_sync()
isn't actually allowed to fail.
To get around this, this patch changes the freeze glock locking scheme
as follows:
At mount time, each node takes the freeze glock in shared mode. To
freeze a filesystem, the initiating node first freezes the filesystem
locally and then drops and re-acquires the freeze glock in exclusive
mode. All other nodes notice that there is contention on the freeze
glock in their go_callback callbacks, and they schedule
gfs2_freeze_func() to run. There, they freeze the filesystem locally
and drop and re-acquire the freeze glock before re-thawing the
filesystem. This is happening outside of the glock state engine, so
there, we are allowed to fail.
From a cluster point of view, taking and immediately dropping a glock is
indistinguishable from taking the glock and only dropping it upon
contention, so this new scheme is compatible with the old one.
Thanks to Li Dong <lidong@vivo.com> for reporting a locking bug in
gfs2_freeze_func() in a previous version of this commit.
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2022-11-14 22:34:50 +00:00
|
|
|
|
|
|
|
|
if (locked)
|
|
|
|
|
mutex_unlock(&sdp->sd_freeze_mutex);
|
|
|
|
|
|
2022-08-18 18:32:37 +00:00
|
|
|
/*
|
|
|
|
|
* Dequeue any pending non-system glock holders that can no
|
|
|
|
|
* longer be granted because the file system is withdrawn.
|
|
|
|
|
*/
|
|
|
|
|
gfs2_gl_dq_holders(sdp);
|
2020-12-22 20:43:28 +00:00
|
|
|
}
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
|
gfs2: Fix BUG during unmount after file system withdraw
Before this patch, when the logd daemon was forced to withdraw, it
would try to request its journal be recovered by another cluster node.
However, in single-user cases with lock_nolock, there are no other
nodes to recover the journal. Function signal_our_withdraw() was
recognizing the lock_nolock situation, but not until after it had
evicted its journal inode. Since the journal descriptor that points
to the inode was never removed from the master list, when the unmount
occurred, it did another iput on the evicted inode, which resulted in
a BUG_ON(inode->i_state & I_CLEAR).
This patch moves the check for this situation earlier in function
signal_our_withdraw(), which avoids the extra iput, so the unmount
may happen normally.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2020-04-24 17:15:21 +00:00
|
|
|
if (sdp->sd_lockstruct.ls_ops->lm_lock == NULL) { /* lock_nolock */
|
|
|
|
|
if (!ret)
|
|
|
|
|
ret = -EIO;
|
|
|
|
|
clear_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags);
|
|
|
|
|
goto skip_recovery;
|
|
|
|
|
}
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
/*
|
|
|
|
|
* Drop the glock for our journal so another node can recover it.
|
|
|
|
|
*/
|
|
|
|
|
if (gfs2_holder_initialized(&sdp->sd_journal_gh)) {
|
|
|
|
|
gfs2_glock_dq_wait(&sdp->sd_journal_gh);
|
|
|
|
|
gfs2_holder_uninit(&sdp->sd_journal_gh);
|
|
|
|
|
}
|
|
|
|
|
sdp->sd_jinode_gh.gh_flags |= GL_NOCACHE;
|
|
|
|
|
gfs2_glock_dq(&sdp->sd_jinode_gh);
|
gfs2: Rework freeze / thaw logic
So far, at mount time, gfs2 would take the freeze glock in shared mode
and then immediately drop it again, turning it into a cached glock that
can be reclaimed at any time. To freeze the filesystem cluster-wide,
the node initiating the freeze would take the freeze glock in exclusive
mode, which would cause the freeze glock's freeze_go_sync() callback to
run on each node. There, gfs2 would freeze the filesystem and schedule
gfs2_freeze_func() to run. gfs2_freeze_func() would re-acquire the
freeze glock in shared mode, thaw the filesystem, and drop the freeze
glock again. The initiating node would keep the freeze glock held in
exclusive mode. To thaw the filesystem, the initiating node would drop
the freeze glock again, which would allow gfs2_freeze_func() to resume
on all nodes, leaving the filesystem in the thawed state.
It turns out that in freeze_go_sync(), we cannot reliably and safely
freeze the filesystem. This is primarily because the final unmount of a
filesystem takes a write lock on the s_umount rw semaphore before
calling into gfs2_put_super(), and freeze_go_sync() needs to call
freeze_super() which also takes a write lock on the same semaphore,
causing a deadlock. We could work around this by trying to take an
active reference on the super block first, which would prevent unmount
from running at the same time. But that can fail, and freeze_go_sync()
isn't actually allowed to fail.
To get around this, this patch changes the freeze glock locking scheme
as follows:
At mount time, each node takes the freeze glock in shared mode. To
freeze a filesystem, the initiating node first freezes the filesystem
locally and then drops and re-acquires the freeze glock in exclusive
mode. All other nodes notice that there is contention on the freeze
glock in their go_callback callbacks, and they schedule
gfs2_freeze_func() to run. There, they freeze the filesystem locally
and drop and re-acquire the freeze glock before re-thawing the
filesystem. This is happening outside of the glock state engine, so
there, we are allowed to fail.
From a cluster point of view, taking and immediately dropping a glock is
indistinguishable from taking the glock and only dropping it upon
contention, so this new scheme is compatible with the old one.
Thanks to Li Dong <lidong@vivo.com> for reporting a locking bug in
gfs2_freeze_func() in a previous version of this commit.
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
2022-11-14 22:34:50 +00:00
|
|
|
gfs2_thaw_freeze_initiator(sdp->sd_vfs);
|
|
|
|
|
wait_on_bit(&i_gl->gl_flags, GLF_DEMOTE, TASK_UNINTERRUPTIBLE);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* holder_uninit to force glock_put, to force dlm to let go
|
|
|
|
|
*/
|
|
|
|
|
gfs2_holder_uninit(&sdp->sd_jinode_gh);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Note: We need to be careful here:
|
|
|
|
|
* Our iput of jd_inode will evict it. The evict will dequeue its
|
|
|
|
|
* glock, but the glock dq will wait for the withdraw unless we have
|
|
|
|
|
* exception code in glock_dq.
|
|
|
|
|
*/
|
|
|
|
|
iput(inode);
|
2022-08-18 18:32:36 +00:00
|
|
|
sdp->sd_jdesc->jd_inode = NULL;
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
/*
|
|
|
|
|
* Wait until the journal inode's glock is freed. This allows try locks
|
|
|
|
|
* on other nodes to be successful, otherwise we remain the owner of
|
|
|
|
|
* the glock as far as dlm is concerned.
|
|
|
|
|
*/
|
2021-01-18 20:18:59 +00:00
|
|
|
if (i_gl->gl_ops->go_free) {
|
|
|
|
|
set_bit(GLF_FREEING, &i_gl->gl_flags);
|
|
|
|
|
wait_on_bit(&i_gl->gl_flags, GLF_FREEING, TASK_UNINTERRUPTIBLE);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Dequeue the "live" glock, but keep a reference so it's never freed.
|
|
|
|
|
*/
|
2021-01-18 20:18:59 +00:00
|
|
|
gfs2_glock_hold(live_gl);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
gfs2_glock_dq_wait(&sdp->sd_live_gh);
|
|
|
|
|
/*
|
|
|
|
|
* We enqueue the "live" glock in EX so that all other nodes
|
|
|
|
|
* get a demote request and act on it. We don't really want the
|
|
|
|
|
* lock in EX, so we send a "try" lock with 1CB to produce a callback.
|
|
|
|
|
*/
|
|
|
|
|
fs_warn(sdp, "Requesting recovery of jid %d.\n",
|
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
2022-04-05 20:39:16 +00:00
|
|
|
gfs2_holder_reinit(LM_ST_EXCLUSIVE,
|
|
|
|
|
LM_FLAG_TRY_1CB | LM_FLAG_NOEXP | GL_NOPID,
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
&sdp->sd_live_gh);
|
|
|
|
|
msleep(GL_GLOCK_MAX_HOLD);
|
|
|
|
|
/*
|
|
|
|
|
* This will likely fail in a cluster, but succeed standalone:
|
|
|
|
|
*/
|
|
|
|
|
ret = gfs2_glock_nq(&sdp->sd_live_gh);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If we actually got the "live" lock in EX mode, there are no other
|
|
|
|
|
* nodes available to replay our journal. So we try to replay it
|
|
|
|
|
* ourselves. We hold the "live" glock to prevent other mounters
|
|
|
|
|
* during recovery, then just dequeue it and reacquire it in our
|
|
|
|
|
* normal SH mode. Just in case the problem that caused us to
|
|
|
|
|
* withdraw prevents us from recovering our journal (e.g. io errors
|
|
|
|
|
* and such) we still check if the journal is clean before proceeding
|
|
|
|
|
* but we may wait forever until another mounter does the recovery.
|
|
|
|
|
*/
|
|
|
|
|
if (ret == 0) {
|
|
|
|
|
fs_warn(sdp, "No other mounters found. Trying to recover our "
|
|
|
|
|
"own journal jid %d.\n", sdp->sd_lockstruct.ls_jid);
|
|
|
|
|
if (gfs2_recover_journal(sdp->sd_jdesc, 1))
|
|
|
|
|
fs_warn(sdp, "Unable to recover our journal jid %d.\n",
|
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
|
|
|
|
gfs2_glock_dq_wait(&sdp->sd_live_gh);
|
2022-04-05 20:39:16 +00:00
|
|
|
gfs2_holder_reinit(LM_ST_SHARED,
|
|
|
|
|
LM_FLAG_NOEXP | GL_EXACT | GL_NOPID,
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
&sdp->sd_live_gh);
|
|
|
|
|
gfs2_glock_nq(&sdp->sd_live_gh);
|
|
|
|
|
}
|
|
|
|
|
|
2021-01-18 20:18:59 +00:00
|
|
|
gfs2_glock_queue_put(live_gl); /* drop extra reference we acquired */
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
clear_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* At this point our journal is evicted, so we need to get a new inode
|
|
|
|
|
* for it. Once done, we need to call gfs2_find_jhead which
|
|
|
|
|
* calls gfs2_map_journal_extents to map it for us again.
|
|
|
|
|
*
|
|
|
|
|
* Note that we don't really want it to look up a FREE block. The
|
|
|
|
|
* GFS2_BLKST_FREE simply overrides a block check in gfs2_inode_lookup
|
|
|
|
|
* which would otherwise fail because it requires grabbing an rgrp
|
|
|
|
|
* glock, which would fail with -EIO because we're withdrawing.
|
|
|
|
|
*/
|
|
|
|
|
inode = gfs2_inode_lookup(sdp->sd_vfs, DT_UNKNOWN,
|
|
|
|
|
sdp->sd_jdesc->jd_no_addr, no_formal_ino,
|
|
|
|
|
GFS2_BLKST_FREE);
|
|
|
|
|
if (IS_ERR(inode)) {
|
|
|
|
|
fs_warn(sdp, "Reprocessing of jid %d failed with %ld.\n",
|
|
|
|
|
sdp->sd_lockstruct.ls_jid, PTR_ERR(inode));
|
|
|
|
|
goto skip_recovery;
|
|
|
|
|
}
|
|
|
|
|
sdp->sd_jdesc->jd_inode = inode;
|
2021-07-30 17:40:25 +00:00
|
|
|
d_mark_dontcache(inode);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Now wait until recovery is complete.
|
|
|
|
|
*/
|
|
|
|
|
for (tries = 0; tries < 10; tries++) {
|
2019-02-18 20:04:13 +00:00
|
|
|
ret = check_journal_clean(sdp, sdp->sd_jdesc, false);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
if (!ret)
|
|
|
|
|
break;
|
|
|
|
|
msleep(HZ);
|
|
|
|
|
fs_warn(sdp, "Waiting for journal recovery jid %d.\n",
|
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
|
|
|
|
}
|
|
|
|
|
skip_recovery:
|
|
|
|
|
if (!ret)
|
|
|
|
|
fs_warn(sdp, "Journal recovery complete for jid %d.\n",
|
|
|
|
|
sdp->sd_lockstruct.ls_jid);
|
|
|
|
|
else
|
2021-07-26 17:53:12 +00:00
|
|
|
fs_warn(sdp, "Journal recovery skipped for jid %d until next "
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
"mount.\n", sdp->sd_lockstruct.ls_jid);
|
|
|
|
|
fs_warn(sdp, "Glock dequeues delayed: %lu\n", sdp->sd_glock_dqs_held);
|
|
|
|
|
sdp->sd_glock_dqs_held = 0;
|
|
|
|
|
wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_RECOVERY);
|
|
|
|
|
}
|
|
|
|
|
|
2020-01-23 17:41:00 +00:00
|
|
|
void gfs2_lm(struct gfs2_sbd *sdp, const char *fmt, ...)
|
|
|
|
|
{
|
|
|
|
|
struct va_format vaf;
|
|
|
|
|
va_list args;
|
|
|
|
|
|
|
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW &&
|
|
|
|
|
test_bit(SDF_WITHDRAWN, &sdp->sd_flags))
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
va_start(args, fmt);
|
|
|
|
|
vaf.fmt = fmt;
|
|
|
|
|
vaf.va = &args;
|
|
|
|
|
fs_err(sdp, "%pV", &vaf);
|
|
|
|
|
va_end(args);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int gfs2_withdraw(struct gfs2_sbd *sdp)
|
2008-01-30 15:34:04 +00:00
|
|
|
{
|
2009-01-12 10:43:39 +00:00
|
|
|
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
|
|
|
|
|
const struct lm_lockops *lm = ls->ls_ops;
|
2008-01-30 15:34:04 +00:00
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW) {
|
2023-08-30 20:09:36 +00:00
|
|
|
unsigned long old = READ_ONCE(sdp->sd_flags), new;
|
|
|
|
|
|
|
|
|
|
do {
|
|
|
|
|
if (old & BIT(SDF_WITHDRAWN)) {
|
|
|
|
|
wait_on_bit(&sdp->sd_flags,
|
|
|
|
|
SDF_WITHDRAW_IN_PROG,
|
|
|
|
|
TASK_UNINTERRUPTIBLE);
|
|
|
|
|
return -1;
|
|
|
|
|
}
|
|
|
|
|
new = old | BIT(SDF_WITHDRAWN) | BIT(SDF_WITHDRAW_IN_PROG);
|
|
|
|
|
} while (unlikely(!try_cmpxchg(&sdp->sd_flags, &old, new)));
|
|
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
fs_err(sdp, "about to withdraw this file system\n");
|
|
|
|
|
BUG_ON(sdp->sd_args.ar_debug);
|
2008-01-30 15:34:04 +00:00
|
|
|
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
signal_our_withdraw(sdp);
|
|
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
kobject_uevent(&sdp->sd_kobj, KOBJ_OFFLINE);
|
2009-01-12 10:43:39 +00:00
|
|
|
|
2013-02-13 12:21:40 +00:00
|
|
|
if (!strcmp(sdp->sd_lockstruct.ls_ops->lm_proto_name, "lock_dlm"))
|
|
|
|
|
wait_for_completion(&sdp->sd_wdack);
|
|
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (lm->lm_unmount) {
|
|
|
|
|
fs_err(sdp, "telling LM to unmount\n");
|
|
|
|
|
lm->lm_unmount(sdp);
|
|
|
|
|
}
|
2016-03-23 18:29:59 +00:00
|
|
|
set_bit(SDF_SKIP_DLM_UNLOCK, &sdp->sd_flags);
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
fs_err(sdp, "File system withdrawn\n");
|
2009-08-24 09:44:18 +00:00
|
|
|
dump_stack();
|
gfs2: Force withdraw to replay journals and wait for it to finish
When a node withdraws from a file system, it often leaves its journal
in an incomplete state. This is especially true when the withdraw is
caused by io errors writing to the journal. Before this patch, a
withdraw would try to write a "shutdown" record to the journal, tell
dlm it's done with the file system, and none of the other nodes
know about the problem. Later, when the problem is fixed and the
withdrawn node is rebooted, it would then discover that its own
journal was incomplete, and replay it. However, replaying it at this
point is almost guaranteed to introduce corruption because the other
nodes are likely to have used affected resource groups that appeared
in the journal since the time of the withdraw. Replaying the journal
later will overwrite any changes made, and not through any fault of
dlm, which was instructed during the withdraw to release those
resources.
This patch makes file system withdraws seen by the entire cluster.
Withdrawing nodes dequeue their journal glock to allow recovery.
The remaining nodes check all the journals to see if they are
clean or in need of replay. They try to replay dirty journals, but
only the journals of withdrawn nodes will be "not busy" and
therefore available for replay.
Until the journal replay is complete, no i/o related glocks may be
given out, to ensure that the replay does not cause the
aforementioned corruption: We cannot allow any journal replay to
overwrite blocks associated with a glock once it is held.
The "live" glock which is now used to signal when a withdraw
occurs. When a withdraw occurs, the node signals its withdraw by
dequeueing the "live" glock and trying to enqueue it in EX mode,
thus forcing the other nodes to all see a demote request, by way
of a "1CB" (one callback) try lock. The "live" glock is not
granted in EX; the callback is only just used to indicate a
withdraw has occurred.
Note that all nodes in the cluster must wait for the recovering
node to finish replaying the withdrawing node's journal before
continuing. To this end, it checks that the journals are clean
multiple times in a retry loop.
Also note that the withdraw function may be called from a wide
variety of situations, and therefore, we need to take extra
precautions to make sure pointers are valid before using them in
many circumstances.
We also need to take care when glocks decide to withdraw, since
the withdraw code now uses glocks.
Also, before this patch, if a process encountered an error and
decided to withdraw, if another process was already withdrawing,
the second withdraw would be silently ignored, which set it free
to unlock its glocks. That's correct behavior if the original
withdrawer encounters further errors down the road. But if
secondary waiters don't wait for the journal replay, unlocking
glocks will allow other nodes to use them, despite the fact that
the journal containing those blocks is being replayed. The
replay needs to finish before our glocks are released to other
nodes. IOW, secondary withdraws need to wait for the first
withdraw to finish.
For example, if an rgrp glock is unlocked by a process that didn't
wait for the first withdraw, a journal replay could introduce file
system corruption by replaying a rgrp block that has already been
granted to a different cluster node.
Signed-off-by: Bob Peterson <rpeterso@redhat.com>
2020-01-28 19:23:45 +00:00
|
|
|
clear_bit(SDF_WITHDRAW_IN_PROG, &sdp->sd_flags);
|
|
|
|
|
smp_mb__after_atomic();
|
|
|
|
|
wake_up_bit(&sdp->sd_flags, SDF_WITHDRAW_IN_PROG);
|
2009-01-12 10:43:39 +00:00
|
|
|
}
|
2009-08-24 09:44:18 +00:00
|
|
|
|
|
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
|
2014-03-06 20:10:45 +00:00
|
|
|
panic("GFS2: fsid=%s: panic requested\n", sdp->sd_fsname);
|
2008-01-30 15:34:04 +00:00
|
|
|
|
|
|
|
|
return -1;
|
|
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_assert_withdraw_i - Cause the machine to withdraw if @assertion is false
|
|
|
|
|
*/
|
|
|
|
|
|
2020-01-23 18:36:21 +00:00
|
|
|
void gfs2_assert_withdraw_i(struct gfs2_sbd *sdp, char *assertion,
|
2020-01-08 17:37:30 +00:00
|
|
|
const char *function, char *file, unsigned int line,
|
|
|
|
|
bool delayed)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2023-12-20 16:16:29 +00:00
|
|
|
if (gfs2_withdrawing_or_withdrawn(sdp))
|
2020-01-08 17:37:30 +00:00
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
fs_err(sdp,
|
|
|
|
|
"fatal: assertion \"%s\" failed\n"
|
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
|
assertion, function, file, line);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If errors=panic was specified on mount, it won't help to delay the
|
|
|
|
|
* withdraw.
|
|
|
|
|
*/
|
|
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
|
|
|
|
|
delayed = false;
|
|
|
|
|
|
|
|
|
|
if (delayed)
|
|
|
|
|
gfs2_withdraw_delayed(sdp);
|
|
|
|
|
else
|
|
|
|
|
gfs2_withdraw(sdp);
|
2006-02-08 11:50:51 +00:00
|
|
|
dump_stack();
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_assert_warn_i - Print a message to the console if @assertion is false
|
|
|
|
|
*/
|
|
|
|
|
|
2020-01-23 18:36:21 +00:00
|
|
|
void gfs2_assert_warn_i(struct gfs2_sbd *sdp, char *assertion,
|
|
|
|
|
const char *function, char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
|
|
|
|
if (time_before(jiffies,
|
|
|
|
|
sdp->sd_last_warning +
|
|
|
|
|
gfs2_tune_get(sdp, gt_complain_secs) * HZ))
|
2020-01-23 18:36:21 +00:00
|
|
|
return;
|
2006-01-16 16:50:04 +00:00
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_WITHDRAW)
|
2014-03-06 20:10:46 +00:00
|
|
|
fs_warn(sdp, "warning: assertion \"%s\" failed at function = %s, file = %s, line = %u\n",
|
|
|
|
|
assertion, function, file, line);
|
2006-01-16 16:50:04 +00:00
|
|
|
|
|
|
|
|
if (sdp->sd_args.ar_debug)
|
|
|
|
|
BUG();
|
2006-02-08 11:50:51 +00:00
|
|
|
else
|
|
|
|
|
dump_stack();
|
2006-01-16 16:50:04 +00:00
|
|
|
|
2009-08-24 09:44:18 +00:00
|
|
|
if (sdp->sd_args.ar_errors == GFS2_ERRORS_PANIC)
|
|
|
|
|
panic("GFS2: fsid=%s: warning: assertion \"%s\" failed\n"
|
|
|
|
|
"GFS2: fsid=%s: function = %s, file = %s, line = %u\n",
|
|
|
|
|
sdp->sd_fsname, assertion,
|
|
|
|
|
sdp->sd_fsname, function, file, line);
|
|
|
|
|
|
2006-01-16 16:50:04 +00:00
|
|
|
sdp->sd_last_warning = jiffies;
|
|
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_consist_i - Flag a filesystem consistency error and withdraw
|
|
|
|
|
*/
|
|
|
|
|
|
2020-01-23 18:31:06 +00:00
|
|
|
void gfs2_consist_i(struct gfs2_sbd *sdp, const char *function,
|
|
|
|
|
char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_lm(sdp,
|
|
|
|
|
"fatal: filesystem consistency error - function = %s, file = %s, line = %u\n",
|
|
|
|
|
function, file, line);
|
2020-01-23 18:31:06 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_consist_inode_i - Flag an inode consistency error and withdraw
|
|
|
|
|
*/
|
|
|
|
|
|
2020-01-23 18:31:06 +00:00
|
|
|
void gfs2_consist_inode_i(struct gfs2_inode *ip,
|
|
|
|
|
const char *function, char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2006-06-14 19:32:57 +00:00
|
|
|
struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode);
|
2020-01-23 17:41:00 +00:00
|
|
|
|
|
|
|
|
gfs2_lm(sdp,
|
|
|
|
|
"fatal: filesystem consistency error\n"
|
|
|
|
|
" inode = %llu %llu\n"
|
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
|
(unsigned long long)ip->i_no_formal_ino,
|
|
|
|
|
(unsigned long long)ip->i_no_addr,
|
|
|
|
|
function, file, line);
|
2021-09-30 12:48:29 +00:00
|
|
|
gfs2_dump_glock(NULL, ip->i_gl, 1);
|
2020-01-23 18:31:06 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_consist_rgrpd_i - Flag a RG consistency error and withdraw
|
|
|
|
|
*/
|
|
|
|
|
|
2020-01-23 18:31:06 +00:00
|
|
|
void gfs2_consist_rgrpd_i(struct gfs2_rgrpd *rgd,
|
|
|
|
|
const char *function, char *file, unsigned int line)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
|
|
|
|
struct gfs2_sbd *sdp = rgd->rd_sbd;
|
2019-08-13 13:25:15 +00:00
|
|
|
char fs_id_buf[sizeof(sdp->sd_fsname) + 7];
|
2018-08-15 17:09:49 +00:00
|
|
|
|
2019-05-09 14:21:48 +00:00
|
|
|
sprintf(fs_id_buf, "fsid=%s: ", sdp->sd_fsname);
|
2020-10-07 11:30:58 +00:00
|
|
|
gfs2_rgrp_dump(NULL, rgd, fs_id_buf);
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_lm(sdp,
|
|
|
|
|
"fatal: filesystem consistency error\n"
|
|
|
|
|
" RG = %llu\n"
|
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
|
(unsigned long long)rgd->rd_addr,
|
|
|
|
|
function, file, line);
|
2021-09-30 12:48:29 +00:00
|
|
|
gfs2_dump_glock(NULL, rgd->rd_gl, 1);
|
2020-01-23 18:31:06 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_meta_check_ii - Flag a magic number consistency error and withdraw
|
|
|
|
|
* Returns: -1 if this call withdrew the machine,
|
|
|
|
|
* -2 if it was already withdrawn
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
int gfs2_meta_check_ii(struct gfs2_sbd *sdp, struct buffer_head *bh,
|
|
|
|
|
const char *type, const char *function, char *file,
|
|
|
|
|
unsigned int line)
|
|
|
|
|
{
|
|
|
|
|
int me;
|
2020-01-23 17:41:00 +00:00
|
|
|
|
|
|
|
|
gfs2_lm(sdp,
|
|
|
|
|
"fatal: invalid metadata block\n"
|
|
|
|
|
" bh = %llu (%s)\n"
|
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
|
(unsigned long long)bh->b_blocknr, type,
|
|
|
|
|
function, file, line);
|
|
|
|
|
me = gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
return (me) ? -1 : -2;
|
|
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_metatype_check_ii - Flag a metadata type consistency error and withdraw
|
|
|
|
|
* Returns: -1 if this call withdrew the machine,
|
|
|
|
|
* -2 if it was already withdrawn
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
int gfs2_metatype_check_ii(struct gfs2_sbd *sdp, struct buffer_head *bh,
|
2006-09-04 16:49:07 +00:00
|
|
|
u16 type, u16 t, const char *function,
|
2006-01-16 16:50:04 +00:00
|
|
|
char *file, unsigned int line)
|
|
|
|
|
{
|
|
|
|
|
int me;
|
2020-01-23 17:41:00 +00:00
|
|
|
|
|
|
|
|
gfs2_lm(sdp,
|
|
|
|
|
"fatal: invalid metadata block\n"
|
|
|
|
|
" bh = %llu (type: exp=%u, found=%u)\n"
|
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
|
(unsigned long long)bh->b_blocknr, type, t,
|
|
|
|
|
function, file, line);
|
|
|
|
|
me = gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
return (me) ? -1 : -2;
|
|
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2006-01-16 16:50:04 +00:00
|
|
|
* gfs2_io_error_i - Flag an I/O error and withdraw
|
|
|
|
|
* Returns: -1 if this call withdrew the machine,
|
|
|
|
|
* 0 if it was already withdrawn
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
int gfs2_io_error_i(struct gfs2_sbd *sdp, const char *function, char *file,
|
|
|
|
|
unsigned int line)
|
|
|
|
|
{
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_lm(sdp,
|
|
|
|
|
"fatal: I/O error\n"
|
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
|
function, file, line);
|
|
|
|
|
return gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
|
2021-03-30 16:44:29 +00:00
|
|
|
/*
|
2018-06-07 10:56:46 +00:00
|
|
|
* gfs2_io_error_bh_i - Flag a buffer I/O error
|
|
|
|
|
* @withdraw: withdraw the filesystem
|
2006-01-16 16:50:04 +00:00
|
|
|
*/
|
|
|
|
|
|
2018-06-07 10:56:46 +00:00
|
|
|
void gfs2_io_error_bh_i(struct gfs2_sbd *sdp, struct buffer_head *bh,
|
|
|
|
|
const char *function, char *file, unsigned int line,
|
|
|
|
|
bool withdraw)
|
2006-01-16 16:50:04 +00:00
|
|
|
{
|
2023-12-20 16:16:29 +00:00
|
|
|
if (gfs2_withdrawing_or_withdrawn(sdp))
|
2019-02-12 20:43:55 +00:00
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
fs_err(sdp, "fatal: I/O error\n"
|
|
|
|
|
" block = %llu\n"
|
|
|
|
|
" function = %s, file = %s, line = %u\n",
|
|
|
|
|
(unsigned long long)bh->b_blocknr, function, file, line);
|
2018-06-07 10:56:46 +00:00
|
|
|
if (withdraw)
|
2020-01-23 17:41:00 +00:00
|
|
|
gfs2_withdraw(sdp);
|
2006-01-16 16:50:04 +00:00
|
|
|
}
|
|
|
|
|
|