linux/fs/ocfs2/dlm
Tariq Saeed 27bf6305cf ocfs2: fix deadlock when two nodes are converting same lock from PR to EX and idletimeout closes conn
Orabug: 18639535

Two node cluster and both nodes hold a lock at PR level and both want to
convert to EX at the same time.  Master node 1 has sent BAST and then
closes the connection due to idletime out.  Node 0 receives BAST, sends
unlock req with cancel flag but gets error -ENOTCONN.  The problem is
this error is ignored in dlm_send_remote_unlock_request() on the
**incorrect** assumption that the master is dead.  See NOTE in comment
why it returns DLM_NORMAL.  Upon getting DLM_NORMAL, node 0 proceeds to
sends convert (without cancel flg) which fails with -ENOTCONN.  waits 5
sec and resends.

This time gets DLM_IVLOCKID from the master since lock not found in
grant, it had been moved to converting queue in response to conv PR->EX
req.  No way out.

Node 1 (master)				Node 0
==============				======

  lock mode PR				PR

  convert PR -> EX
  mv grant -> convert and que BAST
  ...
                     <-------- convert PR -> EX
  convert que looks like this: ((node 1, PR -> EX) (node 0, PR -> EX))
  ...
                        BAST (want PR -> NL)
                     ------------------>
  ...
  idle timout, conn closed
                                ...
                                In response to BAST,
                                sends unlock with cancel convert flag
                                gets -ENOTCONN. Ignores and
                                sends remote convert request
                                gets -ENOTCONN, waits 5 Sec, retries
  ...
  reconnects
                   <----------------- convert req goes through on next try
  does not find lock on grant que
                   status DLM_IVLOCKID
                   ------------------>
  ...

No way out.  Fix is to keep retrying unlock with cancel flag until it
succeeds or the master dies.

Signed-off-by: Tariq Saeed <tariq.x.saeed@oracle.com>
Reviewed-by: Mark Fasheh <mfasheh@suse.de>
Cc: Joel Becker <jlbec@evilplan.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-06-23 16:47:45 -07:00
..
dlmapi.h ocfs2/trivial: Remove trailing whitespaces 2010-01-25 19:20:51 -08:00
dlmast.c ocfs2: use list_for_each_entry() instead of list_for_each() 2013-09-11 15:56:36 -07:00
dlmcommon.h ocfs2: remove some unused code 2014-06-04 16:53:55 -07:00
dlmconvert.c ocfs2: use list_for_each_entry() instead of list_for_each() 2013-09-11 15:56:36 -07:00
dlmconvert.h
dlmdebug.c ocfs2: remove NULL assignments on static 2014-06-04 16:53:53 -07:00
dlmdebug.h ocfs2/dlm: Cleanup dlmdebug.c 2010-12-22 18:34:44 -08:00
dlmdomain.c ocfs2: remove some unused code 2014-06-04 16:53:55 -07:00
dlmdomain.h
dlmlock.c ocfs2: remove NULL assignments on static 2014-06-04 16:53:53 -07:00
dlmmaster.c ocfs2: remove NULL assignments on static 2014-06-04 16:53:53 -07:00
dlmrecovery.c ocfs2/dlm: fix possible convert=sion deadlock 2014-06-04 16:53:54 -07:00
dlmthread.c ocfs2: use list_for_each_entry() instead of list_for_each() 2013-09-11 15:56:36 -07:00
dlmunlock.c ocfs2: fix deadlock when two nodes are converting same lock from PR to EX and idletimeout closes conn 2014-06-23 16:47:45 -07:00
Makefile ocfs2: remove versioning information 2014-01-21 16:19:41 -08:00