cancel
Showing results for 
Search instead for 
Did you mean: 

EFM Dedicated Witness Node

SOLVED
Highlighted
EDB Team Member

Re: EFM Dedicated Witness Node


pcpg wrote:

Does it mean that there is no way we can handle the scenario that I have described in my reply to Deepanshu above?


The short answer is no. That's not a limitation of EFM, but just the way the math works. The longer answer is below:

 

Starting with a simple master/standby/witness cluster, I think the question is "why can't I have master node failover if the witness fails?" An important thing to remember in the following is a design decision of EFM that it is better to have no masters than to have more than one master. Another is that network isolation looks the same as failure between nodes: if node A can no longer talk to B, there is no way for A to know if B failed or is just disconnected (and could be reconnected at any time).

 

Case 1: Witness fails, and some time later the master node fails.

 

In this case, we're starting with one master and one standby node. The master fails, and the standby sees it disappear. To the standby, there is no way to know whether the master node actually failed or simply was disconnected from the standby. The standby simply sees itself as part of a new, smaller, cluster. The new cluster size it sees is not a majority of the previous cluster (1 is not more than half of 2), so it cannot promote in case the master is still running somewhere. To put it another way, there is no way to support this case that would not create multiple masters in the case of simple network isolation.

 

For comparison, if the master is isolated while the witness is still alive, the standby would see it's in a 2-node cluster instead of 3 nodes, which is a majority, and so would promote. The master agent would see that it's in a minority of the cluster (1 compared to 3) and fence itself off. Thus, this case is safe whether the master node actually failed or was isolated from the network.

 

(Note: You suggested adding a witness agent to the existing standby node. As pointed out, this is not possible because both agents would try to use the same resources on that node at the same time. Even if you could do it, I'm not sure it helps with this use case. I guess it would mean that, if the two nodes are separated, the standby/witness one is "more important" and the master should isolate itself. Am not sure whether that would be a good idea or not since you still have a working master that applications may still be able to reach.)

 

Case 2: Witness and master fail at same time.

 

By "same time" here, we really mean close in time to each other, due to the various ways networks can fail. Am not sure this is what you're asking about, but wanted to incude it. In this case, the standby will see its cluster drop from 3 nodes to 1. As above, it has no way to know whether a) 2 nodes died or b) it was isolated from those two nodes. In the latter case, the master and witness would still be working just fine, so it would be very bad for the standby to promote itself. Since it can't tell which case is happening, it has to take the safe route and consider itself isolated from the majority of the cluster.

 

Either way, case 1 or 2, there is no safe way for the standby to know that it should promote.

 

I hope the above helps. The main idea is that, in your use case, if we promote the standby this will produce 2 masters in the network isolation case, and EFM does whatever it can to avoid these split-brain results. If you really need to run with just two nodes, you should at least get notifications about what is happening, and use the 'efm promote' command if needed after the master (half of the cluster) fails.

 

Cheers,

Bobby

 

Level 3 Adventurer

Re: EFM Dedicated Witness Node

Thanks Bobby. Super clear. Appreciate!

Adventurer

Re: EFM Dedicated Witness Node

i. What is the minimum requirement for running Witness node.

ii. Can it run on the same node on which an application server is running. Is this possibility available instead on running this on a dedicated server?

 

Moderator

Re: EFM Dedicated Witness Node

I will try to answer your questions, let me know if that helps::

 

i. What is the minimum requirement for running Witness node.

<Ans> The minimum system requirements would be the same as Operating system which you are using. There is very little overhead for the agent and it should not be resourece intensive.

 

ii. Can it run on the same node on which an application server is running. Is this possibility available instead on running this on a dedicated server?

<Ans>  Yes it can run on the same node as your application theorotically. Just ensure that it is the third node besides the two nodes running the database.