cancel
Showing results for 
Search instead for 
Did you mean: 

node is getting added to standby priority list even promotable property set to false

SOLVED
Level 3 Adventurer

node is getting added to standby priority list even promotable property set to false

In our set up we have 5 databases where 3 databases are in primary dc(master and two standby) and 2 databases are in secondary dc(one stadby and other cascaded).

In all nodes efm services are running, We have set efm propery promotable to false for efm running on secodary dc servers.

Initially when we start the secondary dc efm services nodes are not added to standby priority list. But when we see the cluster-status on next day  we can notice these secondary dc efm nodes being added to standby priority list. automatically,

Need some help on this.

2 ACCEPTED SOLUTIONS

Accepted Solutions
EDB Team Member

Re: node is getting added to standby priority list even promotable property set to false

 

Can you please give some insights on this to understand why it is happening(i.e because of network of some other issue).

 


 

I think I already wrote that. Some parts of the cluster are disconnected from the rest, and then the network issue goes away and they rejoin. My other response had more details.

 

Bobby

 

Level 3 Adventurer

Re: node is getting added to standby priority list even promotable property set to false

Thanks a lot bobby for you valuable insights.

8 REPLIES 8
EDB Team Member

Re: node is getting added to standby priority list even promotable property set to false

What version of EFM are you using?

Did you receive any notifications about a cluster state mismatch?

 

Bobby

 

Level 3 Adventurer

Re: node is getting added to standby priority list even promotable property set to false

Hi Bobby,

 

Sorry for the late reply, I am using EFM version 3.4

 

2019-08-20 05:44:24 com.enterprisedb.efm.nodes.EfmNode lambda$handle$6 INFO: Reloading shared state per coordinator command
2019-08-20 05:44:24 com.enterprisedb.efm.nodes.EfmNode setState INFO: setState called: [ master: host-44463(ip) promoting: null standbys (3): host09-27775(ip) host11-54589(ip) host12-54975(ip) witnesses (1): hostname-59445(ip) idle nodes (0):
]
2019-08-20 05:44:24 org.jgroups.protocols.pbcast.GMS warn WARN: host09-27775(ip): not member of view [host08-44463(ip)|119]; dis
carding it
2019-08-20 05:44:47 com.enterprisedb.efm.nodes.EfmNode viewAccepted INFO: View changed. I am host09-27775(ip) Total jgroups nodes: 6
2019-08-20 05:44:47 com.enterprisedb.efm.nodes.EfmNode handleViewChange INFO: Node was suspect, but has returned to the view: host11-54589(ip)
2019-08-20 05:44:47 com.enterprisedb.efm.nodes.EfmNode updateStableClusterSize INFO: Cluster size now 6, was 5.
2019-08-20 05:44:47 com.enterprisedb.efm.exec.ExecUtil performExec INFO: [/bin/ping -q -c3 -w5 ip]
2019-08-20 05:44:49 org.jgroups.protocols.pbcast.GMS warn WARN: host09-27775(ip): failed to collect all ACKs (expected=4) for view [host08-4
4463(192.168.32.168)|120] after 2000ms, missing 4 ACKs from (4) host08-44463(ip), host13-59445(ip), host12-54975(ip), host11-54589(ip)
2019-08-20 05:44:49 com.enterprisedb.efm.exec.ExecUtil performExec INFO: ProcessResult{exitValue=0, errorOut='', stdOut='PING ip (ip) 56(84) bytes of data.

 

Getting above messages in logs.

Level 3 Adventurer

Re: node is getting added to standby priority list even promotable property set to false

oldr screen.pngafter priority set.png

EDB Team Member

Re: node is getting added to standby priority list even promotable property set to false

It would help to know if you were receiving the notification about cluster state -- otherwise I can just guess.

 

I think you're running into the case where some of the cluster gets separated from the rest. Thus, there is >1 actual efm cluster, each having its own internal state (including failover priority). When they're reconnected, the state has to get sorted out -- each node is asked what type it is, and standbys are added to the priority list at that point, ignoring their promotable=false setting.

 

If that's what you're hitting, it was fixed in EFM 3.5. Note: if you have promotable=true and use efm set-priority to make a standby non-promotable, it will revert back to its promotable=true setting during any case where the cluster state has to be reconstructed.

 

Cheers,

Bobby

 

Level 3 Adventurer

Re: node is getting added to standby priority list even promotable property set to false

2019-08-21 05:49:21 com.enterprisedb.efm.nodes.EfmNode handleViewChange WARN: Scheduling cluster state verification because of cluster merge.
2019-08-21 05:49:21 com.enterprisedb.efm.nodes.EfmWitness handleStatusCall INFO: responding to db status request with status: Witness%%10.16.32.169%%true%%false%%10.16.32.180%%false
2019-08-21 05:49:21 com.enterprisedb.efm.nodes.EfmNode compareToStatus ERROR: Nodes of type Standby were discovered in the cluster that do not match the internal state. Internal state: [mpgnedrpgdb11-54589(10.16.32.167), mpgnedrpgdb12-54975(10.16.32.168), mpgweprdpgdb09-27775(192.168.32.169)], found in cluster: [mpgweprdpgdb09-27775(192.168.32.169), mpgnedrpgdb11-54589(10.16.32.167), mpgnedrpgdb12-54975(10.16.32.168), mpgweprdpgdb10-17752(192.168.32.171)].
2019-08-21 05:49:21 com.enterprisedb.efm.utils.Notifications sendMail INFO: Sending notification:
To: [Karthik.SV@trianz.com, Prasannavenkatesan.G@trianz.com, Vijay.Bhat@trianz.com]
Subject: [SEVERE] EFM Internal state mismatch for cluster proddb
Body:
EFM node: 10.16.32.169
Hi Bobby,

 

Thank you  for your reply.

I am getting the below info in efm witness node.

 

Cluster name: proddb
Database name: edb
VIP: 10.16.32.180 (Inactive)

The failover manager cluster's internal state did not match the
actual state of the cluster members. This is rare and can be caused
by a timing issue of nodes joining the cluster and/or changing their
state. The problem should be resolved, but you should check the cluster
status as well to verify. Details of the mismatch can be found in the
agent log file.

 

Can you please give some insights on this to understand why it is happening(i.e because of network of some other issue).

 

Regards,

Vijay 

EDB Team Member

Re: node is getting added to standby priority list even promotable property set to false

 

Can you please give some insights on this to understand why it is happening(i.e because of network of some other issue).

 


 

I think I already wrote that. Some parts of the cluster are disconnected from the rest, and then the network issue goes away and they rejoin. My other response had more details.

 

Bobby

 

Level 3 Adventurer

Re: node is getting added to standby priority list even promotable property set to false

Thanks a lot bobby for you valuable insights.

Highlighted
EDB Team Member

Re: node is getting added to standby priority list even promotable property set to false

 


@Vijay855 wrote:

Thanks a lot bobby for you valuable insights.


I hope the newer version fixes your issue. :)

 

Cheers,

Bobby