cancel
Showing results for 
Search instead for 
Did you mean: 

EFM-2.1 Not Started after applied patch

SOLVED
Level 2 Adventurer

EFM-2.1 Not Started after applied patch

Hi,

After applied patch efm could not started,

Error:-

--------

cat /var/log/efm2.1/efm.log

12/15/18 10:32:56 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread ERR is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:32:56 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread OUT is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:44:19 AM com.enterprisedb.efm.exec.ExecUtil performExec FINE: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:44:19 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread ERR is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:44:19 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread OUT is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]




----------------------------------------------------------------------------------------------[root@HKDTUATDB03 ~]# systemctl start efm-2.1
Job for efm-2.1.service failed because the control process exited with error code. See "systemctl status efm-2.1.service" and "journalctl -xe" for details.
[root@HKDTUATDB03 ~]#




--------------------------------------------------------------------------------------------
[root@HKDTUATDB03 ~]# systemctl status efm-2.1
? efm-2.1.service - EnterpriseDB Failover Manager 2.1
   Loaded: loaded (/usr/lib/systemd/system/efm-2.1.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2018-12-15 11:00:27 +08; 18s ago
  Process: 18177 ExecStart=/bin/bash -c /usr/efm-2.1/bin/runefm.sh start ${CLUSTER} (code=exited, status=1/FAILURE)

Dec 15 11:00:27 HKDTUATDB03 systemd[1]: Starting EnterpriseDB Failover Manag....
Dec 15 11:00:27 HKDTUATDB03 sudo[18252]: pam_unix(sudo:account): account efm...)
Dec 15 11:00:27 HKDTUATDB03 sudo[18252]:      efm : TTY=unknown ; PWD=/ ; US...m
Dec 15 11:00:27 HKDTUATDB03 systemd[1]: efm-2.1.service: control process exi...1
Dec 15 11:00:27 HKDTUATDB03 systemd[1]: Failed to start EnterpriseDB Failove....
Dec 15 11:00:27 HKDTUATDB03 systemd[1]: Unit efm-2.1.service entered failed ....
Dec 15 11:00:27 HKDTUATDB03 systemd[1]: efm-2.1.service failed.
Hint: Some lines were ellipsized, use -l to show in full.


--
1 ACCEPTED SOLUTION

Accepted Solutions
EDB Team Member

Re: EFM-2.1 Not Started after applied patch


@bbissett wrote:

No, FINEST level should not be needed here. If there's a problem starting the agent, the startup log is the place to look. What's in there? 


 

(Am pretty sure it will be an error about not being able to read the efm.nodes file....)

 

View solution in original post

13 REPLIES 13
Silver Adventurer

Re: EFM-2.1 Not Started after applied patch

Hi @Diny2019

 

Logs you provided contain least detail, which is not sufficient to troubleshoot. Can you please set logging to FINEST in efm.properties and share logs with us?

 

Thanks,

Ninad

EDB Team Member

Re: EFM-2.1 Not Started after applied patch


@nshah wrote:

Hi @Diny2019

 

Logs you provided contain least detail, which is not sufficient to troubleshoot. Can you please set logging to FINEST in efm.properties and share logs with us?

 

Thanks,

Ninad


 

No, FINEST level should not be needed here. If there's a problem starting the agent, the startup log is the place to look. What's in there?

 

Cheers,

Bobby

 

EDB Team Member

Re: EFM-2.1 Not Started after applied patch


@bbissett wrote:

No, FINEST level should not be needed here. If there's a problem starting the agent, the startup log is the place to look. What's in there? 


 

(Am pretty sure it will be an error about not being able to read the efm.nodes file....)

 

View solution in original post

Silver Adventurer

Re: EFM-2.1 Not Started after applied patch

Hi Bobby,

 

You are right. Aparantely, I overlooked here. Logging level is already set to FINEST here.

 

Thanks,

Ninad

EDB Team Member

Re: EFM-2.1 Not Started after applied patch


 

You are right. Aparantely, I overlooked here. Logging level is already set to FINEST here


 

Heh, that went right by me also. The important part is that the startup log hopefully has startup issues in it.

 

FWIW, support for EFM 2.1.X ends in August. I recommend the latest 3.X for anyone interested.

 

Cheers,

Bobby

 

 

 

Silver Adventurer

Re: EFM-2.1 Not Started after applied patch

Hi @Diny2019

 

As Bobby mentioned, We recommend you to upgrade to latest version of EFM 3.x. The latest version of EFM is 3.3. So, It's highly recommended to upgrade to 3.3.

 

Thanks,

Ninad

Level 2 Adventurer

Re: EFM-2.1 Not Started after applied patch

Hi All,

   

                     Note:- i mentioned below of line iam expected what is the main error...

1.may be cannot read the readnode files

2.efm os user account expired  (or) May be license issue..but we have a 1+years more

3.sudoer file cannot executed.

 

This is EFM.LOG

[root@HKDTUATDB01 efm-2.1]# cat efm.log
12/15/18 10:13:28 AM com.enterprisedb.efm.exec.ExecUtil performExec FINE: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:13:28 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread ERR is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:13:28 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread OUT is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:24:35 AM com.enterprisedb.efm.exec.ExecUtil performExec FINE: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:24:35 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread ERR is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:24:35 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread OUT is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:25:46 AM com.enterprisedb.efm.exec.ExecUtil performExec FINE: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:25:46 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread ERR is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:25:46 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread OUT is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:44:52 AM com.enterprisedb.efm.exec.ExecUtil performExec FINE: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:44:52 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread ERR is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:44:52 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread OUT is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:45:09 AM com.enterprisedb.efm.exec.ExecUtil performExec FINE: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:45:09 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread ERR is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

12/15/18 10:45:09 AM com.enterprisedb.efm.exec.ExecUtil verifyClosed FINEST: Verified thread OUT is closed for command: [sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm]

[root@HKDTUATDB01 efm-2.1]#




STARTUP LOG:-
[root@HKDTUATDB01 efm-2.1]# cat startup-efm.log
[ 12/15/18 10:13:28 AM ] Could not parse nodes file :
[ 12/15/18 10:13:28 AM ] sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm
[ 12/15/18 10:13:28 AM ]
[ 12/15/18 10:13:28 AM ] exit status: 1
[ 12/15/18 10:13:28 AM ] error:
[ 12/15/18 10:13:28 AM ] sudo: pam_open_session: System error
[ 12/15/18 10:13:28 AM ] sudo: policy plugin failed session initialization
[ 12/15/18 10:24:35 AM ] Could not parse nodes file :
[ 12/15/18 10:24:35 AM ] sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm
[ 12/15/18 10:24:35 AM ]
[ 12/15/18 10:24:35 AM ] exit status: 1
[ 12/15/18 10:24:35 AM ] error:
[ 12/15/18 10:24:35 AM ] sudo: pam_open_session: System error
[ 12/15/18 10:24:35 AM ] sudo: policy plugin failed session initialization
[ 12/15/18 10:25:46 AM ] Could not parse nodes file :
[ 12/15/18 10:25:46 AM ] sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm
[ 12/15/18 10:25:46 AM ]
[ 12/15/18 10:25:46 AM ] exit status: 1
[ 12/15/18 10:25:46 AM ] error:
[ 12/15/18 10:25:46 AM ] sudo: pam_open_session: System error
[ 12/15/18 10:25:46 AM ] sudo: policy plugin failed session initialization
[ 12/15/18 10:44:52 AM ] Could not parse nodes file :
[ 12/15/18 10:44:52 AM ] sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm
[ 12/15/18 10:44:52 AM ]
[ 12/15/18 10:44:52 AM ] exit status: 1
[ 12/15/18 10:44:52 AM ] error:
[ 12/15/18 10:44:52 AM ] sudo: pam_open_session: System error
[ 12/15/18 10:44:52 AM ] sudo: policy plugin failed session initialization
[ 12/15/18 10:45:09 AM ] Could not parse nodes file :
[ 12/15/18 10:45:09 AM ] sudo /usr/efm-2.1/bin/efm_root_functions readnodes efm
[ 12/15/18 10:45:09 AM ]
[ 12/15/18 10:45:09 AM ] exit status: 1
[ 12/15/18 10:45:09 AM ] error:
[ 12/15/18 10:45:09 AM ] sudo: pam_open_session: System error
[ 12/15/18 10:45:09 AM ] sudo: policy plugin failed session initialization
[root@HKDTUATDB01 efm-2.1]#



NOTE GUYS:-
This Error come's in Three nodes..Master,slave,witness also..
Status log is
[root@HKDTUATDB01 ~]# systemctl status efm-2.1 -l
? efm-2.1.service - EnterpriseDB Failover Manager 2.1
   Loaded: loaded (/usr/lib/systemd/system/efm-2.1.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2018-12-15 13:05:33 +08; 2min 48s ago
  Process: 20207 ExecStart=/bin/bash -c /usr/efm-2.1/bin/runefm.sh start ${CLUSTER} (code=exited, status=1/FAILURE)

Dec 15 13:05:32 HKDTUATDB01 systemd[1]: Starting EnterpriseDB Failover Manager 2.1...
Dec 15 13:05:33 HKDTUATDB01 sudo[20294]: pam_unix(sudo:account): account efm has expired (failed to change password)
Dec 15 13:05:33 HKDTUATDB01 sudo[20294]:      efm : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/efm-2.1/bin/efm_root_functions readnodes efm
Dec 15 13:05:33 HKDTUATDB01 sudo[20294]: PAM pam_set_item: NULL pam handle passed
Dec 15 13:05:33 HKDTUATDB01 systemd[1]: efm-2.1.service: control process exited, code=exited status=1
Dec 15 13:05:33 HKDTUATDB01 systemd[1]: Failed to start EnterpriseDB Failover Manager 2.1.
Dec 15 13:05:33 HKDTUATDB01 systemd[1]: Unit efm-2.1.service entered failed state.
Dec 15 13:05:33 HKDTUATDB01 systemd[1]: efm-2.1.service failed.

 

Level 2 Adventurer

Re: EFM-2.1 Not Started after applied patch

Hi nshah,

          but we have a 1+ year license..so cliebt cant able to do that..

Thanks for your reply

Level 2 Adventurer

Re: EFM-2.1 Not Started after applied patch

[postgres@******** ~]$ cat /etc/efm-2.1/efm.properties | grep -v ^#

efm.license=@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
db.user=efm_check_user
db.password.encrypted=***************************************
db.port=5006
db.database=efm_check_db
db.service.owner=postgres
db.service.name=
db.bin=/POSTGRES/9.6/as9.6/bin
db.recovery.conf.dir=/DATA/9.6/data
jdbc.ssl=false
jdbc.ssl.mode=verify-ca
user.email=db_monitoring@XXXX-yyyyy.com
script.notification=
bind.address=10.138.182.41:9006
admin.port=9008
is.witness=false
local.period=20
local.timeout=60
local.timeout.final=20
remote.timeout=20
node.timeout=60
pingServerIp=*.*.*.*
pingServerCommand=/bin/ping -q -c3 -w5
auto.allow.hosts=true
db.reuse.connection.count=0
auto.failover=true
auto.reconfigure=true
promotable=true
minimum.standbys=0
recovery.check.period=2
auto.resume.period=0
virtualIp=*.*.*.40
virtualIp.interface=eth0:0
virtualIp.netmask=255.255.255.0
script.fence=
script.post.promotion=
script.resumed=
script.db.failure=
script.master.isolated=
sudo.command=sudo
sudo.user.command=sudo -u %u
jgroups.loglevel=FINEST
efm.loglevel=FINEST
jvm.options=-Xmx32m
[postgres@HKDTUATDB01 ~]$

 

 

[postgres@************ efm-2.1]$ ls -lrth
total 56K
-rw-r--r--. 1 root root 13K Sep 19 2017 efm.properties.in
-rw-------. 1 root root 139 Sep 19 2017 efm.nodes.in
-rw-r--r--. 1 root root 13K Dec 15 12:37 efm.properties.bkp
-rw-r--r--. 1 root root 13K Dec 15 15:54 efm.properties
-rw-------. 1 root root 177 Dec 15 16:23 efm.nodes
[postgres@*********** efm-2.1]$

EDB Team Member

Re: EFM-2.1 Not Started after applied patch

Well, you found your problem. The 'efm' user can't run sudo any more on this system.

 

It would be good to know what you've done on the system to change the default behavior (the efm user does not install with a password and can run sudo normally by the sudoers conf file). I think you need to reach out to your support person for more help, but at least now we know what the issue is.

 

Cheers,

Bobby

 

Highlighted
EDB Team Member

Re: EFM-2.1 Not Started after applied patch


@bbissett wrote:

Well, you found your problem. The 'efm' user can't run sudo any more on this system.

 

It would be good to know what you've done on the system to change the default behavior (the efm user does not install with a password and can run sudo normally by the sudoers conf file). I think you need to reach out to your support person for more help, but at least now we know what the issue is.


 

For what it's worth, starting with EFM 3.0 you can run failover manager as the database user and not rely on sudo at all. There's a little more setup involved, but it all works. More info:

https://postgresrocks.enterprisedb.com/t5/Postgres-Gems/Running-EDB-Postgres-Failover-Manager-Withou...

 

Cheers,

Bobby

 

Tags (2)
Level 2 Adventurer

Re: EFM-2.1 Not Started after applied patch

Thanks,

          but testing server again its happend the same issue..not only this postgresql..some other database like mongoDB also..same...what actully i need to do..in my sudo part...

My efm user expired few months back..i suggest to reset of unix os user efm password.but my GM told last month also we did in patch..no issue happend in prod and uat...why we reset the password...

 

Please suggest what i do in my sudo part..or anything someother part's?

EDB Team Member

Re: EFM-2.1 Not Started after applied patch


@Diny2019 wrote:

          but testing server again its happend the same issue..not only this postgresql..some other database like mongoDB also..same...what actully i need to do..in my sudo part...

My efm user expired few months back..i suggest to reset of unix os user efm password.but my GM told last month also we did in patch..no issue happend in prod and uat...why we reset the password...

 

Please suggest what i do in my sudo part..or anything someother part's?


 

I don't know offhand what the solution is. It depends on how your system has sudo set up. My guess is you've run into this, which will affect anything that needs to run sudo:

 

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.6_release_notes/new_...

 

When we tried upgrading the OS we didn't see the problem -- something your the PAM configuration is affected.

 

FWIW, when trying to debug it, you don't actually need to run efm. As the efm user, run something like:

 

sudo <efminstall>/bin/efm_root_functions readnodes <clustername>

 

Cheers,

Bobby