About Clailson

Hi guys,   We have a client who is having problems when the standby is being started. The standby was stopped to reboot the server. When the cluster service was restarted, the WALs files started to be restored from the archive, a consistency point was reached, and the restore of the WALs continued until the error "PANIC: WAL contains references to invalid pages" occurred.   Follows a part of the log.   2018-11-27 16:58:59 -02 [122610]: [49-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=00000 LOG:  restored log file "0000000200004C270000000D" from archive 2018-11-27 16:59:00 -02 [123043]: [1-1] user=postgres@10.0.100.21(33464),db=postgres remoto=10.0.100.21 Sessao=5bfd93f4.1e0a3 T_ID=0 VT_ID= tag= sqlstate=57P03 FATAL:  the database system is starting up 2018-11-27 16:59:00 -02 [123044]: [1-1] user=postgres@[local],db=template1 remoto=[local] Sessao=5bfd93f4.1e0a4 T_ID=0 VT_ID= tag= sqlstate=57P03 FATAL:  the database system is starting up receiving incremental file list 0000000200004C270000000E sent 30 bytes  received 16,779,379 bytes  11,186,272.67 bytes/sec total size is 16,777,216  speedup is 1.00 2018-11-27 16:59:01 -02 [122610]: [50-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=00000 LOG:  restored log file "0000000200004C270000000E" from archive 2018-11-27 16:59:01 -02 [123051]: [1-1] user=postgres@[local],db=template1 remoto=[local] Sessao=5bfd93f5.1e0ab T_ID=0 VT_ID= tag= sqlstate=57P03 FATAL:  the database system is starting up receiving incremental file list 0000000200004C270000000F sent 30 bytes  received 16,779,379 bytes  33,558,818.00 bytes/sec total size is 16,777,216  speedup is 1.00 2018-11-27 16:59:01 -02 [122610]: [51-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=00000 LOG:  restored log file "0000000200004C270000000F" from archive 2018-11-27 16:59:01 -02 [122610]: [52-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=00000 LOG:  consistent recovery state reached at 4C27/F2159B8 2018-11-27 16:59:01 -02 [122600]: [4-1] user=@,db= remoto= Sessao=5bfd93cd.1dee8 T_ID=0 VT_ID= tag= sqlstate=00000 LOG:  database system is ready to accept read only connections receiving incremental file list 0000000200004C2700000010 sent 30 bytes  received 16,779,379 bytes  33,558,818.00 bytes/sec total size is 16,777,216  speedup is 1.00 2018-11-27 16:59:02 -02 [122610]: [53-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=00000 LOG:  restored log file "0000000200004C2700000010" from archive 2018-11-27 16:59:02 -02 [122610]: [54-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=01000 WARNING:  page 1277624320 of relation base/16400/741450880 does not exist 2018-11-27 16:59:02 -02 [122610]: [55-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=01000 CONTEXT:  xlog redo Btree/DELETE: 49 items 2018-11-27 16:59:02 -02 [122610]: [56-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=XX000 PANIC:  WAL contains references to invalid pages 2018-11-27 16:59:02 -02 [122610]: [57-1] user=@,db= remoto= Sessao=5bfd93ce.1def2 T_ID=0 VT_ID=1/0 tag= sqlstate=XX000 CONTEXT:  xlog redo Btree/DELETE: 49 items 2018-11-27 16:59:03 -02 [122600]: [5-1] user=@,db= remoto= Sessao=5bfd93cd.1dee8 T_ID=0 VT_ID= tag= sqlstate=00000 LOG:  startup process (PID 122610) was terminated by signal 6: Aborted 2018-11-27 16:59:03 -02 [122600]: [6-1] user=@,db= remoto= Sessao=5bfd93cd.1dee8 T_ID=0 VT_ID= tag= sqlstate=00000 LOG:  terminating any other active server processes 2018-11-27 16:59:03 -02 [123058]: [1-1] user=postgres@10.0.100.21(33470),db=postgres remoto=10.0.100.21 Sessao=5bfd93f5.1e0b2 T_ID=0 VT_ID=2/0 tag=idle sqlstate=57P02 WARNING:  terminating connection because of crash of another server process 2018-11-27 16:59:03 -02 [123058]: [2-1] user=postgres@10.0.100.21(33470),db=postgres remoto=10.0.100.21 Sessao=5bfd93f5.1e0b2 T_ID=0 VT_ID=2/0 tag=idle sqlstate=57P02 DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2018-11-27 16:59:03 -02 [123058]: [3-1] user=postgres@10.0.100.21(33470),db=postgres remoto=10.0.100.21 Sessao=5bfd93f5.1e0b2 T_ID=0 VT_ID=2/0 tag=idle sqlstate=57P02 HINT:  In a moment you should be able to reconnect to the database and repeat your command. This is already the second time that this error occurs in less than 15 days. In the 2 times, the service was stopped and after some time, reinitiated.   The standy service was restarted a second time after the error, and it occurred again, but in a later WAL segment. The first error was while applying the segment 0000000200004C2700000010, and on the second attempt at 0000000200004C2700000014.   The client is using version 9.5.14.20 of ppas.   I found this thread in the community list. Can this be related?   https://www.postgresql.org/message-id/flat/CAFh8B%3DkvY2BYgikf%3DtpjUGD2h0rsm8GosOeU4TeYR%2BQ6YK-mjw%40mail.gmail.com#745cbb3949e1f66242110d17bf889868   The shared_preload_libraries parameter is configured in the standby as below.   shared_preload_libraries = '$libdir/dbms_pipe,$libdir/edb_gen,$libdir/sql-profiler,$libdir/plugin_debugger,$libdir/pg_stat_statements,$libdir/pg_buffercache'   Maybe this can be a tip to solve the problem.   I appreciate any help and tell me if you need any more information.   Thanks, Clailson de Almeida. ... View more