If the below messages are encountered while starting the database server
2019-04-30 09:00:26 EDT FATAL: the database system is starting up
2019-04-30 09:00:27 EDT FATAL: the database system is starting up
2019-04-30 09:00:27 EDT LOG: invalid checkpoint record
2019-04-30 09:00:27 EDT FATAL: could not locate required checkpoint record
2019-04-30 09:00:27 EDT HINT: If you are not restoring from a backup, try removing the file "/xtra200/edb/postgres_mode/as9.6/data/backup_label".
2019-04-30 09:00:27 EDT LOG: startup process (PID 13039) exited with exit code 1
2019-04-30 09:00:27 EDT LOG: aborting startup due to startup process failure
2019-04-30 09:00:27 EDT LOG: database system is shut down
First step, Don't attempt to delete the backup_lable. Doing so, without checking few things may corrupt your database.
If you are restoring from a backup then make sure that you have all the WAL files that are generated from the time
the backup is started, we can get this information from the backup_lable's 'START WAL LOCATION' wal file and
try to start the database which should fix the issues.
What happens if we remove backup_lable file?
If you are starting the database by restoring a backup and performing PITR and if you remove the
backup_lable file then database server might think it is coming from the crash recovery and starts
to restore from the recent checkpoint instead of replaying from the time when the backup started
that can cause the inconsistency.
Removing the backup_lable can work in below scenario:
In case, the database server is kept in backup mode using the pg_start_backup and fails to execute the pg_stop_backup
it leads to a backup_lable pesence in the $PGDATA directory and in future/later when user attempts to restart the database
it end up looking into backup_lable file which contains old checkpoint related information and fails to start.
You need to check the timestamp of the backup_lable and checkpoint information from pg_controldata
to understand whether the backup_lable is old
Now check the $PGBIN/pg_controldata -D <data directory>