I am trying to set up a GTID multi-master replication between On-Prem MySQL and AWS RDS MySQL, so RDS MySQL should act as replica for On-Prem MySQL and vice versa.
Due to the fact that we have a massive database size , we decided to create a On-Prem MySQL backup with Percona Xtrabackup, copy the backup to S3 and create new brand RDS MySQL from S3 backup.
I created custom database parameter group and put the same parameters as we have in the On-Prem .cnf file
Here is the binlog and gtid parameters used in on-prem .cnf and RDS database parameter group:
log_bin = = /var/lib/mysql/mysql-bin.logsync_binlog = 0gtid-mode=ONenforce-gtid-consistencylog-slave-updatesexpire_logs_days = 10 # added only in On-PremDuring the restoring RDS from S3 backup, I set backup retention period to 7 days.
After restoring RDS instance, I see in logs, that database has been recovered from binlog file, binlog file position and GTID position as well
Then I configured replication between On-Prem MySQL as the master and RDS MySQL as the slave, and it works without errors.
But, when I configure replication from RDS MySQL as the master and On-Prem MySQL as slave, I receive an error on On-Prem MySQL:
Seconds_Behind_Master: NULLMaster_SSL_Verify_Server_Cert: NoLast_IO_Errno: 1236Last_IO_Error: `Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires. Replicate the missing transactions from elsewhere, or provision a new slave from backup. Consider increasing the master's binary log expiration period. The GTID set sent by the slave is '5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-5', and the missing transactions are 'a82ebef1-e265-11ec-964b-0af5faa008d7:1-44'.'`Last_SQL_Errno: 0Last_SQL_Error:Replicate_Ignore_Server_Ids:Master_Server_Id: 219284118Master_UUID: a82ebef1-e265-11ec-964b-0af5faa008d7Master_Info_File: /var/lib/mysql/master.infoSQL_Delay: 0SQL_Remaining_Delay: NULL...Master_Retry_Count: 86400Master_Bind:Last_IO_Error_Timestamp: 220602 12:37:06Last_SQL_Error_Timestamp:Master_SSL_Crl:Master_SSL_Crlpath:Retrieved_Gtid_Set:Executed_Gtid_Set: 5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-5Auto_Position: 1Replicate_Rewrite_DB:Channel_Name:Master_TLS_Version:1 row in set (0.00 sec)After some researching I found that once RDS will become available, RDS starts purging the binlog as soon as possible, because the default parameter binlog retention hours is set to zero.
And even if I specify rds_set_configuration('binlog retention hours') right after restoring RDS MySQL from s3, I still receiving this error after trying to start slave on On-Prem MySQL
Here below my specification:
On-Prem MySQL version - 5.7.29 RDS MySQL version - 5.7.37On-Prem MySQL instance with ID - 5ea52da6-e25c-11ec-ac07-0aa67b862ee7.
RDS Master is AWS RDS for MySQL instance with ID - a82ebef1-e265-11ec-964b-0af5faa008d7
------------- This is my On-Prem MySQL -------------mysql> show global variables like 'gtid_executed'\G*************************** 1. row ***************************Variable_name: gtid_executedValue: 5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-61 row in set (0.01 sec)mysql> show global variables like 'gtid_purged'\G*************************** 1. row ***************************Variable_name: gtid_purgedValue: 5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-41 row in set (0.00 sec)mysql> show master status\G*************************** 1. row ***************************File: mysql-bin.000004Position: 194Binlog_Do_DB:Binlog_Ignore_DB:Executed_Gtid_Set: 5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-61 row in set (0.00 sec)------------- This is my RDS MySQL -------------mysql> show global variables like 'gtid_executed'\G*************************** 1. row ***************************Variable_name: gtid_executedValue: 5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-6, a82ebef1-e265-11ec-964b-0af5faa008d7:1-521 row in set (0.00 sec)mysql> show global variables like 'gtid_purged'\G*************************** 1. row ***************************Variable_name: gtid_purgedValue: 5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-6, a82ebef1-e265-11ec-964b-0af5faa008d7:1-501 row in set (0.00 sec)mysql> show master status\G*************************** 1. row ***************************File: mysql-bin-changelog.000025Position: 234Binlog_Do_DB:Binlog_Ignore_DB:Executed_Gtid_Set: 5ea52da6-e25c-11ec-ac07-0aa67b862ee7:1-6, a82ebef1-e265-11ec-964b-0af5faa008d7:1-521 row in set (0.00 sec)Did I miss something?
