I have two MySQL server which are running on same group replication. The setup had been done by below steps:
- The first server is production server with lot of data.
- I set it readonly and dump the data, then restore it on the fallback MySQL server
- After restored, I executed "START GROUP_REPLICATION" and it success join to the group. All data are sync well between two server.
But I found an other error: Every time I reload the fallback MySQL (by restart services) it will auto join to group but stuck at RECOVERING forever, I waited for 3 days but it still in RECOVERING. I checked on the log and don't see any error on both server, everything look good except the fallback is running as readonly and stay at RECOVERING.
What step did I missed?
My group configuration is (I followed the instruction from DigitalOcean help page at https://www.digitalocean.com/community/tutorials/how-to-configure-mysql-group-replication-on-ubuntu-16-04):
BINARY LOGGING # log_bin = /data/databases/mysql_bin199 expire_logs_days = 14
sync_binlog = 1 binlog_format = ROW
General replication settings gtid_mode = ON enforce_gtid_consistency = ON master_info_repository = TABLE relay_log_info_repository = TABLE binlog_checksum = NONE log_slave_updates = ON
log_bin = binlog
binlog_format = ROW transaction_write_set_extraction = XXHASH64 loose-group_replication_bootstrap_group = OFF
loose-group_replication_start_on_boot = ON loose-group_replication_ssl_mode = REQUIRED loose-group_replication_recovery_use_ssl = 1
Shared replication group configuration loose-group_replication_group_name =
"9dc4ae01-6664-437a-83f8-80546d58e025" loose-group_replication_ip_whitelist = "172.AAA.BBB.166,138.AAA.BBB.199" loose-group_replication_group_seeds = "172.AAA.BBB.166:33061,138.AAA.BBB.199:33061"
Single or Multi-primary mode? Uncomment these two lines
for multi-primary mode, where any host can accept writes loose-group_replication_single_primary_mode = OFF
loose-group_replication_enforce_update_everywhere_checks = ON
Host specific replication configuration server_id = 2 report_host = "138.AAA.BBB.199" loose-group_replication_local_address =
"138.AAA.BBB.199:33061"
Below is MySQL log on first server:
2018-06-08T06:10:12.167400Z 0 [Warning] Plugin group_replication
reported: 'Members removed from the group: 138.AAA.BBB.199:3306'
2018-06-08T06:10:12.167475Z 0 [Note] Plugin group_replication
reported: 'Group membership changed to 172.AAA.BBB.166:3306 on view
15271181169364149:11.' 2018-06-08T06:11:59.032666Z 0 [Note] Plugin
group_replication reported: 'Members joined the group:
138.AAA.BBB.199:3306' 2018-06-08T06:11:59.032722Z 0 [Note] Plugin group_replication reported: 'Group membership changed to
172.AAA.BBB.166:3306, 138.AAA.BBB.199:3306 on view 15271181169364149:12.'
Below is the MySQL log on fallback server:
2018-06-11T09:22:57.490896Z 0 [Warning] option 'max_allowed_packet':
unsigned value 3221225472 adjusted to 1073741824
2018-06-11T09:22:57.490942Z 0 [Warning] The use of InnoDB is mandatory
since MySQL 5.7. The former options like '--innodb=0/1/OFF/ON' or
'--skip-innodb' are ignored. 2018-06-11T09:22:57.491057Z 0 [Warning]
The syntax '--log_warnings/-W' is deprecated and will be removed in a
future release. Please use '--log_error_verbosity' instead.
2018-06-11T09:22:57.491098Z 0 [Warning] TIMESTAMP with implicit
DEFAULT value is deprecated. Please use
--explicit_defaults_for_timestamp server option (see documentation for more details). 2018-06-11T09:22:57.492972Z 0 [Note] /usr/sbin/mysqld
(mysqld 5.7.22-log) starting as process 31633 ...
2018-06-11T09:22:57.500063Z 0 [Warning] InnoDB: Using
innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be
removed in future releases. Please use READ COMMITTED transaction
isolation level instead; Please refer to
http://dev.mysql.com/doc/refman/5.7/en/set-transaction.html
2018-06-11T09:22:57.500175Z 0 [Note] InnoDB: PUNCH HOLE support
available 2018-06-11T09:22:57.500191Z 0 [Note] InnoDB: Mutexes and
rw_locks use GCC atomic builtins 2018-06-11T09:22:57.500200Z 0 [Note]
InnoDB: Uses event mutexes 2018-06-11T09:22:57.500205Z 0 [Note]
InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-06-11T09:22:57.500209Z 0 [Note] InnoDB: Compressed tables use
zlib 1.2.3 2018-06-11T09:22:57.500213Z 0 [Note] InnoDB: Using Linux
native AIO 2018-06-11T09:22:57.500430Z 0 [Note] InnoDB: Number of
pools: 1 2018-06-11T09:22:57.500575Z 0 [Note] InnoDB: Using CPU crc32
instructions 2018-06-11T09:22:57.501015Z 0 [ERROR] InnoDB: Failed to
create check sector file, errno:13 Please confirm O_DIRECT is
supported and remove the file /data/check_sector_size if it exists.
2018-06-11T09:22:57.502305Z 0 [Note] InnoDB: Initializing buffer pool,
total size = 4G, instances = 8, chunk size = 128M
2018-06-11T09:22:57.799065Z 0 [Note] InnoDB: Completed initialization
of buffer pool 2018-06-11T09:22:57.857325Z 0 [Note] InnoDB: If the
mysqld execution user is authorized, page cleaner thread priority can
be changed. See the man page of setpriority().
2018-06-11T09:22:57.870317Z 0 [Note] InnoDB: Highest supported file
format is Barracuda. 2018-06-11T09:22:58.081570Z 0 [Note] InnoDB:
Creating shared tablespace for temporary tables
2018-06-11T09:22:58.081656Z 0 [Note] InnoDB: Setting file
'/data/databases/ibtmp1' size to 12 MB. Physically writing the file
full; Please wait ... 2018-06-11T09:22:58.116190Z 0 [Note] InnoDB:
File '/data/databases/ibtmp1' size is now 12 MB.
2018-06-11T09:22:58.117279Z 0 [Note] InnoDB: 96 redo rollback
segment(s) found. 96 redo rollback segment(s) are active.
2018-06-11T09:22:58.117293Z 0 [Note] InnoDB: 32 non-redo rollback
segment(s) are active. 2018-06-11T09:22:58.117670Z 0 [Note] InnoDB:
Waiting for purge to start 2018-06-11T09:22:58.168094Z 0 [Note]
InnoDB: 5.7.22 started; log sequence number 51745666191
2018-06-11T09:22:58.168309Z 0 [Note] InnoDB: Loading buffer pool(s)
from /data/databases/ib_buffer_pool 2018-06-11T09:22:58.168558Z 0
[Note] Plugin 'FEDERATED' is disabled. 2018-06-11T09:22:58.183268Z 0
[Warning] CA certificate /etc/mysql/mysql-ssl/ca-cert.pem is self
signed. 2018-06-11T09:22:58.184615Z 0 [Note] Server hostname
(bind-address): '138.AAA.BBB.199'; port: 3306
2018-06-11T09:22:58.184636Z 0 [Note] - '138.AAA.BBB.199' resolves to
'138.AAA.BBB.199'; 2018-06-11T09:22:58.184668Z 0 [Note] Server socket
created on IP: '138.AAA.BBB.199'. 2018-06-11T09:22:58.186203Z 0
[Warning] 'user' entry 'mysql.session@localhost' ignored in
--skip-name-resolve mode. 2018-06-11T09:22:58.186220Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve
mode. 2018-06-11T09:22:58.186238Z 0 [Warning] 'user' entry
'phpmadsys@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.186260Z 0 [Warning] 'user' entry
'phpmyadmin@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.186308Z 0 [Warning] 'db' entry 'performance_schema
mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.186313Z 0 [Warning] 'db' entry 'sys
mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.186318Z 0 [Warning] 'db' entry 'phpmyadmin
phpmadsys@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.186322Z 0 [Warning] 'db' entry 'performance_schema
datadog@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.186327Z 0 [Warning] 'db' entry 'phpmyadmin
phpmyadmin@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.186340Z 0 [Warning] 'proxies_priv' entry '@
root@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.188628Z 0 [Warning] 'tables_priv' entry 'user
mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.188649Z 0 [Warning] 'tables_priv' entry
'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-11T09:22:58.192624Z 0 [Warning] Neither --relay-log nor
--relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use
'--relay-log=dvm02-relay-bin' to avoid this problem.
2018-06-11T09:22:58.206545Z 0 [Note] Event Scheduler: Loaded 0 events
2018-06-11T09:22:58.206745Z 0 [Note] /usr/sbin/mysqld: ready for
connections. Version: '5.7.22-log' socket:
'/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server
(GPL) 2018-06-11T09:22:58.207175Z 2 [Note] Plugin group_replication
reported: 'Group communication SSL configuration:
group_replication_ssl_mode: "REQUIRED"; server_key_file:
"/etc/mysql/mysql-ssl/server-key.pem"; server_cert_file:
"/etc/mysql/mysql-ssl/server-cert.pem"; client_key_file:
"/etc/mysql/mysql-ssl/server-key.pem"; client_cert_file:
"/etc/mysql/mysql-ssl/server-cert.pem"; ca_file:
"/etc/mysql/mysql-ssl/ca-cert.pem"; ca_path: ""; cipher: "";
tls_version: "TLSv1,TLSv1.1"; crl_file: ""; crl_path: ""'
2018-06-11T09:22:58.207378Z 2 [Warning] Plugin group_replication
reported: '[GCS] Automatically adding IPv4 localhost address to the
whitelist. It is mandatory that it is added.'
2018-06-11T09:22:58.207820Z 2 [Note] Plugin group_replication
reported: 'Initialized group communication with configuration:
group_replication_group_name: "9dc4ae01-6664-437a-83f8-80546d58e025";
group_replication_local_address: "138.AAA.BBB.199:33061";
group_replication_group_seeds:
"172.AAA.BBB.166:33061,138.AAA.BBB.199:33061";
group_replication_bootstrap_group: false;
group_replication_poll_spin_loops: 0;
group_replication_compression_threshold: 1000000;
group_replication_ip_whitelist: "172.AAA.BBB.166,138.AAA.BBB.199"'
2018-06-11T09:22:58.207853Z 2 [Note] Plugin group_replication
reported: '[GCS] Configured number of attempts to join: 0'
2018-06-11T09:22:58.207859Z 2 [Note] Plugin group_replication
reported: '[GCS] Configured time between attempts to join: 5 seconds'
2018-06-11T09:22:58.207878Z 2 [Note] Plugin group_replication
reported: 'Member configuration: member_id: 2; member_uuid:
"822868f9-52a0-11e8-aa0e-1e45f9551f27"; single-primary mode: "false";
group_replication_auto_increment_increment: 7; '
2018-06-11T09:22:58.209024Z 3 [Note] 'CHANGE MASTER TO FOR CHANNEL
'group_replication_applier' executed'. Previous state
master_host='', master_port= 0, master_log_file='',
master_log_pos= 4, master_bind=''. New state master_host='',
master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2018-06-11T09:22:58.216904Z 6 [Note] Slave SQL thread for channel
'group_replication_applier' initialized, starting replication in log
'FIRST' at position 0, relay log
'./dvm02-relay-bin-group_replication_applier.000071' position: 4
2018-06-11T09:22:58.216931Z 2 [Note] Plugin group_replication
reported: 'Group Replication applier module successfully initialized!'
2018-06-11T09:22:58.241357Z 0 [Note] Plugin group_replication
reported: 'XCom protocol version: 3' 2018-06-11T09:22:58.241397Z 0
[Note] Plugin group_replication reported: 'XCom initialized and ready
to accept incoming connections on port 33061'
2018-06-11T09:22:59.213826Z 0 [Note] InnoDB: Buffer pool(s) load
completed at 180611 11:22:59 2018-06-11T09:23:00.316791Z 0 [Note]
Plugin group_replication reported: 'Group membership changed to
172.AAA.BBB.166:3306, 138.AAA.BBB.199:3306 on view 15271181169364149:16.'
