"GaleraPCSClusterRecover" - Views: 604 · Hits: 604 - Type: Public

Just killall -9 mysqld on controller-2

[[email protected] ~]# ps -ef | grep mysqld
root       21964       1  0 18:37 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql --log-error=/var/log/mysqld.log --user=mysql --open-files-limit=16384 --wsrep-cluster-address=gcomm://
mysql      22284   21964  0 18:37 ?        00:00:47 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep_on=ON --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep-cluster-address=gcomm:// --log-error=/var/log/mysqld.log --open-files-limit=16384 --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
root      464813  462721  0 20:39 pts/0    00:00:00 grep --color=auto mysqld

[[email protected] ~]# killall -9 mysqld
[[email protected] ~]# ps -ef | grep mysqld
root      468000  462721  0 20:40 pts/0    00:00:00 grep --color=auto mysqld

[[email protected] ~]# clustercheck
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 36


Run on controller-0

[[email protected] ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-1 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Thu Mar  2 20:41:00 2017		Last change: Thu Mar  2 20:13:55 2017 by hacluster via crmd on overcloud-controller-0

3 nodes and 19 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 ]
     Slaves: [ overcloud-controller-2 ]
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.24.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
 ip-10.0.0.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
 ip-172.16.2.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2
 ip-172.16.2.9	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
 ip-172.16.1.12	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
 ip-172.16.3.6	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started overcloud-controller-0

Failed Actions:
* galera_monitor_10000 on overcloud-controller-2 'not running' (7): call=74, status=complete, exitreason='none',
    last-rc-change='Thu Mar  2 20:40:42 2017', queued=0ms, exec=0ms


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[[email protected] ~]# pcs resource cleanup galera-master
Cleaning up galera:0 on overcloud-controller-0, removing fail-count-galera
Cleaning up galera:0 on overcloud-controller-1, removing fail-count-galera
Cleaning up galera:0 on overcloud-controller-2, removing fail-count-galera
Waiting for 3 replies from the CRMd... OK


[[email protected] ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-1 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum
Last updated: Thu Mar  2 20:42:07 2017		Last change: Thu Mar  2 20:41:37 2017 by hacluster via crmd on overcloud-controller-0

3 nodes and 19 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.24.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
 ip-10.0.0.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
 ip-172.16.2.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2
 ip-172.16.2.9	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
 ip-172.16.1.12	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
 ip-172.16.3.6	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started overcloud-controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled


Verify on controller-2

[[email protected] ~]# clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 32

Galera cluster node is synced.

  pcsd: active/enabled