基于StatefulSet控制器的MySQL一主多从
1 基于StatefulSet控制器的MySQL一主多从
1.0参考资料
[1] https://kubernetes.io/zh/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/
[2] https://www.kubernetes.org.cn/statefulset
[3]基于StatefulSet实现：https://kubernetes.io/zh/docs/tasks/run-applicationrun-replicated-stateful-application/
Pod调度运行时，如果应用不需要任何稳定的标示、有序的部署、删除和扩展，则应该使用一组无状态副本的控制器
来部署应用。
Deployment或ReplicaSet更适合无状态服务需求，而StatefulSet适合管理所有有状态的服务，比如MySQL、MongoDB集群等。
StatefulSet本质上是Deployment的一种变体，在v1.9版本中已成为GA版本，用以解决有状态服务的问题，StatefulSet所管理的Pod拥有固定的Pod名称，启停顺序，在StatefulSet中，Pod名字称为网络标识(hostname)，且必须要用到共享存储。
在Deployment中，与之对应的服务是service，而在StatefulSet中与之对应的headless service，headless service，即“无头服务”，与service的区别就是它没有Cluster IP，解析它的名称时将返回该Headless Service对应的全部Pod的Endpoint列表。
StatefulSet 特点：
-> 给每个pod分配固定且唯一的网络标识符
-> 给每个pod分配固定且持久化的外部存储
-> 对pod进行有序的部署和扩展
-> 对pod进有序的删除和终止
-> 对pod进有序的自动滚动更新
1.1 StatefulSet的组成部分
Headless Service：用来定义Pod网络标识( DNS domain)，指的是短的service(丢失了domainname)。
StatefulSet：定义具体应用，有多少个Pod副本，并为每个Pod定义了一个域名。
volumeClaimTemplates： 存储卷申请模板，创建PVC，指定pvc名称大小，将自动创建pvc，且pvc必须由存储类供应。
1.2 创建ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql
namespace: myserver
labels:
app: mysql
data:
master.cnf: |
# Apply this config only on the master.
[mysqld]
datadir=/var/lib/mysql
log_bin_trust_function_creators=1
lower_case_table_names=1
log_bin_trust_function_creators=1
lower_case_table_names=1
character-set-server=utf8
max_connections = 6000
max_user_connections = 6000
max_connect_errors = 4000
wait_timeout = 86400
interactive_timeout = 86400
table_open_cache = 512
max_allowed_packet=256M
sort_buffer_size = 2M
join_buffer_size = 2M
thread_cache_size = 8
query_cache_size = 32M
log_bin=ON
log_warnings = 1
slave.cnf: |
# Apply this config only on slaves.
[mysqld]
super-read-only
log_bin_trust_function_creators=1
1.3 创建Headless Service
# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:
namespace: myserver
name: mysql
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
clusterIP: None
selector:
app: mysql
---
# Client service for connecting to any MySQL instance for reads.
# For writes, you must instead connect to the master: mysql-0.mysql.
apiVersion: v1
kind: Service
metadata:
name: mysql-read
namespace: myserver
labels:
app: mysql
spec:
ports:
- name: mysql
port: 3306
selector:
app: mysql
1.3 创建Secret
apiVersion: v1
kind: Secret
metadata:
name: mysql-secret
namespace: myserver
labels:
app: mysql
type: Opaque
data:
password: MTIzNDU2Nzg=
#echo -n "12345678" | base64
1.4 创建StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
namespace: myserver
spec:
selector:
matchLabels:
app: mysql
serviceName: mysql
replicas: 3
template:
metadata:
labels:
app: mysql
spec:
initContainers:
- name: init-mysql #初始化容器1、基于当前pod name匹配角色是master还是slave，并动态生成
#image: harbor.linuxarchitect.io/myserver/mysql:5.7.36
image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/mysql:5.7.36
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
command:
- bash
- "-c"
- |
set -ex
# Generate mysql server-id from pod ordinal index.
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1 #匹配hostname的最后一位、最后是一个顺序叠
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
# Add an offset to avoid reserved server-id=0 value. #
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf #加100当做自己的s
# Copy appropriate conf.d files from config-map to emptyDir. #将适当的conf.d文件从
if [[ $ordinal -eq 0 ]]; then #如果是master、则cp master配置文件
cp /mnt/config-map/master.cnf /mnt/conf.d/
else #否则cp slave配置文件
cp /mnt/config-map/slave.cnf /mnt/conf.d/
fi
volumeMounts:
- name: conf #临时卷、emptyDir
mountPath: /mnt/conf.d
- name: config-map
mountPath: /mnt/config-map
- name: clone-mysql #初始化容器2、用于生成mysql配置文件、并从上一个pod完成首次的全量数,但是后期都是与master实现增量同步)
#image: harbor.linuxarchitect.io/myserver/xtrabackup:1.0
image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/xtrabackup:1.0
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
command:
- bash
- "-c"
- |
set -ex
# Skip the clone if data already exists.
[[ -d /var/lib/mysql/mysql ]] && exit 0 #判断条件如果成功、就是数据已经存在就退出
# Skip the clone on master (ordinal index 0).
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1 #判断主机名是否以一位或多位整数结尾、如果
ordinal=${BASH_REMATCH[1]} #BASH_REMATCH是Bash中的一个数组变量,用于存储最近使用运
[[ $ordinal -eq 0 ]] && exit 0 #如果判断条件成立、最后一位是0(master)则退出clone过
# Clone data from previous peer. #从上一个实例 clone完整数据
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
# Prepare the backup.xue
xtrabackup --prepare --target-dir=/var/lib/mysql #通过xtrabackup恢复binlog
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
containers:
- name: mysql #业务容器1(mysql主容器)
#image: harbor.linuxarchitect.io/myserver/mysql:5.7.36
image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/mysql:5.7.36
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
#- name: MYSQL_ALLOW_EMPTY_PASSWORD
# value: "1"
ports:
- name: mysql
containerPort: 3306
volumeMounts:
- name: data #挂载数据目录至/var/lib/mysql
mountPath: /var/lib/mysql
subPath: mysql
- name: conf #配置文件/etc/mysql/conf.d
mountPath: /etc/mysql/conf.d
resources: #资源限制
requests:
cpu: 500m
memory: 1Gi
livenessProbe: #存活探针
#exec:
# command: ["mysqladmin", "ping"]
tcpSocket:
port: 3306
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 3
readinessProbe: #就绪探针
#exec:
# Check we can execute queries over TCP (skip-networking is off).
#command: ["mysqladmin", "ping"]
#command: ["mysql", "-h", "127.0.0.1","-p","12345678", "-e", "SELECT 1"]
tcpSocket:
port: 3306
initialDelaySeconds: 30
periodSeconds: 5
timeoutSeconds: 2
successThreshold: 1
failureThreshold: 3
- name: xtrabackup #业务容器2(xtrabackup),用于后期同步master 的binglog并恢复数据
#image: harbor.linuxarchitect.io/myserver/xtrabackup:1.0
image: registry.cn-hangzhou.aliyuncs.com/zhangshijie/xtrabackup:1.0
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
ports:
- name: xtrabackup
containerPort: 3307
command:
- bash
- "-c"
- |
set -ex
cd /var/lib/mysql
# Determine binlog position of cloned data, if any.
# 从备份信息文件里读取MASTER_LOG_FILE和MASTER_LOG_POS这2个字段的值，用来拼接后续S
if [[ -f xtrabackup_slave_info ]]; then
# XtraBackup already generated a partial "CHANGE MASTER TO" query
# because we're cloning from an existing slave.
#如果xtrabackup_slave_info文件存在，说明这个备份数据来自于另一个Slave节点
# 这种情况下，XtraBackup工具在备份的时候，就已经在这个文件里自动生成了“CHANGE 接使用即可
mv xtrabackup_slave_info change_master_to.sql.in
# Ignore xtrabackup_binlog_info in this case (it's useless).
#所以就直接忽略xtrabackup_binlog_info(因为不需要)
rm -f xtrabackup_binlog_info
elif [[ -f xtrabackup_binlog_info ]]; then
# We're cloning directly from master. Parse binlog position.
## 如果只是存在xtrabackup_binlog_info文件，说明备份来自于Master节点，就需要解析
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
rm xtrabackup_binlog_info
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in #把两个字段的
fi
# Check if we need to complete a clone by starting replication.
#检查如果存在change_master_to.sql.in文件,就需要开始进行数据复制
if [[ -f change_master_to.sql.in ]]; then
echo "Waiting for mysqld to be ready (accepting connections)"
#等待MySQL容器启动就绪
until mysql -h 127.0.0.1 -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do sleep 1; do
echo "Initializing replication from clone position"
# In case of container restart, attempt this at-most-once.
mv change_master_to.sql.in change_master_to.sql.orig
#将change_master_to.sql.in文件名改为change_master_to.sql.orig, 避免后续重复执行
#执行CHANGE MASTER操作并启动SLAVE
mysql -h 127.0.0.1 -p${MYSQL_ROOT_PASSWORD} <<EOF
$(<change_master_to.sql.orig),
MASTER_HOST='mysql-0.mysql',
MASTER_USER='root',
MASTER_PASSWORD='${MYSQL_ROOT_PASSWORD}',
MASTER_CONNECT_RETRY=10;
START SLAVE;
EOF
fi
# Start a server to send backups when requested by peers. #监听在3307端口，用于为
exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --password
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 100m
memory: 100Mi
volumes:
- name: conf
emptyDir: {}
- name: config-map
configMap:
name: mysql
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "nfs"
resources:
requests:
storage: 10Gi
1.6 验证MySQL主从同步是否正常
# kubectl -n myserver exec -it mysql-2 -- sh
Defaulted container "mysql" out of: mysql, xtrabackup, init-mysql (init), clone-mysql (init)
# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 287
Server version: 5.7.36 MySQL Community Server (GPL)
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: mysql-0.mysql
Master_User: root
Master_Port: 3306
Connect_Retry: 10
Master_Log_File: ON.000004
Read_Master_Log_Pos: 154
Relay_Log_File: mysql-2-relay-bin.000002
Relay_Log_Pos: 313
Relay_Master_Log_File: ON.000004
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
确认Slave_IO_Running和Slave_SQL_Running均为Yes，表示主从同步正常。
1.7 高可用测试：
分别删除pod中的master与slave节点，验证MySQL服务最终能否恢复到正常状态。
1.7.1 删除MySQL master节点
root@master01:~# kubectl delete pod mysql-0 -n myserver
pod "mysql-0" deleted from myserver namespace
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m9s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 1/2 Running 0 12s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m12s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 1/2 Running 0 15s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m15s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 1/2 Running 0 18s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m16s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 1/2 Running 0 19s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m18s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 1/2 Running 0 21s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m19s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 1/2 Running 0 22s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m21s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 1/2 Running 0 24s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~# watch kubectl get pod -n myserver
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m52s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 2/2 Running 0 55s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
可以发现,mysql-0被删除后很快又重新创建了新的pod来替代它，并且最终状态变为Running，说明MySQL服务恢复正常。
1.7.2 删除MySQL slave节点
root@master01:~# watch kubectl get pod -n myserver
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (9m52s ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 2/2 Running 0 55s
mysql-1 2/2 Running 0 19m
mysql-2 2/2 Running 0 18m
root@master01:~# kubectl delete pod mysql-1 -n myserver
pod "mysql-1" deleted from myserver namespace
root@master01:~#
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (12m ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 2/2 Running 0 3m17s
mysql-1 1/2 Running 0 32s
mysql-2 2/2 Running 0 21m
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (12m ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 2/2 Running 0 3m23s
mysql-1 1/2 Running 0 38s
mysql-2 2/2 Running 0 21m
root@master01:~# watch kubectl get pod -n myserver
root@master01:~# kubectl get pod -n myserver
NAME READY STATUS RESTARTS AGE
dns-debug 1/1 Running 70 (12m ago) 3d18h
myserver-jenkins-deployment-7ff8dfd7bc-nxcj9 1/1 Running 0 18h
mysql-0 2/2 Running 0 3m31s
mysql-1 2/2 Running 0 46s
mysql-2 2/2 Running 0 21m
可以发现,mysql-1被删除后很快又重新创建了新的pod来替代它，并且最终状态变为Running，说明MySQL服务恢复正常。
浙公网安备 33010602011771号