Kubernetes ETCD Cluster 設定

Kubernetes 架構主機環境:

在 /etc/hosts 新增:

192.168.60.153 c7-k8s01    #master

192.168.60.154 c7-k8s02

192.168.60.155 c7-k8s03

192.168.60.156 c7-k8s04

TECD Cluster 設定

每台 kubernetes node 都要安裝 etcd 套件

yum install -y etcd

第一台叢集設定:

/etc/etcd/etcd.conf

ETCD_NAME=etcd01

ETCD_DATA_DIR="/var/lib/etcd/etcd01"

ETCD_LISTEN_PEER_URLS="http://192.168.60.153:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.60.153:2379,http://127.0.0.1:2379 "

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.60.153:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://192.168.60.153:2380,etcd02=http://192.168.60.154:2380,etcd03=http://192.168.60.155:2380"    #初始 Cluster 成員列表

ETCD_INITIAL_CLUSTER_STATE="new"    #初始 Cluster 狀態，new 為新建 Cluster

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"    # Cluster 名稱

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.60.153:2379"

第二台叢集設定:

ETCD_NAME=etcd02

ETCD_DATA_DIR="/var/lib/etcd/etcd02"

ETCD_LISTEN_PEER_URLS="http://192.168.60.154:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.60.154:2379,http://127.0.0.1:2379 "

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.60.154:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://192.168.60.153:2380,etcd02=http://192.168.60.154:2380,etcd03=http://192.168.60.155:2380"    #初始 Cluster 成員列表

ETCD_INITIAL_CLUSTER_STATE="new"    #初始 Cluster 狀態，new 為新建 Cluster

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"    # Cluster 名稱

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.60.154:2379"

第三台叢集設定:

ETCD_NAME=etcd03

ETCD_DATA_DIR="/var/lib/etcd/etcd03"

ETCD_LISTEN_PEER_URLS="http://192.168.60.155:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.60.155:2379,http://127.0.0.1:2379 "

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.60.155:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://192.168.60.153:2380,etcd02=http://192.168.60.154:2380,etcd03=http://192.168.60.155:2380"    #初始 Cluster 成員列表

ETCD_INITIAL_CLUSTER_STATE="new"    #初始 Cluster 狀態，new 為新建 Cluster

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"    # Cluster 名稱

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.60.155:2379"

修改啟動設定檔

/usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \
--name=\"${ETCD_NAME}\" \
--data-dir=\"${ETCD_DATA_DIR}\" \
--listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
--advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
--initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
--initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \
--initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" \
--listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

啟動 ETCD 服務

    systemctl restart etcd.service

    systemctl enable etcd.service

    systemctl status etcd.service

觀察 ETCD Cluster 狀態

# etcdctl member list

4be996007148b4a5: name=etcd01 peerURLs=http://192.168.60.153:2380 clientURLs=http://192.168.60.153:2379 isLeader=true

4f79c66abde833f4: name=etcd02 peerURLs=http://192.168.60.154:2380 clientURLs=http://192.168.60.154:2379 isLeader=false

5b938308c73bbf65: name=etcd03 peerURLs=http://192.168.60.155:2380 clientURLs=http://192.168.60.155:2379 isLeader=false

# etcdctl cluster-health

member 4be996007148b4a5 is healthy: got healthy result from http://192.168.60.153:2379

member 4f79c66abde833f4 is healthy: got healthy result from http://192.168.60.154:2379

member 5b938308c73bbf65 is healthy: got healthy result from http://192.168.60.155:2379

將 master apiserver 設定指向 etcd cluster 成員

vi /etc/kubernetes/apiserver

ETCD_INITIAL_CLUSTER="etcd01=http://192.168.60.153:2380,etcd02=http://192.168.60.154:2380,etcd03=http://192.168.60.155:2380"

新增 etcd cluster 節點:etcd04

ETCD_INITIAL_CLUSTER_STATE 初始化值全部為 new,後續加入的 TECD 成員依

etcdctl member add <TECD_NAME> http://new_etcd_cluster:2380

指令產生的設定值來修改 etcd.conf 設定檔

在已有 Cluster 增加新節點 etcd04 的資料

etcdctl member add etcd04 http://192.168.60.156:2380

Added member named etcd04 with ID 4054n89e532b87bc to cluster

ETCD_NAME="etcd04"

ETCD_INITIAL_CLUSTER="etcd01=http://192.168.60.153:2380,etcd02=http://192.168.60.154:2380,etcd03=http://192.168.60.155:2380,etcd04=http://192.168.60.156:2380"

ETCD_INITIAL_CLUSTER_STATE="existing"

增加新的節點資料後產生配置 ETCD_NAME、ETCD_INITIAL_CLUSTE、ETCD_INITIAL_CLUSTER_STATE 用於新的 etcd04 設定檔

修改 /etc/etcd/etcd.conf 設定檔:

ETCD_NAME=etcd04

ETCD_DATA_DIR="/var/lib/etcd/etcd04"

ETCD_LISTEN_PEER_URLS="http://192.168.60.156:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.60.156:2379,http://127.0.0.1:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.60.156:2380"

ETCD_INITIAL_CLUSTER="etcd01=http://192.168.60.153:2380,etcd02=http://192.168.60.154:2380,etcd03=http://192.168.60.155:2380,etcd04=http://192.168.60.156:2380"    #每台的 etcd.conf 都必須修改 

ETCD_INITIAL_CLUSTER_STATE="existing"    #existing 表示加入己存在的 Cluster

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.60.156:2379"

啟動 etcd 服務可以查看 Cluster 多了一個新的節點 etcd04

systemctl restart etcd.service

systemctl enable etcd.service

觀察是否出現新加入的 etcd cluster 成員

etcdctl member list

4be996007148b4a5: name=etcd01

4f79c66abde833f4: name=etcd02

5b938308c73bbf65: name=etcd03

a153e6ec785245d6: name=etcd04

確認 etcd cluster 狀態

etcdctl cluster-health

member 4be996007148b4a5 is healthy

member 4f79c66abde833f4 is healthy

member 5b938308c73bbf65 is healthy

member a153e6ec785245d6 is healthy