金慶的專欄

kubernetes導(dǎo)出有狀態(tài)服務(wù)

(金慶的專欄 2018.7)

網(wǎng)游服務(wù)器中的房間服務(wù)器是有狀態(tài)服務(wù)器，可以用 kubernetes statefulset 開啟多個(gè)實(shí)例。

為了讓客戶端能夠直連房間服務(wù)器，除了 statefulset 要求的 headless 服務(wù)，
還須為每個(gè)實(shí)例創(chuàng)建 NodePort 類型的服務(wù), 并且選擇Pod和禁止轉(zhuǎn)發(fā)。

下面 bootcamp.yml 先創(chuàng)建了 bootcamp headless 服務(wù)(clusterIP: None),
又創(chuàng)建了 bootcamp StatefulSet, 實(shí)例個(gè)數(shù)為 2.
然后創(chuàng)建 bootcamp-0,1,2 服務(wù)，分別對(duì)應(yīng) bootcamp-0,1,2 pod.

服務(wù)個(gè)數(shù)大于實(shí)例個(gè)數(shù)，是想測(cè)試下服務(wù)沒有對(duì)應(yīng)的實(shí)例時(shí)的表現(xiàn)。

網(wǎng)游中的匹配服務(wù)器將分配一個(gè)房間給客戶端，列舉 bootcamp-0,1,2 pod 所在節(jié)點(diǎn)的外網(wǎng) IP,
連同對(duì)應(yīng)服務(wù)的端口，發(fā)送給客戶端，讓客戶端直連。

[jinqing@host-10-240-79-10 statefulset]$ cat bootcamp.yml
apiVersion: v1
kind: Service
metadata:
  name: bootcamp
  namespace: jq
  labels:
    name: bootcamp
spec:
  ports:
    - port: 8080
  clusterIP: None  # StatefulSet要求Headless服務(wù)
  selector:
    app: bootcamp  # 選擇 bootcamp 應(yīng)用

---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: bootcamp
  namespace: jq
spec:
  serviceName: bootcamp  # 上面的 Headless 服務(wù)名
  replicas: 2
  template:
    metadata:
      labels:
        app: bootcamp  # 應(yīng)用名，與服務(wù)中的 selector 對(duì)應(yīng)
    spec:
      containers:
      - name: bootcamp
        image: docker.io/jocatalin/kubernetes-bootcamp:v1

---
kind: Service
apiVersion: v1
metadata:
  name: bootcamp-0
  namespace: jq
spec:
  type: NodePort  # 對(duì)外服務(wù)
  externalTrafficPolicy: Local  # 不要轉(zhuǎn)發(fā)到其他節(jié)點(diǎn)
  selector:
    app: bootcamp
    statefulset.kubernetes.io/pod-name: bootcamp-0  # 選擇 pod
  ports:
  - protocol: TCP
    nodePort: 30880  # 對(duì)外端口
    port: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: bootcamp-1
  namespace: jq
spec:
  type: NodePort
  externalTrafficPolicy: Local
  selector:
    app: bootcamp
    statefulset.kubernetes.io/pod-name: bootcamp-1
  ports:
  - protocol: TCP
    nodePort: 30881
    port: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: bootcamp-2
  namespace: jq
spec:
  type: NodePort
  externalTrafficPolicy: Local
  selector:
    app: bootcamp
    statefulset.kubernetes.io/pod-name: bootcamp-2
  ports:
  - protocol: TCP
    nodePort: 30882
    port: 8080

因?yàn)?statefulset 的每個(gè)實(shí)例有不同的標(biāo)簽，所以可以為服務(wù)選擇一個(gè)實(shí)例。

利用 externalTrafficPolicy: Local 設(shè)置來(lái)禁止轉(zhuǎn)發(fā)。
參考 service.spec.externalTrafficPolicy 的說(shuō)明：

https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport

Setting service.spec.externalTrafficPolicy to the value Local will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. If there are no local endpoints, packets sent to the node are dropped, ...

因?yàn)橛锌赡芏鄠€(gè)Pod開在同一節(jié)點(diǎn)上，所以對(duì)外端口設(shè)成了不同的 30880-30882。
如果限制每個(gè)節(jié)點(diǎn)只開一個(gè)實(shí)例，則對(duì)外端口可以設(shè)成同一個(gè)。

創(chuàng)建服務(wù)：

[jinqing@host-10-240-79-10 statefulset]$ kubectl apply -f bootcamp.yml
service "bootcamp" created
statefulset.apps "bootcamp" created
service "bootcamp-0" created
service "bootcamp-1" created
service "bootcamp-2" created

服務(wù)如下：

[jinqing@host-10-240-79-10 statefulset]$ kubectl get service -n jq
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
bootcamp     ClusterIP   None             <none>        8080/TCP         3m
bootcamp-0   NodePort    10.96.128.137    <none>        8080:30880/TCP   3m
bootcamp-1   NodePort    10.109.2.56      <none>        8080:30881/TCP   3m
bootcamp-2   NodePort    10.102.181.193   <none>        8080:30882/TCP   3m

2個(gè)實(shí)例：

[jinqing@host-10-240-79-10 statefulset]$ kubectl get pod -n jq -o wide
NAME         READY     STATUS    RESTARTS   AGE       IP            NODE
bootcamp-0   1/1       Running   0          4m        10.244.0.42   host-10-240-79-10
bootcamp-1   1/1       Running   0          4m        10.244.1.63   host-10-240-79-11

訪問(wèn)服務(wù)必須指定節(jié)點(diǎn)，不會(huì)自動(dòng)轉(zhuǎn)發(fā)：

[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30880
Hello Kubernetes bootcamp! | Running on: bootcamp-0 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30881
curl: (7) Failed connect to 10.240.79.10:30881; Connection timed out
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30880
curl: (7) Failed connect to 10.240.79.11:30880; Connection timed out
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1

沒有負(fù)載均衡，30881端口總是訪問(wèn) bootcamp-1：

[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1

也可以從外網(wǎng)訪問(wèn).

30882 端口無(wú)法連接：

[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30882
curl: (7) Failed connect to 10.240.79.11:30882; Connection refused
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30882
curl: (7) Failed connect to 10.240.79.10:30882; Connection refused

3個(gè)端口都有監(jiān)聽：

[root@host-10-240-79-11 tmp]# netstat -ntl | grep 3088
tcp6       0      0 :::30881                :::*                    LISTEN     
tcp6       0      0 :::30882                :::*                    LISTEN     
tcp6       0      0 :::30880                :::*                    LISTEN     

iptables-save 輸出如下, 其中 10.244是Pod的網(wǎng)段。

沒有實(shí)例運(yùn)行的節(jié)點(diǎn)上，會(huì)丟棄請(qǐng)求：

-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-XLB-LJXDQ4W47M42IZBH
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-XLB-U5NEOQT6R5WSBVOH

-A KUBE-XLB-LJXDQ4W47M42IZBH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-LJXDQ4W47M42IZBH
-A KUBE-XLB-LJXDQ4W47M42IZBH -m comment --comment "jq/bootcamp-1: has no local endpoints" -j KUBE-MARK-DROP
-A KUBE-XLB-U5NEOQT6R5WSBVOH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-U5NEOQT6R5WSBVOH
-A KUBE-XLB-U5NEOQT6R5WSBVOH -m comment --comment "jq/bootcamp-0: has no local endpoints" -j KUBE-MARK-DROP

有實(shí)例運(yùn)行的節(jié)點(diǎn)上會(huì)轉(zhuǎn)發(fā)給 Pod 8080：

-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-XLB-U5NEOQT6R5WSBVOH
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-XLB-LJXDQ4W47M42IZBH

-A KUBE-XLB-LJXDQ4W47M42IZBH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-LJXDQ4W47M42IZBH
-A KUBE-XLB-LJXDQ4W47M42IZBH -m comment --comment "Balancing rule 0 for jq/bootcamp-1:" -j KUBE-SEP-LJQA4WUIKJUQ5ALU
-A KUBE-XLB-U5NEOQT6R5WSBVOH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-U5NEOQT6R5WSBVOH
-A KUBE-XLB-U5NEOQT6R5WSBVOH -m comment --comment "jq/bootcamp-0: has no local endpoints" -j KUBE-MARK-DROP

-A KUBE-SEP-LJQA4WUIKJUQ5ALU -s 10.244.1.63/32 -m comment --comment "jq/bootcamp-1:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LJQA4WUIKJUQ5ALU -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp -j DNAT --to-destination 10.244.1.63:8080

30882 端口無(wú)法連接
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "jq/bootcamp-2: has no endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 30882 -j REJECT --reject-with icmp-port-unreachable

測(cè)試下擴(kuò)容：

[jinqing@host-10-240-79-10 statefulset]$ kubectl get statefulset -n jq
NAME       DESIRED   CURRENT   AGE
bootcamp   2         2         45m
[jinqing@host-10-240-79-10 statefulset]$ kubectl scale --replicas=3 statefulset/bootcamp -n jq
statefulset.apps "bootcamp" scaled
[jinqing@host-10-240-79-10 statefulset]$ kubectl get statefulset -n jq
NAME       DESIRED   CURRENT   AGE
bootcamp   3         3         47m
[jinqing@host-10-240-79-10 statefulset]$ kubectl get pod -n jq -o wide
NAME         READY     STATUS    RESTARTS   AGE       IP            NODE
bootcamp-0   1/1       Running   0          48m       10.244.0.42   host-10-240-79-10
bootcamp-1   1/1       Running   0          48m       10.244.1.63   host-10-240-79-11
bootcamp-2   1/1       Running   0          45s       10.244.2.60   host-10-240-79-12
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.12:30882
Hello Kubernetes bootcamp! | Running on: bootcamp-2 | v=1