# grpc外部負(fù)載均衡器測(cè)試

(金慶的專(zhuān)欄 2020.4)

grpc 對(duì)每個(gè)請(qǐng)求進(jìn)行負(fù)載均衡。負(fù)載均衡的方式有：

* 代理模式
* 客戶(hù)端實(shí)現(xiàn)
* 外部負(fù)載均衡

參考：gRPC LB https://blog.csdn.net/xiaojia1100/article/details/78842295

gRPC 中負(fù)載均衡的主要機(jī)制是外部負(fù)載均衡。

gRPC 定義了外部負(fù)載均衡服務(wù)的接口：https://github.com/grpc/grpc/tree/master/src/proto/grpc/lb/v1

* load_balancer.proto 客戶(hù)端向 lb 服查詢(xún)后端列表
* load_reporter.proto lb 服向后端服查詢(xún)負(fù)載

https://github.com/bsm/grpclb 實(shí)現(xiàn)了一個(gè) grpc 的外部負(fù)載均衡服。
因?yàn)槠鋵?shí)現(xiàn)早于負(fù)載均衡服的接口規(guī)范，所以接口定義與 grpc 規(guī)范不同。
見(jiàn) issue#26: https://github.com/bsm/grpclb/issues/26#issuecomment-613873655
grpclb 目前僅支持 consul 服務(wù)發(fā)現(xiàn)。

標(biāo)準(zhǔn)的 grpclb 實(shí)現(xiàn)目前好像只有 https://github.com/joa/jawlb。
jawlb 通過(guò) Kubernetes API 來(lái)發(fā)現(xiàn)服務(wù)。

以下測(cè)試 grpc 客戶(hù)端從 jawlb 服查詢(xún)服務(wù)器列表，然后請(qǐng)求服務(wù)。
首先在本機(jī)開(kāi)了多個(gè) greeter 服實(shí)例，端口不同。
然后更改 greeter 客戶(hù)端，不要直接連 greeter 服地址，而是配一個(gè) jawlb 服地址。
同時(shí)更改 jawlb, 刪除服務(wù)發(fā)現(xiàn)，改為固定輸出本機(jī)服務(wù)列表，定時(shí)切換。

greeter 是指 grpc-go 中的例子：grpc-go\examples\helloworld\greeter

## greeter 服更改

添加參數(shù)指定服務(wù)端口。

```
package main

import (
    "fmt"
    "log"
    "net"

    "github.com/spf13/pflag"
    "github.com/spf13/viper"
    "golang.org/x/net/context"
    "google.golang.org/grpc"
    pb "google.golang.org/grpc/examples/helloworld/helloworld"
)

// GreeterServer is used to implement helloworld.GreeterServer.
type GreeterServer struct {
}

// SayHello implements helloworld.GreeterServer
func (s *GreeterServer) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
    msg := fmt.Sprintf("Hello %s from server-%d", in.Name, viper.GetInt("port"))
    return &pb.HelloReply{Message: msg}, nil
}

func main() {
    pflag.Int("port", 8000, "server bind port")
    pflag.Parse()
    viper.BindPFlags(pflag.CommandLine)
    port := viper.GetInt("port")

    addr := fmt.Sprintf(":%d", port)
    lis, err := net.Listen("tcp", addr)
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    s := grpc.NewServer()
    pb.RegisterGreeterServer(s, &GreeterServer{})
    s.Serve(lis)
}
```

## greeter 客戶(hù)端更改
```
package main

import (
    "context"
    "log"
    "os"
    "time"

    "github.com/sirupsen/logrus"
    "google.golang.org/grpc"
    _ "google.golang.org/grpc/balancer/grpclb"
    pb "google.golang.org/grpc/examples/helloworld/helloworld"
    "google.golang.org/grpc/grpclog"
    "google.golang.org/grpc/resolver"
    "google.golang.org/grpc/resolver/manual"
)

const (
    defaultName = "world"
)

func init() {
    grpclog.SetLogger(logrus.New())
}

func main() {
    rb := manual.NewBuilderWithScheme("whatever")
    rb.InitialState(resolver.State{Addresses: []resolver.Address{
        {Addr: "127.0.0.1:8888", Type: resolver.GRPCLB},
    }})

    conn, err := grpc.Dial("whatever:///this-gets-overwritten", grpc.WithInsecure(), grpc.WithBlock(),
        grpc.WithResolvers(rb))
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()
    c := pb.NewGreeterClient(conn)

    name := defaultName
    if len(os.Args) > 1 {
        name = os.Args[1]
    }

    for {
        ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
        r, err := c.SayHello(ctx, &pb.HelloRequest{Name: name})
        cancel()

        if err != nil {
            log.Fatalf("could not greet: %v", err)
            time.Sleep(time.Second)
            continue
        }

        log.Printf("Greeting: %s", r.GetMessage())
        time.Sleep(time.Second)
    }
}
```

有以下更改：
* import _ "google.golang.org/grpc/balancer/grpclb"
* grpc.Dial("whatever:///this-gets-overwritten", grpc.WithResolvers(rb))
    + 采用一個(gè)自定義解析器，用來(lái)獲取 jawlb 地址
    + Scheme("whatever") 可以任意，用作解析器名字
    + 目標(biāo) this-gets-overwritten 可以任意，因?yàn)?jawlb 忽略了該名字
    + 127.0.0.1:8888 是 jawlb 地址
* 改為每秒請(qǐng)求一次

正常的 grpclb 是在 DNS 中設(shè)置 SRV 記錄，
此處測(cè)試避免設(shè)置 DNS, 采用了一個(gè)自定義解析器，
代碼上多了幾行。
用 DNS 設(shè)置的好處是, 可以直接解析為后端 IP, 也可以添加 grpclb, 代碼上如同直接連接后端：
```
    conn, err := grpc.Dial("dns:///myservice.domain.com", grpc.WithInsecure())
```

## jawlb 更改

### main.go

刪除所有配置，改為固定本機(jī) 8888 端口監(jiān)聽(tīng)。

* 刪除 `envconfig.MustProcess("JAWLB", &cfg)`
* listen() 改為
    ```
    func listen() (conn net.Listener, err error) {
        conn, err = net.Listen("tcp", ":8888")
        return
    }
    ```
   
### watch.go

```
package main

import (
    "context"
    "fmt"
    "net"
    "time"
)

func watchService(ctx context.Context) (_ <-chan ServerList, err error) {
    ch := make(chan ServerList)

    go func() {
        ticker := time.NewTicker(10 * time.Second)
        i := 0
        for {
            select {
            case <-ctx.Done():
                ticker.Stop()
                close(ch)
                return
            case <-ticker.C:
                i += 1
                fmt.Printf("i = %d\n", i)
                ports := []int32{8010, 8020}
                var servers []Server
                for _, port := range ports {
                    servers = append(servers, Server{IP: net.ParseIP("127.0.0.1"), Port: port + int32(i%2)})
                }
                ch <- servers
            } // select
        } // for
    }()

    return ch, nil
}
```

刪除所有服務(wù)發(fā)現(xiàn)代碼，改為每10秒切換端口：8010,8020 <-> 8011,8021

## 運(yùn)行

### jawlb
```
λ jawlb.exe
2020/04/16 15:35:17 waiting for TERM
i = 1
2020/04/16 15:35:27 endpoints:
2020/04/16 15:35:27     127.0.0.1:8011
2020/04/16 15:35:27     127.0.0.1:8021
i = 2
2020/04/16 15:35:37 endpoints:
2020/04/16 15:35:37     127.0.0.1:8010
2020/04/16 15:35:37     127.0.0.1:8020
```

### server

運(yùn)行 4 個(gè)實(shí)例：
```
server --port 8010
server --port 8020
server --port 8011
server --port 8021
```

### client

```
λ client
INFO[0002] lbBalancer: handle SubConn state change: 0xc00008a590, CONNECTING
INFO[0002] Channel Connectivity change to CONNECTING
INFO[0002] lbBalancer: handle SubConn state change: 0xc00008a5f0, CONNECTING
INFO[0002] Subchannel picks a new address "127.0.0.1:8021" to connect
INFO[0002] Subchannel Connectivity change to READY
INFO[0002] lbBalancer: handle SubConn state change: 0xc00008a590, READY
INFO[0002] Channel Connectivity change to READY
INFO[0002] Subchannel Connectivity change to READY
INFO[0002] lbBalancer: handle SubConn state change: 0xc00008a5f0, READY
2020/04/16 15:37:47 Greeting: Hello world from server-8021
2020/04/16 15:37:48 Greeting: Hello world from server-8011
2020/04/16 15:37:49 Greeting: Hello world from server-8021
2020/04/16 15:37:50 Greeting: Hello world from server-8011
2020/04/16 15:37:51 Greeting: Hello world from server-8021
2020/04/16 15:37:52 Greeting: Hello world from server-8011
2020/04/16 15:37:53 Greeting: Hello world from server-8021
2020/04/16 15:37:54 Greeting: Hello world from server-8011
2020/04/16 15:37:55 Greeting: Hello world from server-8021
2020/04/16 15:37:56 Greeting: Hello world from server-8011
INFO[0012] lbBalancer: processing server list: servers:<ip_address:"\000\000\000\000\000\000\000\000\000\000\377\377\177\000\000\001" port:8020 > servers:<ip_address:"\000\000\000\000\000\000\000\000\000\000\377\377\177\000\000\001" port:8010 >
INFO[0012] lbBalancer: server list entry[0]: ipStr:|127.0.0.1|, port:|8020|, load balancer token:||
INFO[0012] lbBalancer: server list entry[1]: ipStr:|127.0.0.1|, port:|8010|, load balancer token:||
2020/04/16 15:37:57 Greeting: Hello world from server-8020
2020/04/16 15:37:58 Greeting: Hello world from server-8010
2020/04/16 15:37:59 Greeting: Hello world from server-8020
2020/04/16 15:38:00 Greeting: Hello world from server-8010
2020/04/16 15:38:01 Greeting: Hello world from server-8020
2020/04/16 15:38:02 Greeting: Hello world from server-8010
2020/04/16 15:38:03 Greeting: Hello world from server-8020
2020/04/16 15:38:04 Greeting: Hello world from server-8010
2020/04/16 15:38:05 Greeting: Hello world from server-8020
2020/04/16 15:38:06 Greeting: Hello world from server-8010
INFO[0022] lbBalancer: processing server list: servers:<ip_address:"\000\000\000\000\000\000\000\000\000\000\377\377\177\000\000\001" port:8021 > servers:<ip_address:"\000\000\000\000\000\000\000\000\000\000\377\377\177\000\000\001" port:8011 >
INFO[0022] lbBalancer: server list entry[0]: ipStr:|127.0.0.1|, port:|8021|, load balancer token:||
INFO[0022] lbBalancer: server list entry[1]: ipStr:|127.0.0.1|, port:|8011|, load balancer token:||
2020/04/16 15:38:07 Greeting: Hello world from server-8011
2020/04/16 15:38:08 Greeting: Hello world from server-8021
2020/04/16 15:38:09 Greeting: Hello world from server-8011
```

## 結(jié)論

客戶(hù)端應(yīng)用一個(gè)自定義 resolver 解析 "whatever:///this-gets-overwritten"，
獲取到 `{Addr: "127.0.0.1:8888", Type: resolver.GRPCLB}`,
知道這是一個(gè) grpclb，于是按 load_balancer.proto 的定義查詢(xún) jawlb 來(lái)獲取后端地址列表。

jawlb 每 10s 更新一次服務(wù)器列表，每次輸出多個(gè)地址。客戶(hù)端在多個(gè)地址間輪換請(qǐng)求。

## 其他測(cè)試

* 不開(kāi) jawlb，客戶(hù)端將無(wú)法成功請(qǐng)求，直到 jawlb 開(kāi)啟才成功
* 中途關(guān)閉 jawlb, 請(qǐng)求仍會(huì)成功，但是保持為最后的服務(wù)器列表
    + 同時(shí)會(huì)不斷嘗試重連 jawlb, 但是重連成功后沒(méi)有切換服務(wù)，應(yīng)該是個(gè)錯(cuò)誤
* Dial() 不加 grpc.WithBlock() 參數(shù), 報(bào)錯(cuò)：all SubConns are in TransientFailure

```
λ client
INFO[0000] parsed scheme: "whatever"
INFO[0000] ccResolverWrapper: sending update to cc: {[{127.0.0.1:8888  <nil> 1 <nil>}] <nil> <nil>}
INFO[0000] ClientConn switching balancer to "grpclb"
INFO[0000] Channel switches to new LB policy "grpclb"
INFO[0000] lbBalancer: UpdateClientConnState: {ResolverState:{Addresses:[{Addr:127.0.0.1:8888 ServerName: Attributes:<nil> Type:1 Metadata:<nil>}] ServiceConfig:<nil> Attributes:<nil>} BalancerConfig:<nil>}
INFO[0000] parsed scheme: "grpclb-internal"
INFO[0000] ccResolverWrapper: sending update to cc: {[{127.0.0.1:8888  <nil> 0 <nil>}] <nil> <nil>}
INFO[0000] ClientConn switching balancer to "pick_first"
INFO[0000] Channel switches to new LB policy "pick_first"
INFO[0000] Subchannel Connectivity change to CONNECTING
INFO[0000] blockingPicker: the picked transport is not ready, loop back to repick
INFO[0000] pickfirstBalancer: HandleSubConnStateChange: 0xc00003fb10, {CONNECTING <nil>}
INFO[0000] Channel Connectivity change to CONNECTING
INFO[0000] Subchannel picks a new address "127.0.0.1:8888" to connect
INFO[0000] CPU time info is unavailable on non-linux or appengine environment.
INFO[0000] Subchannel Connectivity change to READY
INFO[0000] pickfirstBalancer: HandleSubConnStateChange: 0xc00003fb10, {READY <nil>}
INFO[0000] Channel Connectivity change to READY
INFO[0000] lbBalancer: processing server list: servers:<ip_address:"\000\000\000\000\000\000\000\000\000\000\377\377\177\000\000\001" port:8010 > servers:<ip_address:"\000\000\000\000\000\000\000\000\000\000\377\377\177\000\000\001" port:8020 >
INFO[0000] lbBalancer: server list entry[0]: ipStr:|127.0.0.1|, port:|8010|, load balancer token:||
INFO[0000] lbBalancer: server list entry[1]: ipStr:|127.0.0.1|, port:|8020|, load balancer token:||
INFO[0000] Subchannel Connectivity change to CONNECTING
INFO[0000] Subchannel Connectivity change to CONNECTING
INFO[0000] Channel Connectivity change to TRANSIENT_FAILURE
INFO[0000] lbBalancer: handle SubConn state change: 0xc00008a220, CONNECTING
INFO[0000] Channel Connectivity change to CONNECTING
INFO[0000] lbBalancer: handle SubConn state change: 0xc00008a280, CONNECTING
2020/04/16 16:40:06 could not greet: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
```