MENU

Kubernetes进阶:欠费了撒

2019 年 08 月 21 日 • 应用服务器

主题功能试水,可能会触发未知BUG撒,就在刚刚博客炸了一次。

我并没有收到七牛云欠费通知的短信或邮件,而且我也没设置可用额度预警,欠费导致CDN服务被冻结,冻结后域名无法访问,触发告警,看欠费保护机制我的欠费时间已经超过24小时了。

记得每次费用结算都是每月月初。

记得前几天收到邮件通知七牛新版欠费流程上线,但是我没去看,还是在愚蠢的以为费用还是每月月初结算。

所以2019-08-21 13:57 CDN服务被冻结,充值后2019-08-21 14:11恢复正常,cdn冻结期间本站所有图片无法访问,庆幸cdn.rj-bai.com没有被同时冻结,如果被冻结本站核心css/js/woff将无法加载。

顺便把kafka/zookeeper/redis集群的YAML文件贴一下吧,适用于kubernetes平台,需要NFS动态供给就行了,先redis集群吧。

redis集群

主要是参考了这篇文章进行的,先创建configmap吧,YAML文件如下,这里使用了代码折叠,PC端代码展开无问题,某些手机浏览器展开后代码显示不完整。

redis-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-conf
data:
  redis.conf: |
    appendonly yes
    cluster-enabled yes
    cluster-config-file /var/lib/redis/nodes.conf
    cluster-node-timeout 5000
    protected-mode no                      
    port 6379
    timeout 10                             
    tcp-keepalive 300
    supervised no                          
    pidfile /var/lib/redis/redis_6379.pid
    loglevel notice
    logfile "/var/lib/redis/redis6379.log"
    databases 16                           
    always-show-logo yes                   
    save 900 1                             
    save 300 10                            
    save 60 10000                          
    stop-writes-on-bgsave-error yes        
    rdbcompression yes                     
    rdbchecksum yes                        
    dbfilename 6379.rdb
    dir /var/lib/redis
    slave-serve-stale-data yes
    slave-read-only yes                    
    repl-diskless-sync no                  
    repl-diskless-sync-delay 5             
    repl-disable-tcp-nodelay no            
    slave-priority 100                     
                                           
    maxclients 10000
    maxmemory 5g                          
    
    lazyfree-lazy-eviction no              
    lazyfree-lazy-expire no                
    lazyfree-lazy-server-del no            
    slave-lazy-flush no             
    
    appendonly yes
    appendfilename "appendonly.aof"
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    aof-load-truncated yes
    aof-use-rdb-preamble no
    
    lua-time-limit 5000
    cluster-enabled yes
    cluster-config-file /var/lib/redis/nodes-6379.conf
    cluster-node-timeout 5000
    
    slowlog-log-slower-than 10000
    slowlog-max-len 128
    latency-monitor-threshold 0
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size -2
    list-compress-depth 0
    
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    hll-sparse-max-bytes 3000
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10                             
    aof-rewrite-incremental-fsync yes

这就是redis的配置文件,酌情自行更改,下面是StatefulSet&Service

redis-statefulsets.yaml

apiVersion: v1
kind: Service
metadata:
  name: redis-headless
  labels:
    app: redis
spec:
  ports:
  - name: redis-headless
    port: 6379
  selector:
    app: redis
  clusterIP: None
---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis
  labels:
    app: redis
spec:
  serviceName: "redis-headless"
  replicas: 6
  template:
    metadata:
      labels:
        app: redis
    spec:
      terminationGracePeriodSeconds: 20
      containers:
      - name: redis
        image: "redis"
        command:
        - "redis-server"
        args:
        - "/etc/redis/redis.conf"
        - "--protected-mode"
        - "no"
        ports:
        - name: redis
          containerPort: 6379
          protocol: "TCP"
        resources:
          limits:
            cpu: 0.2
            memory: 256M
          requests:
            cpu: 0.2
            memory: 64M
        readinessProbe:
          tcpSocket:
            port: 6379
          initialDelaySeconds: 3
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
        livenessProbe:
          tcpSocket:
            port: 6379
          initialDelaySeconds: 5
          timeoutSeconds: 5
          successThreshold: 1
          periodSeconds: 10
        volumeMounts:
        - name: "redis-conf"
          mountPath: "/etc/redis"
        - name: "redis-data"
          mountPath: "/var/lib/redis"
        - name: "time"
          mountPath: "/etc/localtime"
          readOnly: true
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-conf"
      - name: "time"
        hostPath:
          path: "/etc/localtime"
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 512M

还是按自己情况自行更改吧,我这里直接创建了,我用kubeadm的那个集群去跑了,最近又加了两个节点进来,所以正常启动后是这样,

现在只是pod正常启动了,下面创建集群吧,

[root@kubeadm ~]# kubectl exec -it redis-0 bash
root@redis-0:/data# redis-cli --cluster create 10.244.2.2:6379 10.244.3.4:6379 10.244.1.2:6379 10.244.3.5:6379 10.244.2.3:6379 10.244.1.3:6379  --cluster-replicas 1
root@redis-0:/data# redis-cli 
127.0.0.1:6379> cluster info
127.0.0.1:6379> cluster nodes

莫得问题,至于为什么没使用稳定的网络标志去创建集群,原文也解释过了,我这里就不累赘了,过,下面开始zookeeper集群。

zookeeper集群

这个主要是参照官方文档来进行的,集群正常启动后发现zookeeper版本貌似有点太低了,还是17年的版本,所以我这里参照官方文档的镜像层信息重构了一个最新稳定版的zookeeper镜像,zookeeper版本为3.5.5,所以Dockerfile如下。

Dockerfile

FROM ubuntu:xenial
ENV ZK_USER=zookeeper ZK_DATA_DIR=/var/lib/zookeeper/data ZK_DATA_LOG_DIR=/var/lib/zookeeper/log ZK_LOG_DIR=/var/log/zookeeper JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 ZK_DIR=zookeeper-3.5.5 ZK_DIST=apache-zookeeper-3.5.5-bin
RUN /bin/sh -c set -x && apt-get update && apt-get install -y openjdk-8-jre-headless wget tzdata netcat-openbsd && wget -q "http://www.apache.org/dist/zookeeper/$ZK_DIR/$ZK_DIST.tar.gz" && tar -xzf "$ZK_DIST.tar.gz" -C /opt && rm -r "$ZK_DIST.tar.gz" && ln -s /opt/$ZK_DIST /opt/zookeeper && rm -rf /opt/zookeeper/README.md /opt/zookeeper/NOTICE.txt /opt/zookeeper/README_packaging.txt  /opt/zookeeper/docs && apt-get autoremove -y wget && rm -rf /var/lib/apt/lists/* && useradd $ZK_USER && [ `id -u $ZK_USER` -eq 1000 ] && [ `id -g $ZK_USER` -eq 1000 ] && mkdir -p $ZK_DATA_DIR $ZK_DATA_LOG_DIR $ZK_LOG_DIR /usr/share/zookeeper /tmp/zookeeper /usr/etc/ 
ADD start-zookeeper zookeeper-ready /opt/zookeeper/bin/
RUN chown -R "$ZK_USER:$ZK_USER" /opt/$ZK_DIST $ZK_DATA_DIR $ZK_LOG_DIR $ZK_DATA_LOG_DIR /tmp/zookeeper && ln -s /opt/zookeeper/conf/ /usr/etc/zookeeper && ln -s /opt/zookeeper/bin/* /usr/bin && ln -s /opt/zookeeper/$ZK_DIST.jar /usr/share/zookeeper/ && ln -s /opt/zookeeper/lib/* /usr/share/zookeeper && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo Asia/Shanghai > /etc/timezone

添加了两个文件到镜像里,内容如下,先是启动脚本。

start-zookeeper

#!/usr/bin/env bash
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#
#Usage: start-zookeeper [OPTIONS]
# Starts a ZooKeeper server based on the supplied options.
#     --servers           The number of servers in the ensemble. The default 
#                         value is 1.

#     --data_dir          The directory where the ZooKeeper process will store its
#                         snapshots. The default is /var/lib/zookeeper/data.

#     --data_log_dir      The directory where the ZooKeeper process will store its 
#                         write ahead log. The default is 
#                         /var/lib/zookeeper/data/log.

#     --conf_dir          The directoyr where the ZooKeeper process will store its
#                         configuration. The default is /opt/zookeeper/conf.

#     --client_port       The port on which the ZooKeeper process will listen for 
#                         client requests. The default is 2181.

#     --election_port     The port on which the ZooKeeper process will perform 
#                         leader election. The default is 3888.

#     --server_port       The port on which the ZooKeeper process will listen for 
#                         requests from other servers in the ensemble. The 
#                         default is 2888. 

#     --tick_time         The length of a ZooKeeper tick in ms. The default is 
#                         2000.

#     --init_limit        The number of Ticks that an ensemble member is allowed 
#                         to perform leader election. The default is 10.

#     --sync_limit        The maximum session timeout that the ensemble will 
#                         allows a client to request. The default is 5.

#     --heap              The maximum amount of heap to use. The format is the 
#                         same as that used for the Xmx and Xms parameters to the 
#                         JVM. e.g. --heap=2G. The default is 2G.

#     --max_client_cnxns  The maximum number of client connections that the 
#                         ZooKeeper process will accept simultaneously. The 
#                         default is 60.

#     --snap_retain_count The maximum number of snapshots the ZooKeeper process 
#                         will retain if purge_interval is greater than 0. The 
#                         default is 3.

#     --purge_interval    The number of hours the ZooKeeper process will wait 
#                         between purging its old snapshots. If set to 0 old 
#                         snapshots will never be purged. The default is 0.

#     --max_session_timeout The maximum time in milliseconds for a client session 
#                         timeout. The default value is 2 * tick time.

#     --min_session_timeout The minimum time in milliseconds for a client session 
#                         timeout. The default value is 20 * tick time.

#     --log_level         The log level for the zookeeeper server. Either FATAL,
#                         ERROR, WARN, INFO, DEBUG. The default is INFO.


USER=`whoami`
HOST=`hostname -s`
DOMAIN=`hostname -d`
LOG_LEVEL=INFO
DATA_DIR="/var/lib/zookeeper/data"
DATA_LOG_DIR="/var/lib/zookeeper/log"
LOG_DIR="/var/log/zookeeper"
CONF_DIR="/opt/zookeeper/conf"
CLIENT_PORT=2181
SERVER_PORT=2888
ELECTION_PORT=3888
TICK_TIME=2000
INIT_LIMIT=10
SYNC_LIMIT=5
HEAP=2G
MAX_CLIENT_CNXNS=60
SNAP_RETAIN_COUNT=3
PURGE_INTERVAL=0
SERVERS=1

function print_usage() {
echo "\
Usage: start-zookeeper [OPTIONS]
Starts a ZooKeeper server based on the supplied options.
    --servers           The number of servers in the ensemble. The default 
                        value is 1.

    --data_dir          The directory where the ZooKeeper process will store its
                        snapshots. The default is /var/lib/zookeeper/data.

    --data_log_dir      The directory where the ZooKeeper process will store its 
                        write ahead log. The default is 
                        /var/lib/zookeeper/data/log.

    --conf_dir          The directoyr where the ZooKeeper process will store its
                        configuration. The default is /opt/zookeeper/conf.

    --client_port       The port on which the ZooKeeper process will listen for 
                        client requests. The default is 2181.

    --election_port     The port on which the ZooKeeper process will perform 
                        leader election. The default is 3888.

    --server_port       The port on which the ZooKeeper process will listen for 
                        requests from other servers in the ensemble. The 
                        default is 2888. 

    --tick_time         The length of a ZooKeeper tick in ms. The default is 
                        2000.

    --init_limit        The number of Ticks that an ensemble member is allowed 
                        to perform leader election. The default is 10.

    --sync_limit        The maximum session timeout that the ensemble will 
                        allows a client to request. The default is 5.

    --heap              The maximum amount of heap to use. The format is the 
                        same as that used for the Xmx and Xms parameters to the 
                        JVM. e.g. --heap=2G. The default is 2G.

    --max_client_cnxns  The maximum number of client connections that the 
                        ZooKeeper process will accept simultaneously. The 
                        default is 60.

    --snap_retain_count The maximum number of snapshots the ZooKeeper process 
                        will retain if purge_interval is greater than 0. The 
                        default is 3.

    --purge_interval    The number of hours the ZooKeeper process will wait 
                        between purging its old snapshots. If set to 0 old 
                        snapshots will never be purged. The default is 0.

    --max_session_timeout The maximum time in milliseconds for a client session 
                        timeout. The default value is 2 * tick time.

    --min_session_timeout The minimum time in milliseconds for a client session 
                        timeout. The default value is 20 * tick time.

    --log_level         The log level for the zookeeeper server. Either FATAL,
                        ERROR, WARN, INFO, DEBUG. The default is INFO.
"
}

function create_data_dirs() {
    if [ ! -d $DATA_DIR  ]; then
        mkdir -p $DATA_DIR
        chown -R $USER:$USER $DATA_DIR
    fi

    if [ ! -d $DATA_LOG_DIR  ]; then
        mkdir -p $DATA_LOG_DIR
        chown -R $USER:USER $DATA_LOG_DIR
    fi

    if [ ! -d $LOG_DIR  ]; then
        mkdir -p $LOG_DIR
        chown -R $USER:$USER $LOG_DIR
    fi
    if [ ! -f $ID_FILE ] && [ $SERVERS -gt 1 ]; then
        echo $MY_ID >> $ID_FILE
    fi
}

function print_servers() {
    for (( i=1; i<=$SERVERS; i++ ))
    do
        echo "server.$i=$NAME-$((i-1)).$DOMAIN:$SERVER_PORT:$ELECTION_PORT"
    done
}

function create_config() {
    rm -f $CONFIG_FILE
    echo "#This file was autogenerated DO NOT EDIT" >> $CONFIG_FILE
    echo "clientPort=$CLIENT_PORT" >> $CONFIG_FILE
    echo "dataDir=$DATA_DIR" >> $CONFIG_FILE
    echo "dataLogDir=$DATA_LOG_DIR" >> $CONFIG_FILE
    echo "tickTime=$TICK_TIME" >> $CONFIG_FILE
    echo "initLimit=$INIT_LIMIT" >> $CONFIG_FILE
    echo "syncLimit=$SYNC_LIMIT" >> $CONFIG_FILE
    echo "maxClientCnxns=$MAX_CLIENT_CNXNS" >> $CONFIG_FILE
    echo "minSessionTimeout=$MIN_SESSION_TIMEOUT" >> $CONFIG_FILE
    echo "maxSessionTimeout=$MAX_SESSION_TIMEOUT" >> $CONFIG_FILE
    echo "autopurge.snapRetainCount=$SNAP_RETAIN_COUNT" >> $CONFIG_FILE
    echo "autopurge.purgeInteval=$PURGE_INTERVAL" >> $CONFIG_FILE
    echo "4lw.commands.whitelist=*" >> $CONFIG_FILE
     if [ $SERVERS -gt 1 ]; then
        print_servers >> $CONFIG_FILE
    fi
    cat $CONFIG_FILE >&2
}

function create_jvm_props() {
    rm -f $JAVA_ENV_FILE
    echo "ZOO_LOG_DIR=$LOG_DIR" >> $JAVA_ENV_FILE
    echo "JVMFLAGS=\"-Xmx$HEAP -Xms$HEAP\"" >> $JAVA_ENV_FILE
}

function create_log_props() {
    rm -f $LOGGER_PROPS_FILE
    echo "Creating ZooKeeper log4j configuration"
    echo "zookeeper.root.logger=CONSOLE" >> $LOGGER_PROPS_FILE
    echo "zookeeper.console.threshold="$LOG_LEVEL >> $LOGGER_PROPS_FILE
    echo "log4j.rootLogger=\${zookeeper.root.logger}" >> $LOGGER_PROPS_FILE
    echo "log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender" >> $LOGGER_PROPS_FILE
    echo "log4j.appender.CONSOLE.Threshold=\${zookeeper.console.threshold}" >> $LOGGER_PROPS_FILE
    echo "log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout" >> $LOGGER_PROPS_FILE
    echo "log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n" >> $LOGGER_PROPS_FILE
}

optspec=":hv-:"
while getopts "$optspec" optchar; do

    case "${optchar}" in
        -)
            case "${OPTARG}" in
                servers=*)
                    SERVERS=${OPTARG##*=}
                    ;;
                data_dir=*)
                    DATA_DIR=${OPTARG##*=}
                    ;;
                data_log_dir=*)
                    DATA_LOG_DIR=${OPTARG##*=}
                    ;;
                log_dir=*)
                    LOG_DIR=${OPTARG##*=}
                    ;;
                conf_dir=*)
                    CONF_DIR=${OPTARG##*=}
                    ;;
                client_port=*)
                    CLIENT_PORT=${OPTARG##*=}
                    ;;
                election_port=*)
                    ELECTION_PORT=${OPTARG##*=}
                    ;;
                server_port=*)
                    SERVER_PORT=${OPTARG##*=}
                    ;;
                tick_time=*)
                    TICK_TIME=${OPTARG##*=}
                    ;;
                init_limit=*)
                    INIT_LIMIT=${OPTARG##*=}
                    ;;
                sync_limit=*)
                    SYNC_LIMIT=${OPTARG##*=}
                    ;;
                heap=*)
                    HEAP=${OPTARG##*=}
                    ;;
                max_client_cnxns=*)
                    MAX_CLIENT_CNXNS=${OPTARG##*=}
                    ;;
                snap_retain_count=*)
                    SNAP_RETAIN_COUNT=${OPTARG##*=}
                    ;;
                purge_interval=*)
                    PURGE_INTERVAL=${OPTARG##*=}
                    ;;
                max_session_timeout=*)
                    MAX_SESSION_TIMEOUT=${OPTARG##*=}
                    ;;
                min_session_timeout=*)
                    MIN_SESSION_TIMEOUT=${OPTARG##*=}
                    ;;
                log_level=*)
                    LOG_LEVEL=${OPTARG##*=}
                    ;;
                *)
                    echo "Unknown option --${OPTARG}" >&2
                    exit 1
                    ;;
            esac;;
        h)
            print_usage
            exit
            ;;
        v)
            echo "Parsing option: '-${optchar}'" >&2
            ;;
        *)
            if [ "$OPTERR" != 1 ] || [ "${optspec:0:1}" = ":" ]; then
                echo "Non-option argument: '-${OPTARG}'" >&2
            fi
            ;;
    esac
done

MIN_SESSION_TIMEOUT=${MIN_SESSION_TIMEOUT:- $((TICK_TIME*2))}
MAX_SESSION_TIMEOUT=${MAX_SESSION_TIMEOUT:- $((TICK_TIME*20))}
ID_FILE="$DATA_DIR/myid"
CONFIG_FILE="$CONF_DIR/zoo.cfg"
LOGGER_PROPS_FILE="$CONF_DIR/log4j.properties"
JAVA_ENV_FILE="$CONF_DIR/java.env"
if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
    NAME=${BASH_REMATCH[1]}
    ORD=${BASH_REMATCH[2]}
else
    echo "Fialed to parse name and ordinal of Pod"
    exit 1
fi

MY_ID=$((ORD+1))

create_config && create_jvm_props && create_log_props && create_data_dirs && exec zkServer.sh start-foreground

然后是健康检查脚本,

zookeeper-ready

#!/usr/bin/env bash
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# zkOk.sh uses the ruok ZooKeeper four letter work to determine if the instance
# is health. The $? variable will be set to 0 if server responds that it is 
# healthy, or 1 if the server fails to respond.

OK=$(echo ruok | nc 127.0.0.1 $1)
if [ "$OK" == "imok" ]; then
    exit 0
else
    exit 1
fi

就这样,看这个四字指令让我想到了雷总,ruok?,就这样,自行构建把,在国内构建可能比较慢,我还是在轻量云服务上构建的,我这里构建完了,直接传到了dockerhub上,所以我的YAML文件如下,和官方的一样,就是镜像地址不一样,也改了一下启动内存,酌情更改吧,创建了撒。

zookeeper-statefulset.yaml

apiVersion: v1
kind: Service
metadata:
  name: zk-hs
  labels:
    app: zk
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk
---
apiVersion: v1
kind: Service
metadata:
  name: zk-cs
  labels:
    app: zk
spec:
  ports:
  - port: 2181
    name: client
  selector:
    app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-pdb
spec:
  selector:
    matchLabels:
      app: zk
  maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-hs
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: zk
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: kubernetes-zookeeper
        imagePullPolicy: Always
        image: "bairuijie/k8s-zk:3.5.5"
        resources:
          requests:
            memory: "512M"
            cpu: "1"
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        command:
        - sh
        - -c
        - "start-zookeeper \
          --servers=3 \
          --data_dir=/var/lib/zookeeper/data \
          --data_log_dir=/var/lib/zookeeper/data/log \
          --conf_dir=/opt/zookeeper/conf \
          --client_port=2181 \
          --election_port=3888 \
          --server_port=2888 \
          --tick_time=2000 \
          --init_limit=10 \
          --sync_limit=5 \
          --heap=512M \
          --max_client_cnxns=60 \
          --snap_retain_count=3 \
          --purge_interval=12 \
          --max_session_timeout=40000 \
          --min_session_timeout=4000 \
          --log_level=INFO "
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - "zookeeper-ready 2181"
          initialDelaySeconds: 10
          timeoutSeconds: 5
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/zookeeper
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

pod正常启动后就可以去看zookeeper状态了,两个follower一个leader,看一哈。

没啥子问题,最后kafka集群。

kafka集群

这个镜像我就是简单的改了一下时间,别的什么都没改撒,版本是2.3.0的,也放在了dockerhub上,kafka是需要zookeeper的,直接用刚刚的那个zookeeper吧,所以YAML文件如下,

kafka-statefulset.yaml

apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  labels:
    app: kafka
spec:
  ports:
  - port: 9092
    name: server
  clusterIP: None
  selector:
    app: kafka
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
spec:
  selector:
    matchLabels:
      app: kafka
  minAvailable: 2
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  template:
    metadata:
      labels:
        app: kafka
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values: 
                    - kafka
              topologyKey: "kubernetes.io/hostname"
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
             - weight: 1
               podAffinityTerm:
                 labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values: 
                        - zk
                 topologyKey: "kubernetes.io/hostname"
      terminationGracePeriodSeconds: 300
      imagePullSecrets:
      - name: registry-secret
      containers:
      - name: k8skafka
        image: bairuijie/k8s-kafka:2.3.0 
        resources:
          requests:
            memory: "512M"
            cpu: 500m
        ports:
        - containerPort: 9092
          name: server
        command:
        - sh
        - -c
        - "exec kafka-server-start.sh /opt/kafka/config/server.properties --override broker.id=${HOSTNAME##*-} \
          --override zookeeper.connect=zk-0.zk-hs.default:2181,zk-1.zk-hs.default:2181,zk-2.zk-hs.default:2181 \
          --override log.dir=/var/log \
          --override auto.create.topics.enable=true \
          --override auto.leader.rebalance.enable=true \
          --override background.threads=10 \
          --override compression.type=producer \
          --override delete.topic.enable=false \
          --override leader.imbalance.check.interval.seconds=300 \
          --override leader.imbalance.per.broker.percentage=10 \
          --override log.flush.interval.messages=9223372036854775807 \
          --override log.flush.offset.checkpoint.interval.ms=60000 \
          --override log.flush.scheduler.interval.ms=9223372036854775807 \
          --override log.retention.bytes=-1 \
          --override log.retention.hours=168 \
          --override log.roll.hours=168 \
          --override log.roll.jitter.hours=0 \
          --override log.segment.bytes=1073741824 \
          --override log.segment.delete.delay.ms=60000 \
          --override message.max.bytes=1000012 \
          --override min.insync.replicas=1 \
          --override num.io.threads=8 \
          --override num.network.threads=3 \
          --override num.recovery.threads.per.data.dir=1 \
          --override num.replica.fetchers=1 \
          --override offset.metadata.max.bytes=4096 \
          --override offsets.commit.required.acks=-1 \
          --override offsets.commit.timeout.ms=5000 \
          --override offsets.load.buffer.size=5242880 \
          --override offsets.retention.check.interval.ms=600000 \
          --override offsets.retention.minutes=1440 \
          --override offsets.topic.compression.codec=0 \
          --override offsets.topic.num.partitions=50 \
          --override offsets.topic.replication.factor=3 \
          --override offsets.topic.segment.bytes=104857600 \
          --override queued.max.requests=500 \
          --override quota.consumer.default=9223372036854775807 \
          --override quota.producer.default=9223372036854775807 \
          --override replica.fetch.min.bytes=1 \
          --override replica.fetch.wait.max.ms=500 \
          --override replica.high.watermark.checkpoint.interval.ms=5000 \
          --override replica.lag.time.max.ms=10000 \
          --override replica.socket.receive.buffer.bytes=65536 \
          --override replica.socket.timeout.ms=30000 \
          --override request.timeout.ms=30000 \
          --override socket.receive.buffer.bytes=102400 \
          --override socket.request.max.bytes=104857600 \
          --override socket.send.buffer.bytes=102400 \
          --override unclean.leader.election.enable=true \
          --override zookeeper.session.timeout.ms=6000 \
          --override zookeeper.set.acl=false \
          --override broker.id.generation.enable=true \
          --override connections.max.idle.ms=600000 \
          --override controlled.shutdown.enable=true \
          --override controlled.shutdown.max.retries=3 \
          --override controlled.shutdown.retry.backoff.ms=5000 \
          --override controller.socket.timeout.ms=30000 \
          --override default.replication.factor=1 \
          --override fetch.purgatory.purge.interval.requests=1000 \
          --override group.max.session.timeout.ms=300000 \
          --override group.min.session.timeout.ms=6000 \
          --override inter.broker.protocol.version=0.11.0-IV0 \
          --override log.cleaner.backoff.ms=15000 \
          --override log.cleaner.dedupe.buffer.size=134217728 \
          --override log.cleaner.delete.retention.ms=86400000 \
          --override log.cleaner.enable=true \
          --override log.cleaner.io.buffer.load.factor=0.9 \
          --override log.cleaner.io.buffer.size=524288 \
          --override log.cleaner.io.max.bytes.per.second=1.7976931348623157E308 \
          --override log.cleaner.min.cleanable.ratio=0.5 \
          --override log.cleaner.min.compaction.lag.ms=0 \
          --override log.cleaner.threads=1 \
          --override log.cleanup.policy=delete \
          --override log.index.interval.bytes=4096 \
          --override log.index.size.max.bytes=10485760 \
          --override log.preallocate=false \
          --override log.retention.check.interval.ms=300000 \
          --override max.connections.per.ip=2147483647 \
          --override num.partitions=1 \
          --override producer.purgatory.purge.interval.requests=1000 \
          --override replica.fetch.backoff.ms=1000 \
          --override replica.fetch.max.bytes=1048576 \
          --override replica.fetch.response.max.bytes=10485760 \
          --override reserved.broker.max.id=1000 "
        env:
        - name: KAFKA_HEAP_OPTS
          value : "-Xmx512M -Xms512M"
        - name: KAFKA_OPTS
          value: "-Dlogging.level=INFO"
        volumeMounts:
        - name: kafka-data
          mountPath: /var/lib/kafka
  volumeClaimTemplates:
  - metadata:
      name: kafka-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

那个zookeeper的地址按着自己的改一下吧,别的就没啥了,暂时还没加健康检查,不知道该怎么加,TCPsocket检查9200端口一直失败,不知道为啥,后期再琢磨,我这里已经创建完了,

[root@kubeadm ~]# kubectl get pod -l app=kafka
NAME      READY   STATUS    RESTARTS   AGE
kafka-0   1/1     Running   0          2m33s
kafka-1   1/1     Running   0          2m24s
kafka-2   1/1     Running   0          2m21s

所以去测试吧,再开一个创建,登陆到kafka容器,创建一个生产者,一个消费者,发消息试试。

[root@kubeadm ~]# kubectl exec -it kafka-0 sh
/ # kafka-console-producer.sh --broker-list localhost:9092 --topic test #创建生产者
[root@kubeadm ~]# kubectl exec -it kafka-1 sh
/ # kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test #消费者

然后在生产者随便写点东西,消费者那里就会有输出了。

就是这种效果,kafka也没问题,像是什么activemq&mongodb这类的我就不贴了,一个StatefulSet加一个无头服务就搞定了,结束。

最后编辑于: 2019 年 10 月 25 日
返回文章列表 文章二维码 打赏
本页链接的二维码
打赏二维码