上文我们使用最新的kubernetes v1.16.0搭建了集群,但是由于版本导致了遇到问题,首先是dashboard不兼容1.16.0的Api,其次是flannel-cni的问题,本文重新用稳定版v1.15.3部署。

基础环境配置

准备三台虚拟机

docker 192.168.100.10
node01 192.168.100.11 kube-node01 master
node01 192.168.100.12 kube-node02 minion
node01 192.168.100.13 kube-node03 minion

配置hosts文件

1
2
3
4
5
cat >> /etc/hosts<<EOF  
192.168.100.11  kube-node01
192.168.100.12  kube-node02
192.168.100.13  kube-node03
EOF

修改hostname文件

sudo hostnamectl set-hostname

关闭系统防火墙

systemctl stop firewalld && systemctl disable firewalld

禁用swap内存交换

1
2
3
swapoff -a
echo "swapoff -a" >>/etc/rc.d/rc.local
chmod +x /etc/rc.d/rc.local

注意:或开机禁用swap: 编辑/etc/fstab –> 注释掉swap 分区

关闭selinux服务

临时关闭:setenforce 0 永久关闭:vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled 设置后需要重启才能生效,命令如下:
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/sysconfig/selinux

配置iptable管理ipv4/6请求

1
2
3
4
5
6
sudo echo "1" > /proc/sys/net/ipv4/ip_forward
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

执行 sysctl –system 使配置生效

校对系统时间

1
2
yum -y install ntp
systemctl start ntpd && systemctl enable ntpd

集群环境配置

安装docker服务

配置源wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

安装docker-ce容器服务:yum -y install docker-ce
查看docker版本号:docker –version和详细信息:docker info

添加开机自启动和启动服务:systemctl enable docker && systemctl start docker
修改docker启动参数:

1
2
3
4
5
6
7
cat  > /etc/docker/daemon.json  <<EOF  
{
     "registry-mirrors": ["http://192.168.100.10"],
     "insecure-registries":\["192.168.100.10"\],
     "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF


修改docker的启动服务脚本docker.service:
在[Service]节点下增加:ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
修改完成使用:systemctl daemon-reload && systemctl restart docker重启服务

安装Kubernetes组件

配置源:

1
2
3
4
5
6
7
8
9
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装组件:yum install -y kubelet-1.15.3 kubeadm-1.15.3 kubectl-1.15.3

配置启动kubelet 组件

配置kubelet使用国内pause镜像和配置kubelet的cgroups:
cgroups要和docker的配置一样,使用dokcer info可查看
vi /var/lib/kubelet/kubeadm-flags.env

vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
添加环境变量:
Environment=”KUBELET_CGROUP_ARGS=–runtime-cgroups=/systemd/system.slice –kubelet-cgroups=/systemd/system.slice”
并将环境变量KUBELET_CGROUP_ARGS写入启动参数

如果上面/var/lib/kubelet/kubeadm-flags.env没有cgroup-driver和pod-infra-container-image的信息,则写入/etc/sysconfig/kubelet的KUBELET_EXTRA_ARGS启动参数中加载

1
2
3
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --pod-infra-container-image=192.168.100.10/kubernetes/pause:3.1 
EOF

使配置生效:systemctl daemon-reload
添加自启动:systemctl enable kubelet

配置Master节点

在master节点上创建初始化脚本:vi /etc/kubernetes/conf/kubeadm-init.sh

1
2
3
4
5
kubeadm init \
--kubernetes-version=v1.16.0 \
--pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=192.168.100.11 \
--image-repository=192.168.100.10/kubernetes

修改脚本权限:chmod +x /etc/kubernetes/conf/kubeadm-init.sh

由于初始化时,默认会从k8s.gcr.io拉取镜像,该镜像被墙,这里从私有docker镜像库拉取,由–image-repository=192.168.100.10/kubernetes指定
私有docker镜像库搭建请参照我的另一篇博文:《Docker搭建私有镜像仓库》

5.初始化Master节点

初始化时及搭建集群过程中所需要的所有镜像我们都已托管到我们私有镜像仓库中:

执行/etc/kubernetes/conf/kubeadm-init.sh此时会初始化
注意:如果初始化过程出现问题,使用如下命令重置

1
2
kubeadm reset
rm -rf /var/lib/cni/ $HOME/.kube/config

初始化成功如下图:

1
2
3
kubeadm join 192.168.100.11:6443
--token qjqxwd.2k79lzjvxc3t6hsj
--discovery-token-ca-cert-hash sha256:091e12d369cd83fda3187b5eddb1d07db074b220ff32cb0f21d8b82fc19d4ccb

上面这一句是给其他节点加入集群用的,要保存下来,后面要用。
配置master上通过 kubectl 管理集群,执行下面的命令:

1
2
3
4
rm -rf $HOME/.kube
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


使用kubectl get nodes -owide查看刚初始化的主节点信息:

我们看到master节点的状态时未就绪状态,需要配置使用网络flannel插件:
下载flannel配置文件:
wget -P /etc/kubernetes/conf
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
修改下载的flannel.yml文件,删除多余部分,并指定网卡信息:

启动flannel组件:kubectl apply -f /etc/kubernetes/conf/kube-flannel.yml

加入各Node节

在每个节点的机器上执行下面的加入集群的命令:

1
2
3
kubeadm join 192.168.100.11:6443
--token qjqxwd.2k79lzjvxc3t6hsj
--discovery-token-ca-cert-hash sha256:091e12d369cd83fda3187b5eddb1d07db074b220ff32cb0f21d8b82fc19d4ccb


部署Kubernetes Web

从kubernetes官方github下载配置文件:
wget /etc/kubernetes/conf https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

修改配置文件访问类型为:NodePort

启动webui组件:kubectl apply -f /etc/kubernetes/conf/kubernetes-dashboard.yaml

查看dashboard pod的状态:kubectl get pods -n kube-system -owide

然后通过:https://192.168.100.11:31080/访问

我们看到有两种访问方式,下面我们配置这两种访问方式:
我们创建dashboard用户yaml文件:
vi /etc/kubernetes/conf/kubernetes-dashboard-admin.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Create Dashboard Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
name: dashboard-admin-user
namespace: kube-system
---
# Create ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: dashboard-admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: dashboard-admin-user
namespace: kube-system


然后kubectl apply -f /etc/kubernetes/conf/kubernetes-dashboard-user.yaml
完成后执行kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep dashboard-admin-user | awk ‘{print $1}’) 查看token

获取的token即可用来在页面上输入登录:

1
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdXNlci10b2tlbi13NGg1biIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdXNlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImNjNzE5NTMwLWY0NDAtNGNmYi1hNDkwLTZiZjkzNWNkMDYxOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpkYXNoYm9hcmQtYWRtaW4tdXNlciJ9.No8ic7iNedWgX9iX1cIPHrK81vDTApsayPHm2U5No65yndQX7l_GgE8Ze-oTqx1JQVN85EWdJGM4T_LdbxflkIiBElmKRST7m3VpAeLgNl1f56NyyUwf0GEVzsdZdhb0E-i1DJO0ofXkqQiHwtcpWMmq4wqlYymu_fDxblJl4LzTpPUV89emPSQhSA3ZopbNBKJr6u5kRYgsTiz8EGAhRRqWGzCyEkyp8Y_SJlYsnXXx-F8gmneBiyaqeCrNiO71eczz6sQDsEI37fhxs41xeLJNF7sA2LihAE9KD3YBBgRjxxMmfmsGb-IkTn68bhhsHiCN0AMW5xJFSXNiLyWGMQ


此时的dashboard没有展示cpu、内存等图标可视化信息,我们需要安装heapster服务(因为dashboard要显示图表数据需要依赖heapster服务)
下载 heapster 相关 yaml 文件:

1
2
3
4
wget /etc/kubernetes/heapster https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget /etc/kubernetes/heapster https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget /etc/kubernetes/heapster https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
wget /etc/kubernetes/heapster https://raw.githubusercontent.com/kubernetes-retired/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml


查看需要部署的镜像:

1
2
3
cat grafana.yaml | grep image
cat heapster.yaml | grep image
cat influxdb.yaml | grep image


修改yaml文件:
因为k8s高版本的api版本进行了变化,将上面四个yaml文件中的apiVersion: extensions/v1beta1 改为apiVersion: apps/v1
因为kubelet 只在 10250 监听 https 请求,将heapster.yaml中的- –source=kubernetes:https://kubernetes.default 修改为:

1
- --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true


修改上面四个yaml文件中的spec节点,增加selector,如下图:

然后在heapster配置文件的当前目录下执行部署:kubectl apply -f .
注意:如果部署发生错误,我们执行kubectl delete -f . 进行回退
Heapster各个组件部署成功如下:

然后我们生成使用config登录的文件:
##将secret中的token使用base64方式进行解码,然后使用变量引用
DASH_TOCKEN=$(kubectl get secret -n kube-system dashboard-admin-token-q6pz8 -o jsonpath={.data.token}|base64 -d)
##创建一个集群
kubectl config set-cluster cluster-admin –server=192.168.100.11:6443 –kubeconfig=/etc/kubernetes/conf/dashbord-admin.conf
##创建一个集群用户,并引用sa的token
kubectl config set-credentials dashboard-admin-user –token=$DASH_TOCKEN –kubeconfig=/etc/kubernetes/conf/dashbord-admin.conf
##创建一个上下文,指定集群名、集群用户名
kubectl config set-context dashboard-admin-user@cluster-admin –cluster=cluster-admin –user=dashboard-admin-user
–kubeconfig=/etc/kubernetes/conf/dashbord-admin.conf
##设置集群中当前使用的用户
kubectl config use-context dashboard-admin-user@cluster-admin –kubeconfig=/etc/kubernetes/conf/dashbord-admin.conf
然后使用token或生成的文件登录成功如下:

验证集群状态

使用kubectl get nodes -n kube-system -owide 查看节点列表

使用kubectl get pods -n kube-system -owide查看pod列表

使用kubectl get svc -n kube-system -owide 查看服务列表

集群问题解决

初始化集群异常

问题描述:执行kubeadm init时报出/proc/sys/net/ipv4/ip_forward contents are not set to 1的错误
解决方案:sudo echo “1” > /proc/sys/net/ipv4/ip_forward

部署heapster提示版本问题

问题描述:部署heapster 组件提示no matches for king “Deployment” in version “extensions/v1beta1”

解决方案:是因为k8s高版本的api版本进行了变化,将对应的yaml文件中的extensions/v1beta1 改为apiVersion: apps/v1

部署heapster提示selector错误

问题描述:部署heapster 组件提示missing required field “selector” in io.k8s.api.apps.v1.DeploymentSpec的错误

解决方案:修改heapster上面四个yaml文件中的spec节点,增加selector,如下图:

部署heapster后不显示图表信息

问题描述:heapster正常部署后,pods状态都正常,但是dashboard不显示图表信息
查看pods的日志,发现一直提示“403 Forbidden”, response: “Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)”
解决方案:
查看ClusterRole: system:heapster的权限,发现的确没有针对Resource: nodes/stats 的create权限
kubectl describe clusterrole system:heapster
添加Resource: nodes/stats的create权限,vi /etc/kubernetes/conf/heapster/heapster-role.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:heapster
rules:
- apiGroups:
- ""
resources:
- events
- namespaces
- nodes
- pods
- nodes/stats
verbs:
- create
- get
- list
- watch
- apiGroups:
- extensions
resources:
- deployments
verbs:
- get
- list
- watch
1
2
3
cd /etc/kubernetes/conf/heapster
kubectl delete -f .
kubectl apply -f .