kube-benech

kube-beach

CIS 安全基准

• CIS安全基准介绍

• K8s安全基准工具 kube-bench

CIS安全基准

互联网安全中心（CIS，Center for Internet Security），是一个非盈利组织，致力为互联网提供免费的安全防御解决方案。

官网：https://www.cisecurity.org/

Kubernetes CIS基准：https://www.cisecurity.org/benchmark/kubernetes/

已经下载好的pdf如下：

📌 附件：《CIS_Kubernetes_Benchmark_v1.6.0.pdf》

CIS_Kubernetes_Benchmark_v1.6.0.pdf

下载时需要提供一个邮箱：

CIS基准测试工具

下载pdf后，根据里面的基准来检查K8s集群配置，但内容量太大，一般会采用相关工具来完成这项工作。

Kube-bench是容器安全厂商Aquq推出的工具，以CIS K8s基准作为基础，来检查K8s是否安全部署。

主要查找不安全的配置参数、敏感的文件权限、不安全的帐户或公开端口等等。

项目地址：https://github.com/aquasecurity/kube-bench

kube-bench是用go去写的。

CIS基准测试工具：kube-beach部署

==💘 实战：CIS基准测试工具：kube-beach部署(测试成功)-2022.11.13（成功测试）==

实验环境

1、win10,vmwrokstation虚机；
2、k8s集群：3台centos7.6 1810虚机，2个master节点,1个node节点 
   k8s version：v1.20
   CONTAINER-RUNTIME：docker://20.10.7

实验软件

kube-bench_0.6.3_linux_amd64.tar.gz

链接：https://pan.baidu.com/s/1Ja2AwjQDs6rzheihN41EpQ?pwd=n2uq

提取码：n2uq

部署过程如下：

1、下载二进制包

https://github.com/aquasecurity/kube-bench/releases

2、解压使用

[root@k8s-master1 ~]#ll -h  kube-bench_0.6.3_linux_amd64.tar.gz
-rw-r--r-- 1 root root 7.7M Jun 13  2021 kube-bench_0.6.3_linux_amd64.tar.gz

[root@k8s-master1 ~]#tar zxf kube-bench_0.6.3_linux_amd64.tar.gz
[root@k8s-master1 ~]#mkdir /etc/kube-bench # 创建默认配置文件路径
[root@k8s-master1 ~]#mv cfg /etc/kube-bench/cfg
[root@k8s-master1 ~]#mv kube-bench /usr/bin/

3、测试

[root@k8s-master1 ~]#kube-bench --help

部署完成。😘

CIS基准测试工具：kube-beach使用

使用kube-bench run进行测试，该指令有以下常用参数：

常用参数：

-s, --targets 指定要基础测试的目标，这个目标需要匹配cfg/<version>中的文件名称，已有目标：master, controlplane, node, etcd, policies

• --version：指定k8s版本，如果未指定会自动检测

• --benchmark：手动指定CIS基准版本，不能与--version一起使用

ls /etc/kube-bench/cfg/

目前通常使用的是：1.6.0版本。

ls /etc/kube-bench/cfg/cis-1.6/

案例：检查master组件安全配置

kube-bench run --targets=master

执行后会逐个检查安全配置并输出修复方案及汇总信息输出：

[PASS]：测试通过

[FAIL]：测试未通过，重点关注，在测试结果会给出修复建议

[WARN]：警告，可做了解

[INFO]：信息

🍀 测试过程：

[root@k8s-master1 ~]#kube-bench run --targets=master

CIS基准测试：

修复建议：

汇总信息：

注意：这个检查项就是按这个CIS_Kubernetes_Benchmark_v1.6.0.pdf上来做的配置。

🍀 找出FAILED的配置项：

[FAIL]  1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
[FAIL]  1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) #不太熟，忽略掉。
[FAIL]  1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) #后面会单独讲解。
[FAIL]  1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。

#审计日志
[FAIL]  1.2.22 Ensure that the --audit-log-path argument is set (Automated)
[FAIL]  1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)
[FAIL]  1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)
[FAIL]  1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)

[FAIL]  1.3.2 Ensure that the --profiling argument is set to false (Automated)
[FAIL]  1.4.1 Ensure that the --profiling argument is set to false (Automated)

检查项

因为当前k8s环境的etcd是通过静态pod启动起来的，因此这个检查项可忽略。

[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)

进一步测试

[FAIL]  1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。

#建议配置如下
1.2.21 Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml
on the master node and set the below parameter.
--profiling=false

开始配置：

[root@k8s-master1 ~]#vim /etc/kubernetes/manifests/kube-apiserver.yaml

配置完/etc/kubernetes/manifests/kube-apiserver.yaml配置文件后，可以看到kube-apiserver-k8s-master1pod发生了重启，此时kubelet也是活动的。（这里是不用重启kublet服务的）

此时，我们再次运行下kube-bench run --targets=master命令，观察现象：

可以看到，原来的1.2.21选项此时的检查结果为PASS了，符合预期。

配置完成。😘

例如：检查node组件安全配置

检查node时，会重点检查这个kubelet。

[root@k8s-master1 ~]#kube-bench run --targets=node
[INFO] 4 Worker Node Security Configuration
[INFO] 4.1 Worker Node Configuration Files
[PASS] 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated)
[PASS] 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated)
[PASS] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual)
[PASS] 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Manual)
[PASS] 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated)
[PASS] 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual)
[PASS] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual)
[PASS] 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual)
[PASS] 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated)
[PASS] 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated)
[INFO] 4.2 Kubelet
[PASS] 4.2.1 Ensure that the anonymous-auth argument is set to false (Automated)
[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
[PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated)
[PASS] 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual)
[PASS] 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual)
[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated)
[PASS] 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated)
[PASS] 4.2.8 Ensure that the --hostname-override argument is not set (Manual)
[WARN] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual)
[WARN] 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual)
[PASS] 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual)
[PASS] 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual)
[WARN] 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual)

== Remediations node ==
4.2.6 If using a Kubelet config file, edit the file to set protectKernelDefaults: true.
If using command line arguments, edit the kubelet service file
/lib/systemd/system/kubelet.service on each worker node and
set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
--protect-kernel-defaults=true
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service

4.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level.
If using command line arguments, edit the kubelet service file
/lib/systemd/system/kubelet.service on each worker node and
set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service

4.2.10 If using a Kubelet config file, edit the file to set tlsCertFile to the location
of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile
to the location of the corresponding private key file.
If using command line arguments, edit the kubelet service file
/lib/systemd/system/kubelet.service on each worker node and
set the below parameters in KUBELET_CERTIFICATE_ARGS variable.
--tls-cert-file=<path/to/tls-certificate-file>
--tls-private-key-file=<path/to/tls-key-file>
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service

4.2.13 If using a Kubelet config file, edit the file to set TLSCipherSuites: to
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
or to a subset of these values.
If using executable arguments, edit the kubelet service file
/lib/systemd/system/kubelet.service on each worker node and
set the --tls-cipher-suites parameter as follows, or to a subset of these values.
--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service


== Summary node ==
19 checks PASS
1 checks FAIL
3 checks WARN
0 checks INFO

== Summary total ==
19 checks PASS
1 checks FAIL
3 checks WARN
0 checks INFO

[root@k8s-master1 ~]#

例如：检查etcd组件安全配置

案例：如何跳过检查项

测试项目配置文件：/etc/kube-bench/cfg/cis-1.6/

• id：编号

• text：提示的文本
• audit：
• tests：测试项目
• remediation：修复方案
• scored：如果为true，kube-bench无法正常测试，则会生成FAIL，如果为false，无法正常测试，则会
生成WARN。#如果设置为false，则会生成WARN。
• type：如果为manual则会生成WARN，如果为skip，则会生成INFO

案例测试：

[root@k8s-master1 ~]#vim /etc/kube-bench/cfg/cis-1.6/master.yaml

kube-bench run --targets=mster的检测未通过项。

[FAIL]  1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
[FAIL]  1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) #不太熟，忽略掉。
[FAIL]  1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) #后面会单独讲解。
[FAIL]  1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。

#审计日志
[FAIL]  1.2.22 Ensure that the --audit-log-path argument is set (Automated)
[FAIL]  1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)
[FAIL]  1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)
[FAIL]  1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)

[FAIL]  1.3.2 Ensure that the --profiling argument is set to false (Automated)
[FAIL]  1.4.1 Ensure that the --profiling argument is set to false (Automated)

我们可以看一下官网：

kube-controller-manager和kube-apiserver的这个profiling功能还没弃用，但是kube-scheduler的这个profiling功能已经被弃用了；

但是这个kube-bench这个脚本的检测程序还在，说明kube-bench并没有及时地区更新；

此时，想要把kube-scheduler组件的profiling检查项给忽略掉，该怎么办呢？

因此需要用到如下选项：type：如果为manual则会生成WARN，如果为skip，则会生成INFO

[root@k8s-master1 ~]#vim /etc/kube-bench/cfg/cis-1.6/master.yaml

再次执行kube-bench run --targets=master，可以看到1.4.1检查项此时为INFO了；

测试结束。😘

注意事项

[FAIL]  1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)
[FAIL]  1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) #不太熟，忽略掉。
[FAIL]  1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated) #后面会单独讲解。PodSecurityPolicy这个本来就是个独立的功能，可用可不用；
[FAIL]  1.2.21 Ensure that the --profiling argument is set to false (Automated) #pprod 性能分析程序，go语言。默认是打开的。

#审计日志
[FAIL]  1.2.22 Ensure that the --audit-log-path argument is set (Automated)
[FAIL]  1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)
[FAIL]  1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)
[FAIL]  1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)

[FAIL]  1.3.2 Ensure that the --profiling argument is set to false (Automated)
[FAIL]  1.4.1 Ensure that the --profiling argument is set to false (Automated)

一般在master上是可以检查node的；（当然直接在node节点安装kube-bench跑程序也是可以的）
例如有3个master节点，只需要在1个master节点执行检查项就行，然后在all master节点配置相同的加固箱就好；（同样，node加固项也是需要做相同的操作）
可以以json格式输出的：kube-bench run --help，--json Prints the results as JSON。