prometheus从外部监控k8s集群

前言

K8s为阿里云平台的服务，其中的监控组件已经安装，直接调用即可。自建k8s的话，需要先安装相对应的监控组件。
本次监控是从外部监控k8s集群的。

一、配置k8s

1、创建用于 Prometheus 访问 Kubernetes 资源对象的 RBAC 对象

#查看监控pod所在namespace
kubectl get pods -A |grep kube-state

vim prom.rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: arms-prom   #填写对应的namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
    - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: arms-prom   #填写对应的namespace

2、在 Kubernetes 集群中创建上面的资源对象

kubectl apply -f prom.rbac.yaml

#获取上面的 Prometheus 对应的 Secret 的信息：
kubectl get sa prometheus -n arms-prom -o yaml

kubectl describe secret prometheus-token-6fz27 -n arms-prom

保存token为一个文件。

二、配置Prometheus

1、保存token文件

cd /data/prometheus-2.19.1.linux-amd64/
mkdir k8s
cd k8s ; vim token

2、配置采集任务

vim prometheus.yml
  - job_name: k8s-cadvisor
    honor_timestamps: true
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
    - api_server: https://172.16.7.8:6443
      role: node
      bearer_token_file: k8s/token
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: k8s/token
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - separator: ;
      regex: (.*)
      target_label: __address__
      replacement: 172.16.7.8:6443
      action: replace
    - source_labels: [__meta_kubernetes_node_name]
      separator: ;
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
      action: replace

  - job_name: kube-state-metrics
    static_configs:
            - targets: ['kubestate.c2ed8c8e8b6724f54b5c1896ded3a8370.cn-beijing.alicontainer.com']

  - job_name: 'ingress-nginx-endpoints'
    honor_timestamps: true
    metrics_path: /metrics
    scheme: http
    kubernetes_sd_configs:
    - api_server: https://172.16.7.8:6443
      role: pod
      bearer_token_file: k8s/token
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: k8s/token
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
    - source_labels: [__meta_kubernetes_service_name]
      regex: prometheus-server
      action: drop

3、重启Prometheus

成功，k8s主机监控使用常见的node-export监控就可以了。

4、画图