Skip to main content
Version: 2.4.0

Setting up Data source


This guide provides sample scrape job to be used for Prometheus deployment’s scrape-configmap and service monitors to be used with Prometheus operator for the different architectural topologies for integrating Prometheus (connecting a data source link) with Chaos center.

Before you begin#

To setup a data source for a chaos center project, you must know about open observability and data source considerations in Litmus 2.0

Topologies#

Listed below are three among many topologies in which a data source can be setup for collecting agent cluster's metrics along with chaos metrics for chaos center.

For Control plane agent / Self agent with Prometheus - scraping chaos-exporter metrics and events along with other metrics#

Data flow and topological diagram

Scrape jobs:

  • Chaos exporter
- job_name: 'chaos-exporter'  static_configs:    - targets: ['chaos-exporter.litmus.svc.cluster.local:8080']  relabel_configs:    - target_label: instance      replacement: 'chaos-exporter-service'
  • Kube state metrics exporter
- job_name: 'kube-state-metrics'  static_configs:    - targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']

Detailed setup guide for Prometheus deployment with scrape configuration

Service monitors:

  • Chaos exporter
apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  name: chaos-exporter  labels:    k8s-app: chaos-exporter  namespace: litmusspec:  jobLabel: app  selector:    matchLabels:      app: chaos-exporter  namespaceSelector:    matchNames:      - litmus  endpoints:    - port: tcp      interval: 1s      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'
  • Kube state metrics exporter
apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    app.kubernetes.io/name: kube-state-metrics    app.kubernetes.io/version: latest    k8s-app: kube-state-metrics  name: kube-state-metrics  namespace: monitoringspec:  endpoints:    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token      honorLabels: true      interval: 30s      port: http-metrics      relabelings:        - action: labeldrop          regex: (pod|service|endpoint|namespace)      scheme: http      scrapeTimeout: 30s      tlsConfig:        insecureSkipVerify: true    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token      interval: 30s      port: telemetry      scheme: http      tlsConfig:        insecureSkipVerify: true  jobLabel: app.kubernetes.io/name  selector:    matchLabels:      app.kubernetes.io/name: kube-state-metrics

Detailed setup guide for Prometheus operator with service monitors

Note:#

Chaos exporter

labels

labels:  app: chaos-exporter  release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1kind: PodMonitormetadata:  name: chaos-exporter-monitor  namespace: monitoring  labels:    release: prometheus-stackspec:  selector:    matchLabels:      app: chaos-exporter  namespaceSelector:    matchNames:      - litmus  podMetricsEndpoints:    - port: tcp    - interval: 1s      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'

Black box exporter

labels

labels:  app: prometheus-blackbox-exporter  release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1kind: PodMonitormetadata:  name: black-box-exporter-monitor  namespace: monitoring  labels:    release: prometheus-stackspec:  selector:    matchLabels:      app: prometheus-blackbox-exporter  namespaceSelector:    matchNames:      - monitoring  podMetricsEndpoints:    - port: http    - interval: 1s

Detailed setup guide for Prometheus operator with pod monitors

For Multiple agents with multiple prometheus instances - scraping chaos-exporter metrics and events along with other metrics#

Data flow and topological diagram
  • Scrape job and service monitor remain same as in case of Control plane agent / self agent, the individual Prometheus instances can be connected as separate data sources to the Chaos center. Separate dashboards can be created by selecting specific agents and their corresponding data source which is essentially a Prometheus time series database, collecting metrics from the agent cluster for application / infra metrics, chaos events and chaos verdicts.

For Multiple agents with single prometheus - scraping chaos-exporter metrics and events along with other metrics#

  • Separate dashboards can be created by selecting specific agents and the data source which is essentially a Prometheus time series database, collecting metrics from the agent cluster for application / infra metrics, chaos events and chaos verdicts.
Data flow and topological diagram

Scrape jobs:

  • Chaos-exporter - agent-1
- job_name: 'chaos-exporter-agent-1'  static_configs:    - targets: ['<AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']  relabel_configs:    - target_label: instance      replacement: 'chaos-exporter-service'
  • Chaos-exporter - agent-2
- job_name: 'chaos-exporter-agent-2'  static_configs:    - targets: ['<AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']  relabel_configs:    - target_label: instance      replacement: 'chaos-exporter-service'
  • Kube state metrics exporter - agent-1
- job_name: 'kube-state-metrics-agent-1'  static_configs:    - targets: ['<AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']
  • Kube state metrics exporter - agent-2
- job_name: 'kube-state-metrics-agent-2'  static_configs:    - targets: ['<AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']

Detailed setup guide for Prometheus deployment with scrape configuration

Service endpoint, spec and monitors:

  • Chaos-exporter - agent-1
kind: ServiceapiVersion: v1metadata:  name: chaos-exporter-agent-1  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: chaos-exporter-agent-1  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: chaos-exporter-agent-1  name: chaos-exporter-agent-1  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: chaos-exporter-agent-1  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 1s      port: tcp      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'
  • Chaos-exporter - agent-2
kind: ServiceapiVersion: v1metadata:  name: chaos-exporter-agent-2  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: chaos-exporter-agent-2  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: chaos-exporter-agent-2  name: chaos-exporter-agent-2  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: chaos-exporter-agent-2  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 1s      port: tcp      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'
  • Kube state metrics exporter - agent-1
kind: ServiceapiVersion: v1metadata:  name: kube-state-metrics-agent-1  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: kube-state-metrics-agent-1  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: kube-state-metrics-agent-1  name: kube-state-metrics-agent-1  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: kube-state-metrics-agent-1  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 30s      port: tcp
  • Kube state metrics exporter - agent-2
kind: ServiceapiVersion: v1metadata:  name: kube-state-metrics-agent-2  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: kube-state-metrics-agent-2  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: kube-state-metrics-agent-2  name: kube-state-metrics-agent-2  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: kube-state-metrics-agent-2  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 30s      port: tcp

Detailed setup guide for Prometheus operator with service monitors

Resources#

Learn more#