Setting up Data source

---

This guide provides sample scrape job to be used for Prometheus deployment’s scrape-configmap and service monitors to be used with Prometheus operator for the different architectural topologies for integrating Prometheus (connecting a data source link) with Chaos center.

Before you begin

To setup a data source for a chaos center project, you must know about open observability and data source considerations in Litmus 2.0

Topologies

Listed below are three among many topologies in which a data source can be setup for collecting agent cluster's metrics along with chaos metrics for chaos center.

For Control plane agent / Self agent with Prometheus - scraping chaos-exporter metrics and events along with other metrics

Data flow and topological diagram

Scrape jobs:

• Chaos exporter

- job_name: 'chaos-exporter'  static_configs:    - targets: ['chaos-exporter.litmus.svc.cluster.local:8080']  relabel_configs:    - target_label: instance      replacement: 'chaos-exporter-service'

• Kube state metrics exporter

- job_name: 'kube-state-metrics'  static_configs:    - targets: ['kube-state-metrics.monitoring.svc.cluster.local:8080']

• Node exporter
• Black box exporter
• cAdvisor

Detailed setup guide for Prometheus deployment with scrape configuration

Service monitors:

• Chaos exporter

apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  name: chaos-exporter  labels:    k8s-app: chaos-exporter  namespace: litmusspec:  jobLabel: app  selector:    matchLabels:      app: chaos-exporter  namespaceSelector:    matchNames:      - litmus  endpoints:    - port: tcp      interval: 1s      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'

• Kube state metrics exporter

apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    app.kubernetes.io/name: kube-state-metrics    app.kubernetes.io/version: latest    k8s-app: kube-state-metrics  name: kube-state-metrics  namespace: monitoringspec:  endpoints:    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token      honorLabels: true      interval: 30s      port: http-metrics      relabelings:        - action: labeldrop          regex: (pod|service|endpoint|namespace)      scheme: http      scrapeTimeout: 30s      tlsConfig:        insecureSkipVerify: true    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token      interval: 30s      port: telemetry      scheme: http      tlsConfig:        insecureSkipVerify: true  jobLabel: app.kubernetes.io/name  selector:    matchLabels:      app.kubernetes.io/name: kube-state-metrics

• Node exporter
• Black box exporter

Detailed setup guide for Prometheus operator with service monitors

Note:

• Scrape jobs as per (above links) can also be added under this additionalScrapeConfigs for the Prometheus community’s Kube prometheus stack

• Pod labels to be used with Prometheus operator installed via helm as part of the Prometheus community’s Kube prometheus stack configured for podMonitors or with Prometheus’s inbuilt Kubernetes service discovery

Chaos exporter

labels

labels:  app: chaos-exporter  release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1kind: PodMonitormetadata:  name: chaos-exporter-monitor  namespace: monitoring  labels:    release: prometheus-stackspec:  selector:    matchLabels:      app: chaos-exporter  namespaceSelector:    matchNames:      - litmus  podMetricsEndpoints:    - port: tcp    - interval: 1s      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'

Black box exporter

labels

labels:  app: prometheus-blackbox-exporter  release: prometheus-stack

pod monitor

apiVersion: monitoring.coreos.com/v1kind: PodMonitormetadata:  name: black-box-exporter-monitor  namespace: monitoring  labels:    release: prometheus-stackspec:  selector:    matchLabels:      app: prometheus-blackbox-exporter  namespaceSelector:    matchNames:      - monitoring  podMetricsEndpoints:    - port: http    - interval: 1s

Detailed setup guide for Prometheus operator with pod monitors

For Multiple agents with multiple prometheus instances - scraping chaos-exporter metrics and events along with other metrics

Data flow and topological diagram

• Scrape job and service monitor remain same as in case of Control plane agent / self agent, the individual Prometheus instances can be connected as separate data sources to the Chaos center. Separate dashboards can be created by selecting specific agents and their corresponding data source which is essentially a Prometheus time series database, collecting metrics from the agent cluster for application / infra metrics, chaos events and chaos verdicts.

For Multiple agents with single prometheus - scraping chaos-exporter metrics and events along with other metrics

• Separate dashboards can be created by selecting specific agents and the data source which is essentially a Prometheus time series database, collecting metrics from the agent cluster for application / infra metrics, chaos events and chaos verdicts.

Data flow and topological diagram

Scrape jobs:

• Chaos-exporter - agent-1

- job_name: 'chaos-exporter-agent-1'  static_configs:    - targets: ['<AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']  relabel_configs:    - target_label: instance      replacement: 'chaos-exporter-service'

• Chaos-exporter - agent-2

- job_name: 'chaos-exporter-agent-2'  static_configs:    - targets: ['<AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']  relabel_configs:    - target_label: instance      replacement: 'chaos-exporter-service'

• Kube state metrics exporter - agent-1

- job_name: 'kube-state-metrics-agent-1'  static_configs:    - targets: ['<AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']

• Kube state metrics exporter - agent-2

- job_name: 'kube-state-metrics-agent-2'  static_configs:    - targets: ['<AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP:8080>']

• Node exporter
• Black box exporter
• cAdvisor

Detailed setup guide for Prometheus deployment with scrape configuration

Service endpoint, spec and monitors:

• Chaos-exporter - agent-1

kind: ServiceapiVersion: v1metadata:  name: chaos-exporter-agent-1  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: chaos-exporter-agent-1  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_1_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: chaos-exporter-agent-1  name: chaos-exporter-agent-1  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: chaos-exporter-agent-1  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 1s      port: tcp      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'

• Chaos-exporter - agent-2

kind: ServiceapiVersion: v1metadata:  name: chaos-exporter-agent-2  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: chaos-exporter-agent-2  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_2_CHAOS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: chaos-exporter-agent-2  name: chaos-exporter-agent-2  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: chaos-exporter-agent-2  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 1s      port: tcp      metricRelabelings:        - targetLabel: instance          replacement: 'chaos-exporter-service'

• Kube state metrics exporter - agent-1

kind: ServiceapiVersion: v1metadata:  name: kube-state-metrics-agent-1  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: kube-state-metrics-agent-1  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_1_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: kube-state-metrics-agent-1  name: kube-state-metrics-agent-1  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: kube-state-metrics-agent-1  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 30s      port: tcp

• Kube state metrics exporter - agent-2

kind: ServiceapiVersion: v1metadata:  name: kube-state-metrics-agent-2  namespace: monitoringspec:  type: ClusterIP  ports:    - name: tcp      port: 8080      targetPort: 8080---kind: EndpointsapiVersion: v1metadata:  name: kube-state-metrics-agent-2  namespace: monitoringsubsets:  - addresses:      - ip: <AGENT_2_KUBE_STATE_METRICS_EXPORTER_SERVICE_PUBLIC_HOSTNAME/IP>    ports:      - name: tcp        port: 8080---apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  labels:    k8s-app: kube-state-metrics-agent-2  name: kube-state-metrics-agent-2  namespace: monitoringspec:  jobLabel: app  selector:    matchLabels:      app: kube-state-metrics-agent-2  namespaceSelector:    matchNames:      - monitoring  endpoints:    - interval: 30s      port: tcp

• Node exporter
• Black box exporter

Detailed setup guide for Prometheus operator with service monitors

Resources

Learn more

• Observability Setup
• Configure Data Source
• Manage Application Dashboards