Prometheus

Prerequisites

The prometheus-stack has been setup by the cluster administrator

Component	Use	Access
Prometheus		prometheus.127.0.0.1.nip.io
Alert Manager		alertmanager.127.0.0.1.nip.io
prometheus-operator	Declarative management of prometheus configuration
Grafana	Monitoring Dashboards	grafana.127.0.0.1.nip.io
Grafana Operator	Declarative/GitOps configuration of grafana dashboards
Karma	Multi-Cluster alert viewer	karma.127.0.0.1.nip.io

Scraping metrics

In order to ingest metrics you need to configure a service monitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name:  YOUR-APP-MONITOR
  namespace: YOUR-NAMESPACE
spec:
  endpoints:
    - interval: 30s
      targetPort: 8080
  jobLabel: canary-checker
  namespaceSelector:
    matchNames:
      - YOUR-NAMESPACE
  selector:
    matchLabels:
      app: YOUR-APP

See ServiceMonitorSpec for all available options

Creating custom dashboards

Warning

The default password for Grafana is root/secret but note that any changes will not persist across restarts. To make changes persistent export the dashboard as JSON and create a GrafanaDashboard CRD

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: simple-dashboard
  labels:
    app: grafana
spec:
  name: simple-dashboard.json
  json: >
    {
      "id": null,
      "title": "Simple Dashboard",
      "tags": [],
      "style": "dark",
      "timezone": "browser",
      "editable": true,
      "hideControls": false,
      "graphTooltip": 1,
      "panels": [],
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "timepicker": {
        "time_options": [],
        "refresh_intervals": []
      },
      "templating": {
        "list": []
      },
      "annotations": {
        "list": []
      },
      "refresh": "5s",
      "schemaVersion": 17,
      "version": 0,
      "links": []
    }