Usage Back to Top

Installation

We won’t cover further details how to properly setup Prometheus itself, we will only cover some basic setup based on docker-compose. But if you want to run this service discovery without docker-compose you should be able to adopt that to your needs.

First of all we need to prepare a configuration for Prometheus that includes the service discovery which simply maps to a node exporter.

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m

scrape_configs:
- job_name: node
  file_sd_configs:
  - files: [ "/etc/sd/vcd.json" ]
  relabel_configs:
  - source_labels: [__meta_vcd_network_internal]
    replacement: "${1}:9100"
    target_label: __address__
  - source_labels: [__meta_vcd_name]
    target_label: instance
- job_name: vcd-sd
  static_configs:
  - targets:
    - vcd-sd:9000

After preparing the configuration we need to create the docker-compose.yml within the same folder, this docker-compose.yml starts a simple Prometheus instance together with the service discovery. Don’t forget to update the envrionment variables with the required credentials. If you are using a different volume for the service discovery you have to make sure that the container user is allowed to write to this volume.

version: '2'

volumes:
  prometheus:

services:
  prometheus:
    image: prom/prometheus:latest
    restart: always
    ports:
      - 9090:9090
    volumes:
      - prometheus:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./service-discovery:/etc/sd

  vcd-sd:
    image: promhippie/prometheus-vcd-sd:latest
    restart: always
    environment:
      - PROMETHEUS_VCD_LOG_PRETTY=true
      - PROMETHEUS_VCD_OUTPUT_FILE=/etc/sd/vcd.json
      - PROMETHEUS_VCD_URL=https://vdc.example.com/api
      - PROMETHEUS_VCD_USERNAME=username
      - PROMETHEUS_VCD_PASSWORD=p455w0rd
      - PROMETHEUS_VCD_ORG=MY-ORG1
      - PROMETHEUS_VCD_VDC=MY-ORG1-DC1
    volumes:
      - ./service-discovery:/etc/sd

Since our latest tag always refers to the master branch of the Git repository you should always use some fixed version. You can see all available tags at DockerHub or Quay, there you will see that we also provide a manifest, you can easily start the exporter on various architectures without any change to the image name. You should apply a change like this to the docker-compose.yml file:

  vcd-sd:
-   image: promhippie/prometheus-vcd-sd:latest
+   image: promhippie/prometheus-vcd-sd:0.3.0
    restart: always
    environment:
      - PROMETHEUS_VCD_LOG_PRETTY=true
      - PROMETHEUS_VCD_OUTPUT_FILE=/etc/sd/vcd.json
      - PROMETHEUS_VCD_URL=https://vdc.example.com/api
      - PROMETHEUS_VCD_USERNAME=username
      - PROMETHEUS_VCD_PASSWORD=p455w0rd
      - PROMETHEUS_VCD_ORG=MY-ORG1
      - PROMETHEUS_VCD_VDC=MY-ORG1-DC1
    volumes:
      - ./service-discovery:/etc/sd

Depending on how you have launched and configured Prometheus it’s possible that it’s running as user nobody, in that case you should run the service discovery as this user as well, otherwise Prometheus won’t be able to read the generated JSON file:

  vcd-sd:
    image: promhippie/prometheus-vcd-sd:latest
    restart: always
+   user: '65534'
    environment:
      - PROMETHEUS_VCD_LOG_PRETTY=true
      - PROMETHEUS_VCD_OUTPUT_FILE=/etc/sd/vcd.json
      - PROMETHEUS_VCD_URL=https://vdc.example.com/api
      - PROMETHEUS_VCD_USERNAME=username
      - PROMETHEUS_VCD_PASSWORD=p455w0rd
      - PROMETHEUS_VCD_ORG=MY-ORG1
      - PROMETHEUS_VCD_VDC=MY-ORG1-DC1
    volumes:
      - ./service-discovery:/etc/sd

If you want to secure the access to the exporter or also the HTTP service discovery endpoint you can provide a web config. You just need to provide a path to the config file in order to enable the support for it, for details about the config format look at the documentation section:

  vcd-sd:
    image: promhippie/prometheus-vcd-sd:latest
    restart: always
    environment:
+     - PROMETHEUS_VCD_WEB_CONFIG=path/to/web-config.json
      - PROMETHEUS_VCD_LOG_PRETTY=true
      - PROMETHEUS_VCD_OUTPUT_FILE=/etc/sd/vcd.json
      - PROMETHEUS_VCD_URL=https://vdc.example.com/api
      - PROMETHEUS_VCD_USERNAME=username
      - PROMETHEUS_VCD_PASSWORD=p455w0rd
      - PROMETHEUS_VCD_ORG=MY-ORG1
      - PROMETHEUS_VCD_VDC=MY-ORG1-DC1
    volumes:
      - ./service-discovery:/etc/sd

To avoid the dependency on a shared filesystem between this service discovery and the Prometheus configuration directory, you are able to use the new HTTP service discovery starting with Prometheus >= v2.28, you just need to switch the engine for this service discovery:

  vcd-sd:
    image: promhippie/prometheus-vcd-sd:latest
    restart: always
    environment:
      - PROMETHEUS_VCD_LOG_PRETTY=true
+     - PROMETHEUS_VCD_OUTPUT_ENGINE=http
      - PROMETHEUS_VCD_OUTPUT_FILE=/etc/sd/vcd.json
      - PROMETHEUS_VCD_URL=https://vdc.example.com/api
      - PROMETHEUS_VCD_USERNAME=username
      - PROMETHEUS_VCD_PASSWORD=p455w0rd
      - PROMETHEUS_VCD_ORG=MY-ORG1
      - PROMETHEUS_VCD_VDC=MY-ORG1-DC1
    volumes:
      - ./service-discovery:/etc/sd

To use the HTTP service discovery you just need to change the Prometheus configuration mentioned above a little bit:

scrape_configs:
- job_name: node
  http_sd_configs:
  - url: http://vcd-sd:9000/sd
  relabel_configs:
  - source_labels: [__meta_vcd_network_internal]
    replacement: "${1}:9100"
    target_label: __address__
  - source_labels: [__meta_vcd_name]
    target_label: instance
    target_label: instance

Finally the service discovery should be configured fine, let’s start this stack with docker-compose], you just need to execute docker-compose up within the directory where you have stored prometheus.yml and docker-compose.yml. That’s all, the service discovery should be up and running. You can access Prometheus at http://localhost:9090.

Prometheus service discovery for VMWare vCloud Director

Configuration

Envrionment variables

If you prefer to configure the service with environment variables you can see the available variables below, in case you want to configure multiple accounts with a single service you are forced to use the configuration file as the environment variables are limited to a single account. As the service is pretty lightweight you can even start an instance per account and configure it entirely by the variables, it’s up to you.

PROMETHEUS_VCD_LOG_LEVEL: Only log messages with given severity, defaults to info
PROMETHEUS_VCD_LOG_PRETTY: Enable pretty messages for logging, defaults to false
PROMETHEUS_VCD_WEB_ADDRESS: Address to bind the metrics server, defaults to 0.0.0.0:9000
PROMETHEUS_VCD_WEB_PATH: Path to bind the metrics server, defaults to /metrics
PROMETHEUS_VCD_WEB_CONFIG: Path to web-config file
PROMETHEUS_VCD_OUTPUT_ENGINE: Enabled engine like file or http, defaults to file
PROMETHEUS_VCD_OUTPUT_FILE: Path to write the file_sd config, defaults to /etc/prometheus/vcd.json
PROMETHEUS_VCD_OUTPUT_REFRESH: Discovery refresh interval in seconds, defaults to 30
PROMETHEUS_VCD_URL: URL for the vCloud Director API
PROMETHEUS_VCD_INSECURE: Accept self-signed certs for the vCloud Director API, defaults to false
PROMETHEUS_VCD_USERNAME: Username for the vCloud Director API
PROMETHEUS_VCD_PASSWORD: Password for the vCloud Director API
PROMETHEUS_VCD_ORG: Organization for the vCloud Director API
PROMETHEUS_VCD_VDC: vDatacenter for the vCloud Director API
PROMETHEUS_VCD_CONFIG: Path to vCloud Director configuration file

Web Configuration

If you want to secure the service by TLS or by some basic authentication you can provide a YAML configuration file whch follows the Prometheus toolkit format. You can see a full configration example within the toolkit documentation.

Configuration file

Especially if you want to configure multiple accounts within a single service discovery you got to use the configuration file. So far we support the file formats JSON and YAML, if you want to get a full example configuration just take a look at our repository, there you can always see the latest configuration format. These example configurations include all available options, they also include the default values.

Labels

__address__
__meta_vcd_id
__meta_vcd_metadata_<name>
__meta_vcd_name
__meta_vcd_network_<name>
__meta_vcd_num_cores_per_socket
__meta_vcd_num_cpus
__meta_vcd_org
__meta_vcd_os_type
__meta_vcd_project
__meta_vcd_status
__meta_vcd_storage_profile
__meta_vcd_vdc

Metrics

prometheus_vcd_sd_request_duration_seconds{project, type}: Histogram of latencies for requests to the VMWare vCloud Director API
prometheus_vcd_sd_request_failures_total{project, type}: Total number of failed requests to the VMWare vCloud Director API

Kubernetes Back to Top

Kubernetes

Currently we are covering the most famous installation methods on Kubernetes, you can choose between Kustomize and Helm.

Kustomize

We won’t cover the installation of Kustomize within this guide, to get it installed and working please read the upstream documentation. After the installation of Kustomize you just need to prepare a kustomization.yml wherever you like similar to this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: prometheus-vcd-sd

resources:
  - github.com/promhippie/prometheus-vcd-sd//deploy/kubernetes?ref=master

configMapGenerator:
  - name: prometheus-vcd-sd
    behavior: merge
    literals: []

secretGenerator:
  - name: prometheus-vcd-sd
    behavior: merge
    literals: []

After that you can simply execute kustomize build | kubectl apply -f - to get the manifest applied. Generally it’s best to use fixed versions of the container images, this can be done quite easy, you just need to append this block to your kustomization.yml to use this specific version:

images:
  - name: quay.io/promhippie/prometheus-vcd-sd
    newTag: 1.1.0

After applying this manifest the exporter should be directly visible within your Prometheus instance if you are using the Prometheus Operator as these manifests are providing a ServiceMonitor.

To consume the service discovery within Prometheus you got to configre matching scrape targets using the HTTP engine, just add a block similar to this one to your Prometheus configuration:

scrape_configs:
- job_name: node
  http_sd_configs:
  - url: http://vcd-sd.prometheus-vcd-sd.svc.cluster.local:9000/sd
  relabel_configs:
  - source_labels: [__meta_vcd_network_internal]
    replacement: "${1}:9100"
    target_label: __address__
  - source_labels: [__meta_vcd_name]
    target_label: instance
    target_label: instance

Helm

We won’t cover the installation of Helm within this guide, to get it installed and working please read the upstream documentation. After the installation of Helm you just need to execute the following commands:

helm repo add promhippie https://promhippie.github.io/charts
helm show values promhippie/prometheus-vcd-sd
helm install prometheus-vcd-sd promhippie/prometheus-vcd-sd

You can also watch that available values and generally the details of the chart provided by us within our chart repository.

After applying this manifest the exporter should be directly visible within your Prometheus instance depending on your installation if you enabled the annotations or the service monitor.