Kubernetes

Installing Prometheus and Grafana on VMware Cloud PKS

There are several quoted variations of “You can’t ___ what you can’t measure” being referenced ad nauseum in both business and technological contexts across the internet. Among the adherents to this shifting mantra appears to be the Cloud Native Computing Foundation, which began its existence in 2017 with Kubernetes and three monitoring projects under its banner, including Prometheus. This post will detail the process to deploy Prometheus as a metrics monitoring framework for Kubernetes, in conjunction with Grafana as a visualization engine for those metrics.

 

Kubernetes

Kubernetes is an open-source system for automating the deployment, scaling and management of containerized applications. Kubernetes is based on technology which was originally designed by Google and is now maintained by the Cloud Native Computing Foundation. Between deployments from the open-source codebase and the commercial offerings based on that code, Kubernetes is becoming a dominant industry platform for containers in production. The items pertaining to extracting and analyzing metrics from the containers themselves may apply to other container ecosystems as well.

 

How Is Monitoring Kubernetes Different?

Containerized systems such as Kubernetes environments present new monitoring challenges as compared to virtual machine-based compute environments. These differences include:

  • The ephemeral nature of containers
  • An increasing density of objects, services, and metrics within a given node
  • The focus moves to services, rather than machines
  • The consumers of monitoring data have become more diverse
  • Changes in the software development life cycle

Prometheus is an open source monitoring and alerting tool-kit which is designed to address the shifts in monitoring detailed above. As is frequently the case in the Kubernetes ecosystem, Prometheus will deployed in conjunction with Grafana as a visualization engine for the metrics extracted from the example Kubernetes environment.

 

VMware Cloud PKSTM

VMware Cloud PKSTM (CPKS) is an enterprise-grade Kubernetes-as-a-Service offering in the VMware Cloud Services portfolio that provides easy to use, secure, cost effective, and fully managed Kubernetes clusters. CPKS enables users to run containerized applications without the cost and complexity of implementing and operating Kubernetes.

 

At the heart of CPKS is VMware Smart Cluster™. Smart Cluster automates the selection of compute resources to constantly optimize resource usage, provide high availability, and reduce cost. This construct removes the need for educated guesses around cluster definition and sizing for optimal compute resources. It also enables the management of cost-effective, scalable Kubernetes clusters that are optimized for application requirements. These clusters provide built-in high availability with multi-master deployment, routine health checks and self-healing capabilities for Kubernetes clusters. CPKS allows users to run applications in a highly available environment without manual infrastructure configuration and maintenance.

 

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. It is now a standalone open source project and maintained independently of any company. Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

 

Prometheus is well suited to recording time series metrics. It fits both machine-centric monitoring as well as monitoring of highly dynamic service-oriented architectures. In a Kubernetes environment, its support for multi-dimensional data collection and querying is a particular strength. Each Prometheus server is standalone, not depending on network storage or other remote services.

 

 

Grafana

Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB. It is a commonly used open source visualization tool for time series metrics.

Visualizing metrics in Grafana revolves around the Dashboard construct. A dashboard is a set of one or more panels arranged into row(s). Each panel is the result of a query run against a data source (such as Pormetheus).

 

Helm

In the examples detailed here, both Prometheus and Grafana are deployed using the tool Helm. The Helm project describes itself as ‘The package manager for Kubernetes’ and is a Cloud Native Computing Foundation project in its own right. Helm charts represent an easy way to define, install, and upgrade Kubernetes based applications, regardless of the number of services they contain. For a more detail on Helm, please visit the project page.

 

Building and Preparing the Kubernetes Cluster

In CPKS, the process of creating a new cluster is simple and straightforward. Having already authenticated to the service, creating a new Kubernetes cluster can be easily accomplished via the command line package. In order to create the cluster used for this article, the following command was run:

$vke cluster create -n vcs-demo-cluster -r us-west-2

This command uses two flagged options. The ‘-n’ flag allows the cluster to be named (vcs-demo-cluster) and the ‘-r’ flag allows the region of the cluster deployment to be specified.

Following the successful creation of the cluster, command line access to Kuberenetes will need to be established. To do so, another command will need to be executed:

$vke cluster merge-kubectl-auth vcs-dem-cluster

 

Assuming the ‘kubectl’ command line utility is already installed for interacting with Kubernetes clusters, this command will merge the active user’s CPKS service identity with the necessary credentials to directly interface with the specified cluster (in this case ‘vcs-demo-cluster’) and change to the kubectl context to reflect that intent.

Once kubectl access has been established, the component of Helm which resides within the Kubernetes cluster (Tiller) will need to be initialized. If Helm is installed on the user’s system, the following command will accomplish this:

$helm init

 

If Helm is not currently installed, instructions to do so can be found here.

 

Installing Prometheus

As mentioned above, for the purposes of this article Prometheus was installed on a CPKS cluster via a helm chart. In the previous section, a new Kubernetes cluster was created on CPKS and the Tiller component of Helm was initialized on it. At this point, the cluster is prepared for the installation of the Prometheus Helm chart.

A repository of stable Helm charts is maintained by the Helm project in Github. For the purposes of this article, this chart was used to install Prometheus. The installation was accomplished via this command:

$helm install --name prometheus-release stable/promethues --set server.service.type=NodePort

 

This single command has several options defined in it. First, all Helm deployments require a release name. Here, the name ‘prometheus-release’ has been chosen. If no release name is specified, one will be generated based on the name of the selected chart. Next, the location of the chart itself is specified (stable/prometheus). In this case, stable is the branch within the repository, while ‘prometheus’ is the chart name.

Finally, the ‘–set’ flag is used to override one of the chart’s default values. In this example, the Kubernetes service deployment type of the Prometheus server is being changed from ‘Cluster IP’ (the default value), to ‘NodePort’. This was done in order to make the Prometheus server easily accessible from via the CPKS ingress controller. Altering the default values in a Helm chart can be accomplished via the command line in this manner, or multiple settings can be altered by referencing a ‘values.yaml’ file. To specify this file, the ‘-f’ flag is used in the ‘helm install’ command, and the default values themselves can be found in the documentation for the Helm chart in question.

Verification that the correct settings were applied may be accomplished with this command:

$ kubectl get services

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api-fitcycle NodePort 10.0.0.19 <none> 5000:30431/TCP 124d
fitcycle-mysql ClusterIP None <none> 3306/TCP 124d
grafana-release NodePort 10.0.0.68 <none> 80:30401/TCP 141d
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 141d
prometheus-release-alertmanager ClusterIP 10.0.0.165 <none> 80/TCP 141d
prometheus-release-kube-state-metrics ClusterIP None <none> 80/TCP 141d
prometheus-release-node-exporter ClusterIP None <none> 9100/TCP 141d
prometheus-release-pushgateway ClusterIP 10.0.0.159 <none> 9091/TCP 141d
prometheus-release-server NodePort 10.0.0.84 <none> 80:30402/TCP 141d
wavefront-proxy ClusterIP 10.0.0.132 <none> 2878/TCP,4242/TCP 124d
web-server NodePort 10.0.0.225 <none> 8000:30440/TCP 124d

 

In order to access the Prometheus interface, we take the output in the ‘Address:’ line of another CLI command:

$ vke cluster show vcs-demo-cluster | grep Address:
Address: vcs-demo-cluster-2ca804a0-8a4e-11e8-80fc-0a365bcc232c.bfd7a751-b2f5-43a3-b2b3-0c82d6896aee.vke-user.com

 

And combine it with the ‘NodePort’ value of the ‘prometheus-release-server’ service to access the Prometheus UI via a web browser.

 

Navigating through Prometheus ‘Status’ menu to the ‘Targets’ page, the various endpoints available from which Prometheus can scrape metrics can be viewed.

 

 

Installing Grafana

The Grafana installation in this environment was also accomplished via a Helm chart, and the chart in question is significantly more simple than the Prometheus one discussed above.

To install the Grafana chart, the following command was run:

$helm install --name grafana-release stable/grafana --set service.type=NodePort

 

Similarly to the Prometheus example, the Grafana UI can then be reached by combining the CPKS cluster ingress URL with the service NodePort value (30401).

After accessing the Grafana UI, some standard dashboards will be available, or others can be created and/or imported. One such example is an imported Kubernetes dashboard, which can be seen in Figure 4.

 

 

Conclusion

Utilizing open source monitoring tools such as Prometheus and Grafana provides a straightforward path to visibility and insight based on time series metrics. These concepts are applicable to multiple application environments, from the more traditional virtual machine or instance based domains to the rapidly expanding kubernetes platform.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *