Java World Knowledge Sharing Portal: Monitoring SpringBoot Microservice in kubernetes using Promotheus Operator, Graphana and ServiceMonitor

Please refer to our below url old blog before proceeding. In this blog we had demo you all the belwo three aspect

1- We will configure prometheus using K8 operator
2- We will install Mongodb statefulset using helm chart. And check we are able to access MongoDb using Mongoexpress port-forward method.
3- We will install mongodb exporter using helm and service monitor. We will be scrapping Mongodb using mongodb exporter in K8 with Premetheus operator and ServiceMinotor using helm (Mongodb + Mongoexpress + MongoDb Exporter + ServiceMonitor)

http://siddharathadhumale.blogspot.com/2021/09/scrapping-mongodb-and-our-microservice.html

In this blog we are trying to execute

we will try to scrap our own Springboot microservice application using Micrometer and ServiceMonitor concept. i.e. in this blog we will try to achieve our 4th point

4- Finally we will have our own spring boot microservice using micrometer to expose our springboot metrics to prometheus. We will create docker image of this springboot application and uplaod the same on dockerhub. In this step we will not configure prometheus.yaml file to directly set our microservice url in prometheus for scrapping but we will use the concept of serviceMonitor in K8 to register our Springboot application and scrap the metrics. We will also create an helm chart for simple execution of this step.

Lets start !!!!!!

Lets first create a springboot application. Please follow below steps religiously for the same.

1

https://start.spring.io/starter.zip?name=springboot-prometheus-micrometer-servicemonitor&groupId=com.siddhu&artifactId=springboot-prometheus-micrometer-servicemonitor&version=0.0.1-SNAPSHOT&description=This+is+simple+swagger+spring+boot+example+showing+intergration+of+SpringBoot+with+Prometheus+using+ServiceMonitor&packageName=com.siddhu&type=maven-project&packaging=jar&javaVersion=11&language=java&bootVersion=2.6.0-SNAPSHOT&dependencies=prometheus&dependencies=actuator&dependencies=web

Make sure to check pom.xml having following dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
 
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <scope>runtime</scope>
</dependency>
 
<!-- https://mvnrepository.com/artifact/io.micrometer/micrometer-core -->
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-core</artifactId>            
        </dependency>

By adding above dependencies makes metrics endpoint available at /actuator/prometheus, but by default this endpoint isn’t reachable by outside services. To Enable them we have to add the values in the application.properties files, alongside the default health and metrics endpoints. You’ll also want to provide an application tag to your metrics so Grafana knows how to organize the metrics. Below, the application tag is set to match the name of the application, defined in the spring.application.name property

Also add following lines in your application.properties files

spring.application.name: springboot-prometheus-micrometer-servicemonitor
management.endpoints.web.exposure.include: health, metrics, prometheus
management.metrics.tags.application: ${spring.application.name}

Now lets add our own controller to accept the request and add following code

1- SiddhuController.java

package com.siddhu;
 
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
 
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
 
@RestController
public class SiddhuController {
 
    @GetMapping("/")
    public String index() {
        visitCounter.increment();
        return "Default page that we are showing to the users.";
    }
     
}

Now lets run the application and check this url and see if we are able to the list of metrics

http://localhost:8080/

1	`http://localhost:8080/actuator/prometheus`

Now let say we want to add some custome matrics to our appication that need to be scrapped i.e. how many time user comes and visit to our site pericualr pages etc.

For more details on custome matrics visit below sites

https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-metrics-custom

Lets add simple counter

https://docs.spring.io/spring-metrics/docs/current/public/prometheus#counters

that increas every time the page is requested. Take a look at the following RestController, which sets up a counter called userCounter and adds it to the default MeterRegistry also we increment the userCounter is every time the / endpoint is hit by the user using index() method.We also exposed the count using url /uservisit. Our final SiddhuController will be like below.

package com.siddhu;
 
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
 
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
 
@RestController
public class SiddhuController {
 
    Counter userCounter;
 
    public SiddhuController(MeterRegistry registry) {
        //registering the counter
        userCounter = Counter.builder("user_counter")
            .description("Number of times user visits to the site")
            .register(registry);
    }
 
    @GetMapping("/")
    public String index() {
        userCounter.increment();
        return "Default page that we are showing to the users.";
    }
 
    @GetMapping("/uservisit")
    public double visitCount() {
        return userCounter.count();
    }
     
}

Lets check the url is working

http://localhost:8080/uservisit

Now hit the http://localhost:8080/ and again visit the http://localhost:8080/uservisit you will find the counter value is increased along with value of user_counter in below url.

http://localhost:8080/actuator/prometheus

user_counter_total{application=”springboot-prometheus-micrometer-servicemonitor”,} 3.0

Now lets create an docker image of this springboot application so that we can use them in our Kubernetes environment.

Follow the below step religiously to create an docker file and image for this springboot application.

First run our application to create jar or war files using maven clean install command

Once the build and jar file is created successfully start creating docker files as shown below.

Create a docker file named as Dockerfile and write below code in it.

From openjdk:11
copy ./target/springboot-prometheus-micrometer-servicemonitor-0.0.1-SNAPSHOT.jar springboot-prometheus-micrometer-servicemonitor-0.0.1-SNAPSHOT.jar
CMD [“java”,”-jar”,”springboot-prometheus-micrometer-servicemonitor-0.0.1-SNAPSHOT.jar”]

Now lets build the image Make sure you have docker installed and run in your machine.I am using docker desktop in windows for the same. Start the docker desktop and execute belwo command.

use below command to create images.

docker image build -t springboot-prometheus-micrometer-servicemonitor .

execute the above docker image and check if everything is working fine by accessing belwo url

docker container run –name springboot-prometheus-micrometer-servicemonitor -p 8080:8080 -d springboot-prometheus-micrometer-servicemonitor

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>docker container run –name springboot-prometheus-micrometer-servicemonitor -p 8080:8080 -d springboot-prometheus-micrometer-servicemonitor

Check the url we are able to access it.

Now lets push the repo to the github so that we can use it in our yaml or helm chart when integrating it with prometheus operator.


C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>docker login –username shdhumale
Authenticating with existing credentials...
time="2021-09-24T18:43:26+05:30" level=error msg="(5663dc44) c49e7583-CredentialHelperPKG C<-S No response POST /registry/credstore-updated (1.0038285s): Post \"http://ipc/registry/credstore-updated\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[[STACK]]"
time="2021-09-24T18:43:27+05:30" level=error msg="(127a4525) c49e7583-CredentialHelperPKG C<-S No response POST /registry/credstore-updated (1.0023107s): Post \"http://ipc/registry/credstore-updated\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)[[STACK]]"
Login Succeeded

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>docker tag springboot-prometheus-micrometer-servicemonitor shdhumale/springboot-prometheus-micrometer-servicemonitor
 
C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>docker push shdhumale/springboot-prometheus-micrometer-servicemonitor

Now lets pull our docker images and check if it is working fine.

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>docker pull shdhumale/springboot-prometheus-micrometer-servicemonitor

and run the images using below command and check we are able to access it properly.

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>docker run -p 8080:8080 shdhumale/springboot-prometheus-micrometer-servicemonitor
 
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::       (v2.6.0-SNAPSHOT)
 
2021-09-24 14:19:48.013  INFO 1 --- [           main] theusMicrometerServicemonitorApplication : Starting SpringbootPrometheusMicrometerServicemonitorApplication v0.0.1-SNAPSHOT using Java 11.0.12 on 887220300b34 with PID 1 (/springboot-prometheus-micrometer-servicemonitor-0.0.1-SNAPSHOT.jar started by root in /)
2021-09-24 14:19:48.019  INFO 1 --- [           main] theusMicrometerServicemonitorApplication : No active profile set, falling back to default profiles: default
2021-09-24 14:19:51.212  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2021-09-24 14:19:51.240  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2021-09-24 14:19:51.241  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.53]
2021-09-24 14:19:51.358  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2021-09-24 14:19:51.359  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 3160 ms
2021-09-24 14:19:52.624  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 3 endpoint(s) beneath base path '/actuator'
2021-09-24 14:19:52.693  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2021-09-24 14:19:52.721  INFO 1 --- [           main] theusMicrometerServicemonitorApplication : Started SpringbootPrometheusMicrometerServicemonitorApplication in 5.836 seconds (JVM running for 6.895)

Now lets close the running pod and remove docker images and container using below command

http://localhost:8080/actuator/prometheus

http://localhost:8080/

http://localhost:8080/uservisit

Now lets install our prometheus in k8 env using prometheus operator using below commands.

Note we are using helm.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack –values=C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor\yamlfiles\MyValues.yaml

check all pod and svc is running properly.

Now lets check We are able to open the Prometheus and Grafana ui also.

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>kubectl port-forward service/prometheus-kube-prometheus-prometheus 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
Handling connection for 9090

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>kubectl port-forward service/prometheus-grafana 80:80
Forwarding from 127.0.0.1:80 -> 3000
Forwarding from [::1]:80 -> 3000

Default
userid:- admin
password :- prom-operator

Now lets try to configure our springboot micrometer application using yaml files as given below.

My_SpringBoot_Docker_K8.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: springboot-prometheus-micrometer-servicemonitor
  labels:
    app: springboot-prometheus-micrometer-servicemonitor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: springboot-prometheus-micrometer-servicemonitor
  template:
    metadata:
      labels:
        app: springboot-prometheus-micrometer-servicemonitor
    spec:
      containers:
      - name: springboot-prometheus-micrometer-servicemonitor
        image: shdhumale/springboot-prometheus-micrometer-servicemonitor
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
 
---
apiVersion: v1
kind: Service
metadata:
  name: springboot-prometheus-micrometer-servicemonitor-service
  labels:
    app: springboot-prometheus-micrometer-servicemonitor
spec:
  selector:
    app: springboot-prometheus-micrometer-servicemonitor
  ports:
    - protocol: TCP
      name: http-traffic
      port: 8080
      targetPort: 8080

The above creates a Deployment from the container image of the application personally build shdhumale/springboot-prometheus-micrometer-servicemonitor. Lets give the label of app: springboot-prometheus-micrometer-servicemonitor. The Service then uses that label as a selector to know how to attach to the application. The Service also receives a label of app: springboot-prometheus-micrometer-servicemonitor, which the ServiceMonitor will use to find the Service. One thing to note is that the port receives a name of http-traffic, which you can see the ServiceMonitor reference below:

Now lets create a Service Monitor yaml for our springboot micrometer and given releaes: prometheus so that it can be scraped automatically.

My_SpringBoot_Docker_K8_ServiceMonitor.yaml

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: springboot-prometheus-micrometer-servicemonitor-service-monitor
  labels:
    app: springboot-prometheus-micrometer-servicemonitor
    release: prometheus
spec:
  selector:
    matchLabels:
      app: springboot-prometheus-micrometer-servicemonitor
  endpoints:
  - port: http-traffic
    path: "/actuator/prometheus"

Here a ServiceMonitor is created, which looks for a Service with the label app: springboot-prometheus-micrometer-servicemonitor. It then defines an endpoint to use to scrape metrics, referring to the port named http-traffic and the path /actuator/prometheus, which as you saw is where Spring Boot exposes the Prometheus-formatted metrics.

Now lets execute the above files

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>kubectl apply -f C:\prometheus-operator-helm-example\My_SpringBoot_Docker_K8_ServiceMonitor.yaml
servicemonitor.monitoring.coreos.com/springboot-prometheus-micrometer-servicemonitor-service-monitor created

C:\STS-Workspace\springboot-prometheus-micrometer-servicemonitor>kubectl apply -f C:\prometheus-operator-helm-example\My_SpringBoot_Docker_K8.yaml
deployment.apps/springboot-prometheus-micrometer-servicemonitor created
service/springboot-prometheus-micrometer-servicemonitor-service created

check all pod and svc are running properly