The Power of Spring Boot Actuator and Prometheus in Enhancing Service Monitoring

What is Spring Boot Actuator & Prometheus?

Spring Boot Actuator is a powerful sub-module of Spring Boot designed for monitoring and managing your application. Let’s explore it alongside Prometheus, a monitoring solution that integrates smoothly with Spring Boot Actuator.

Spring Initializr

If you’re ready to dive into the practical aspects, you can initiate your project with Spring Initializr. For our example, we’ll use Java 17, Maven, and Eclipse. The Spring Boot version will be 3.1.3, with dependencies including Spring Web, Prometheus, and Actuator. You can generate a template for your project at Spring Initializr:

Spring Initializr

Github

For those who would rather skip the setup process, the complete project is available on GitHub for cloning here.

Starting the Application

Now if you run the application in your IDE and send a get request to localhost:8080/actuator you will see that Spring has already started the actuator and is providing health indicators via its endpoints.

Below is the response:

{
    "_links": {
        "self": {
            "href": "http://localhost:8080/actuator",
            "templated": false
        },
        "health-path": {
            "href": "http://localhost:8080/actuator/health/{*path}",
            "templated": true
        },
        "health": {
            "href": "http://localhost:8080/actuator/health",
            "templated": false
        }
    }
}

/actuator shows all open actuator endpoints including itself. There are several default actuator endpoints and they can be exposed by adding the configuration below in your application.properties file:

management.endpoints.web.exposure.include= *

Now, you’ll get a comprehensive list of endpoints when you make the same request:

{
    "_links": {
        "self": {
            "href": "http://localhost:8080/actuator",
            "templated": false
        },
        "beans": {
            "href": "http://localhost:8080/actuator/beans",
            "templated": false
        },
        "caches-cache": {
            "href": "http://localhost:8080/actuator/caches/{cache}",
            "templated": true
        },
        "caches": {
            "href": "http://localhost:8080/actuator/caches",
            "templated": false
        },
        "health": {
            "href": "http://localhost:8080/actuator/health",
            "templated": false
        },
        "health-path": {
            "href": "http://localhost:8080/actuator/health/{*path}",
            "templated": true
        },
        "info": {
            "href": "http://localhost:8080/actuator/info",
            "templated": false
        },
        "conditions": {
            "href": "http://localhost:8080/actuator/conditions",
            "templated": false
        },
        "configprops": {
            "href": "http://localhost:8080/actuator/configprops",
            "templated": false
        },
        "configprops-prefix": {
            "href": "http://localhost:8080/actuator/configprops/{prefix}",
            "templated": true
        },
        "env": {
            "href": "http://localhost:8080/actuator/env",
            "templated": false
        },
        "env-toMatch": {
            "href": "http://localhost:8080/actuator/env/{toMatch}",
            "templated": true
        },
        "loggers": {
            "href": "http://localhost:8080/actuator/loggers",
            "templated": false
        },
        "loggers-name": {
            "href": "http://localhost:8080/actuator/loggers/{name}",
            "templated": true
        },
        "heapdump": {
            "href": "http://localhost:8080/actuator/heapdump",
            "templated": false
        },
        "threaddump": {
            "href": "http://localhost:8080/actuator/threaddump",
            "templated": false
        },
        "prometheus": {
            "href": "http://localhost:8080/actuator/prometheus",
            "templated": false
        },
        "metrics": {
            "href": "http://localhost:8080/actuator/metrics",
            "templated": false
        },
        "metrics-requiredMetricName": {
            "href": "http://localhost:8080/actuator/metrics/{requiredMetricName}",
            "templated": true
        },
        "scheduledtasks": {
            "href": "http://localhost:8080/actuator/scheduledtasks",
            "templated": false
        },
        "mappings": {
            "href": "http://localhost:8080/actuator/mappings",
            "templated": false
        }
    }
}

As you can see, the actuator provides numerous endpoints right out of the box. Reviewing each individually would be quite time-consuming, so I’ll focus on examining and configuring the actuator/health endpoint, which is crucial for determining the health status of the application.

Here is the response of actuator/health endpoint:

{
    "status": "UP"
}

That response indicates that our application is healthy. The health endpoint determines the operational status of an application using various health indicators. Spring Boot Actuator offers several built-in indicators, and we also have the ability to define custom health indicators. These custom indicators can be used in conjunction with those provided by the Actuator. Next, I will demonstrate how to create a custom health indicator. But before that, let’s configure the health endpoint to display detailed health indicators along with the overall status.

Let’s open the application.properties file and add the following property:

management.endpoint.health.show-details = "always"

Now if we restart the application and send the get request to localhost:8080/actuator again, we will see the response below:

{
    "status": "UP",
    "components": {
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 499460935680,
                "free": 30051192832,
                "threshold": 10485760,
                "path": "C:\\projects\\actuator-demo-app\\.",
                "exists": true
            }
        },
        "ping": {
            "status": "UP"
        }
    }
}

Perhaps now the function of health indicators becomes more apparent. In the given response, we observe two default health indicators provided by Spring Boot Actuator: one for ping and another for disk space. If either of these indicators were to report a status of ‘DOWN,’ then the overall health status of the application would similarly be marked as ‘DOWN’.

Custom Health Indicators

The Actuator sub-module comes equipped with a wealth of default health indicators, including ping, disk space, and database validation, the latter of which is automatically added to the health endpoint if a datasource is present. Nevertheless, there are scenarios where an application’s health may hinge on the status of an external service. For such instances, Spring Boot facilitates the straightforward creation of custom health indicators.

Here is an example custom health indicator class:

import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class CustomHealthIndicator implements HealthIndicator {

 @Override
 public Health health() {
  return Health.down().withDetail("External Service", "DOWN").build();
 }
}

Now if we restart the application again and make sure CustomHealthIndicator component is running, then the response of actuator/health endpoint will become:

{
    "status": "DOWN",
    "components": {
        "custom": {
            "status": "DOWN",
            "details": {
                "External Service": "DOWN"
            }
        },
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 499460935680,
                "free": 32768573440,
                "threshold": 10485760,
                "path": "C:\\projects\\actuator-demo-app\\.",
                "exists": true
            }
        },
        "ping": {
            "status": "UP"
        }
    }
}

As demonstrated, we’ve successfully integrated a custom health indicator. Given that the status of this indicator is ‘DOWN’, it has caused the overall status to reflect the same, leading to our application being deemed unhealthy.

Configuration Examples

Endpoints exposed by actuator sub-module are easy to customize. Let me try to show some configurations to change endpoint behaviors.

With the below property, we can change the actuator prefix that we use to access endpoints from actuator/health to actuator_demo/health.

management.endpoints.web.base-path = /actuator_demo

Similarly, we can also change the name of the health endpoint to my_health by using the property below:

management.endpoints.web.path_mapping.health = my_health

We can create health indicator groups:

management.endpoint.health.group.first_group.include = ping, diskSpace
management.endpoint.health.group.second_group.include = custom

Here are the response values of localhost:8080/actuator/health/first_group and localhost:8080/actuator/health/second_group respectively:

{
    "status": "UP",
    "components": {
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 499460935680,
                "free": 29814325248,
                "threshold": 10485760,
                "path": "C:\\projects\\actuator-demo-app\\.",
                "exists": true
            }
        },
        "ping": {
            "status": "UP"
        }
    }
}

{
    "status": "DOWN",
    "components": {
        "custom": {
            "status": "DOWN",
            "details": {
                "External Service": "DOWN"
            }
        }
    }
}

So, the examples above are some configuration options. Yet, there are lots of other options to customize endpoints and actuator behavior. You may find them in Spring Boot Actuator documents.

More Details About Spring Boot Actuator

So far, we have discussed the actuator/health endpoint and health indicators, as well as their importance in monitoring an application. However, Spring Boot Actuator has much more to offer, and you can find detailed information in this document.

Prometheus

In this section, we will install and run a Prometheus server. Prometheus servers are used to collect metrics from applications and display them on their user interface. In the previous sections, we added the Prometheus dependency to our project while generating it using Spring Initializr.

<dependency>
   <groupId>io.micrometer</groupId>
   <artifactId>micrometer-registry-prometheus</artifactId>
   <scope>runtime</scope>
 </dependency>

When we add this dependency, Micrometer will generate some metric logs and feed them to Actuator. Then, Actuator will expose those logs at the /actuator/prometheus endpoint. Let's see the response from the Prometheus endpoint by sending a GET request to localhost:8080/actuator/prometheus .

# HELP jvm_compilation_time_ms_total The approximate accumulated elapsed time spent in compilation
# TYPE jvm_compilation_time_ms_total counter
jvm_compilation_time_ms_total{compiler="HotSpot 64-Bit Tiered Compilers",} 6763.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds 994.222
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 22.0
# HELP executor_queued_tasks The approximate number of tasks that are queued for execution
# TYPE executor_queued_tasks gauge
executor_queued_tasks{name="applicationTaskExecutor",} 0.0
.
.
.
.
.
.

Since the response is quite long, I have included only part of it above. Basically, the response includes default application metric logs recorded by Micrometer, and Spring Boot. We can integrate our custom metrics within the code, but first, let’s start a Prometheus server and monitor our application.”

How To Install And Run Prometheus

To install and run Prometheus, I followed these instructions up to title Getting Started. After that, I updated the prometheus.yml file inside the directory as below:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "actuator-demo-app"
    metrics_path: "/actuator/prometheus"
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    scrape_interval: 30s
    static_configs:
      - targets: ["localhost:8080"]

Now, let’s start the Prometheus server with the below command (inside the Prometheus folder we downloaded) and open localhost:9090 in our browser to have a look at its interface.

prometheus --config.file=prometheus.yml

Prometheus UI

Now, let’s open the graph view by clicking the graph button shown above. Then, type system_cpu_usage on the search bar and execute. Here is my output:

Excellent, we have now monitored our application’s metrics with remarkable speed. Beyond metrics like system_cpu_usage available at the actuator/prometheus endpoint, there's an even more significant aspect to consider: connecting to a Spring Datasource or exposing a REST service in your Spring Boot application automatically enriches the Prometheus endpoint with specific metrics for these operations. Let’s proceed by connecting to a database and implementing a REST service to observe the new metrics. First, we need to include these two dependencies in our pom.xml to establish a database connection:

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
   <groupId>com.h2database</groupId>
   <artifactId>h2</artifactId>
   <scope>runtime</scope>
</dependency>

Additionally, I’ve added Lombok dependency below to make my code cleaner with getter-setter annotations:

<dependency>
   <groupId>org.projectlombok</groupId>
   <artifactId>lombok</artifactId>
   <scope>provided</scope>
  </dependency>

Lastly, I add h2 database configurations to my application.properties file:

spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=password
spring.jpa.database-platform=org.hibernate.dialect.H2Dialect

I won’t delve into the details of the code changes. In essence, I implemented three REST endpoints: /hello, which returns 'Hello world!'; /createUser, which inserts an ActuatorUser entity into the H2 database; and /findUser, which retrieves ActuatorUser entities from the database by their ID. I integrated these endpoints into my application to demonstrate Prometheus' automatic capture of REST endpoint and repository metrics.

If you’re interested in the specific code changes, you can refer to this commit.

Now, I will call each endpoint several times, and then I will show some new metrics added to the Prometheus logs by Spring Boot.

In the image below you can see the count of the HTTP requests sent to our application. As you can see, each endpoint has its own metrics with details such as the status code, HTTP method, and even the exception class if there is any.

As I said, repository metrics are also added to the logs automatically by Spring Data JPA. Here are the metrics of the repository invocations.

By using PromSQL, we can filter metrics and create custom visuals. Here is a simple example in which count of repository invocations that use the “findById” method.

PromSQL has many features and is easy to learn. For more detail about it you may check this link.

Business Case

We have been exploring how to operate a Prometheus server and monitor metrics utilizing the capabilities of Spring Boot Actuator and Micrometer frameworks. Now, let’s delve into the impact these technologies have in a practical software environment.

The core purpose of integrating these frameworks is to enhance our service monitoring and enable informed actions based on application metrics. In my company, we recognized the need to establish actuator endpoints for detailed application monitoring, enables us to take both manual and automated actions based on real-time metrics. While we had robust monitoring systems in place, primarily leveraging API Gateway traffic logs and server pings for service availability, Spring Boot Actuator and Prometheus have taken us a step further. They empower us with the ability to harness application-generated metrics and create custom health indicators on the fly.

Consider a scenario where your company employs load balancer technologies to distribute service traffic. These technologies could be configured to cease routing requests to a service if certain metrics or health indicators fall outside acceptable ranges. For instance, traffic could be halted if the CPU usage of a service exceeds a predetermined threshold or if a service is unresponsive due to memory or disk space constraints. Load balancers typically possess filtering capabilities, and we’ve configured ours to stop directing traffic to servers whenever the load balancer’s health check endpoint returns a status other than “UP”.

Recall the configuration we implemented to establish health groups? Here is a reminder:

management.endpoint.health.group.first_group.include = ping, diskSpace
management.endpoint.health.group.second_group.include = custom

Now, let’s change group names to make them mean something:

management.endpoint.health.group.loadbalancer.include = ping, diskSpace
management.endpoint.health.group.monitor.include = custom, db, kafka

I have also added db and kafka health indicators which could be provided by frameworks or implemented manually, as demonstrated earlier.

You may wonder why we didn’t just combine all the checks into one health endpoint and use it for traffic management. In our scenario, we wanted to differentiate the endpoints. We believe that a load balancer should only cut off traffic to a web server if the server itself is unable to respond. If external services are down, the application should continue to serve and handle the situation independently. On the other hand, the monitoring endpoint is intended to verify the application’s ability to access external services and to alert the monitoring and development teams if there are connectivity issues. This is how we designed our system. Of course, as conditions change, there may be more effective ways to design such a system.

Integrating Prometheus with Grafana is quite straightforward. Although Prometheus offers a solid visual interface, it’s quite basic compared to Grafana. Grafana can be used to create more complex and informative visualizations from metrics gathered by Prometheus servers. Additionally, both Grafana and Prometheus come equipped with alert and notification features, which are user-friendly and simple to set up.

Conclusion

Today, we delved into the capabilities of Spring Boot Actuator and Prometheus, illustrating their practical applications through a real-world use case. For more detailed information, please visit the respective websites of these products:

Production-ready Features

Prometheus - Monitoring system & time series database

Thank you for staying with me so far. I look forward to connecting with you in a future article.