Unfortunately, it can be expensive to outsource this responsibility; however, there are open-source solutions, and one of the best is the Grafana platform. Grafana is the visualization tool for a variety of data sources, but the Grafana team also has its own data sources for logs (Loki), metrics (Mimir), and traces (Tempo). In this article, we will see how to connect a Spring Boot application to this ecosystem.
Our first step is to set up our observability environment. We will use docker compose to create this environment with this docker-compose.yaml file:
version: "3.9"
networks:
telemetry:
volumes:
influxdb-storage:
grafana-storage:
services:
grafana:
image: grafana/grafana:9.3.1
depends_on:
- influxdb
volumes:
- ./docker/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
- grafana-storage:/var/lib/grafana
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin1
- GF_SERVER_HTTP_PORT=3000
- INFLUXDB_HOST=influxdb
- INFLUXDB_PORT=8086
- INFLUXDB_NAME=db0
- INFLUXDB_USER=influxuser
- INFLUXDB_PASS=influxuser1
ports:
- "3000:3000"
networks:
- telemetry
influxdb:
image: influxdb:latest
ports:
- '8086:8086'
volumes:
- influxdb-storage:/var/lib/influxdb
environment:
- INFLUXDB_URL=http://influxdb:8086
- INFLUXDB_ADMIN_USER=influxuser
- INFLUXDB_ADMIN_PASSWORD=influxuser1
loki:
image: grafana/loki:2.7.1
ports:
- "3100:3100"
command: -config.file=/etc/loki/loki.yaml
volumes:
- ./docker/loki.yaml:/etc/loki/loki.yaml
networks:
- telemetry
tempo:
image: grafana/tempo:1.5.0
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./docker/tempo.yaml:/etc/tempo.yaml
- ./data/tempo:/tmp/tempo
ports:
- "14268:14268" # jaeger ingest
- "3200:3200" # tempo
- "55680:55680" # otlp grpc
- "55681:55681" # otlp http
- "9411:9411" # zipkin
- "4318:4318" # new http
- "4317:4317" # new grpc
networks:
- telemetry
mimir:
image: grafana/mimir:2.5.0
command: "-config.file=/etc/mimir/mimir.yaml"
ports:
- "9009:9009"
volumes:
- "./docker/mimir.yaml:/etc/mimir/mimir.yaml"
- "/tmp/mimir/rules:/tmp/mimir/rules"
networks:
- telemetryHere you can see that we are spinning up a Grafana instance backed by an Influxdb along with instances of Loki, Tempo, and Mimir. We expose the appropriate ports configure each instance with the corresponding files from our docker folder. Most interesting is the grafana-datasources.yaml which tells configures grafana to connect to Loki, Mimir, and Tempo.
apiVersion: 1
datasources:
- name: Tempo
type: tempo
access: proxy
orgId: 1
url: http://tempo:3200
basicAuth: false
isDefault: true
version: 1
editable: false
apiVersion: 1
uid: tempo
- name: Loki
type: loki
access: proxy
orgId: 1
url: http://loki:3100
basicAuth: false
isDefault: false
version: 1
editable: false
apiVersion: 1
jsonData:
derivedFields:
- datasourceUid: tempo
matcherRegex: (?:traceID|trace_id)=(\w+)
name: TraceID
url: $${__value.raw}
- name: Mimir
type: prometheus
access: proxy
orgId: 1
url: http://mimir:9009/prometheus
isDefault: false
version: 1
editable: trueOnce we run “docker compose up” we can hit localhost:9000 and login to Grafana with our admin/admin1 credentials defined in the docker compose file. Now let’s work on getting some data from our Spring Boot app out there. We’ll start with adding logs to loki.
Logs
By default, Spring Boot includes slf4j so we don’t need to add anything to our build.gradle. However, in this example, we’ll use Grafana Agent to scrape our logs and export them to Loki so we need to update our application.properties file to write our logs to a file and set a root logging level like this:
logging.file.name=logs/app.log
logging.level.root=INFO
Next we add the Grafana Agent to our docker-compose.yaml file:
grafana-agent:
image: grafana/agent:v0.22.0
volumes:
- ./docker/grafana-agent.yaml:/etc/agent-config/grafana-agent.yaml
- ./logs/:/var/log/
entrypoint:
- /bin/agent
- -config.file=/etc/agent-config/grafana-agent.yaml
- -prometheus.wal-directory=/tmp/agent/wal
ports:
- "12345:12345"
networks:
- telemetry
extra_hosts:
- "host.docker.internal:host-gateway"And we tell Grafana Agent how to scrape our logs by setting its config file (grafana-agent.yaml).
server:
log_level: debug
http_listen_port: 12345
logs:
configs:
- name: default
positions:
filename: /tmp/localhost-positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- labels:
job: localhostlogs
__path__: /var/log/*log
env: "local"
app: "observability-example"Metrics
To send our metrics we need to include a few dependencies in our gradle.build
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
//enable /actuator/prometheus
runtimeOnly 'io.micrometer:micrometer-registry-prometheus'
//for timed aspect
implementation 'org.springframework:spring-aspects'
}Now let’s enable the metrics endpoints in our app by adding some properties to the application.properties:
management.endpoint.health.show-details=always management.endpoints.web.exposure.include=health,info,prometheusFinally, we tell Grafana Agent how where to read our metrics from and where to write them to by updating the grafana-agent.yaml config file:
management.endpoint.health.show-details=always management.endpoints.web.exposure.include=health,info,prometheusOur Spring Boot app acutaor endpoint contains a bunch of great metrics, if you want to see example of some custom metrics you can look at the FactorService in my example application.
Traces
To add trace data to our app we need to include a few more dependencies in our gradle.build
dependencies {
implementation("io.micrometer:micrometer-tracing")
implementation("io.micrometer:micrometer-tracing-bridge-otel")
implementation("io.opentelemetry:opentelemetry-exporter-zipkin")
}Now let’s enable the add our trace info to our logs and enable tracing and endpoints in our app by adding some properties to the application.properties:
logging.pattern.level="trace_id=%mdc{traceId} span_id=%mdc{spanId} trace_flags=%mdc{traceFlags} %p"
management.tracing.enabled=true
management.tracing.sampling.probability=1.0
management.zipkin.tracing.endpoint=http://localhost:9411That’s it! Start the app and relaunch your docker compose so that we pick up all the changes. You can find the whole example application repo here https://github.com/scottbock/observability-example
