Learn about the VMware Spring Cloud Data Flow for Kubernetes Integration.

This page provides an overview of what you can do with the VMware Spring Cloud Data Flow for Kubernetes integration. The documentation pages only for a limited number of integrations contain the setup steps and instructions. If you do not see the setup steps here, navigate to the Operations for Applications GUI. The detailed instructions for setting up and configuring all integrations, including the VMware Spring Cloud Data Flow for Kubernetes integration are on the Setup tab of the integration.

  1. Log in to your Operations for Applications instance.
  2. Click Integrations on the toolbar, search for and click the VMware Spring Cloud Data Flow for Kubernetes tile.
  3. Click the Setup tab and you will see the most recent and up-to-date instructions.

Spring Cloud Data Flow Integration

Wavefront provides a comprehensive solution for monitoring Spring Cloud Data Flow (SCDF). This integration uses the Micrometer Wavefront registry to collect detailed metrics from SCDF server as well as the Stream and Task data pipelines managed by the SCDF servers.

The following diagram illustrates the Spring Cloud Data Flow metrics collection architecture:

images/scdf_wavefront_architecture.png

The Micrometer instrumentation library powers the delivery of application metrics from Spring Boot, including metrics for message rates and errors, which are critical to the monitoring of deployed event streaming and batch data pipelines in Spring Cloud Data Flow.

For Streaming data pipelines that use Kafka message binder, the native Apache Kafka Client metrics are collected and plotted in a dedicated dashboard:

images/scdf_kafka_stream_metrics_architecture.png

Servers Monitoring

All Spring Cloud Data Flow and the Spring Cloud Skipper are instrumented for Wavefront metrics collection.
This dashboard provides real-time visibility into the Spring Cloud Data Flow and Spring Cloud Skipper servers.

The Spring Cloud Stream applications add an additional application metrics tag, that allows metrics aggregation by server type (SCDF or Skipper):

  • application: The name of the Server (SCDF or Skipper applications name) to show metrics for.

Streams Monitoring

All Spring Cloud Stream Applications are instrumented for Wavefront metrics collection.

The Spring Cloud Stream applications add several, stream specific tags (below), that allow metrics aggregation by application type, instance, stream name, and so on:

  • stream.name: The name of the Stream that contains the applications emitting the metrics.
  • application.name: The name (or the label) of the application within the Stream emitting the metrics.
  • application.type: The application role (e.g. source, processor, or sink) within the Stream emitting the metrics.
  • application.guid: Unique application instance identifier. Every application.name can have multiple instance.
  • application.index: Application instance ID (when available).

Tasks Monitoring

The integration supports monitoring of Task applications that were deployed as part of a Task definition in Data Flow.

The Spring Cloud Task applications add several task-specific tags that allow metric aggregation by application type, instance id or task name:

  • task.name: The name of the Task application that emits the metrics.
  • task.execution.id: The instance id of the executed task. A task can be executed many times with the same name but a different execution ID.
  • task.external.execution.id: Task identifier inside the external platforms (e.g. Cloud Foundry or Kubernetes) where the task is run.
  • task.parent.execution.id: If a task is run by another task the parent id is used to identify the task run it.

Dashboards

In addition to setting up the metrics flow, this integration also installs dashboards:

  • Spring Cloud Data Flow and Skipper Servers: Real-time visibility into the Spring Cloud Data Flow and Spring Cloud Skipper servers.
  • Spring Cloud Data Flow Streams Summary: Performance overview for all event streaming data pipelines deployed by DataFlow. One can compare the average performance per stream, CPU, memory, message throughput, latency, and other metrics.
  • Spring Cloud Data Flow Stream Applications: A detailed real-time performance report of all stream applications that are part of a single event streaming data pipeline. One can filter down metrics for a particular application, instance, or channel.
  • Spring Cloud Data Flow Kafka Stream Applications: A detailed real-time performance report of all Kafka stream applications that are part of a single event streaming data pipeline. One can filter down metrics for a particular application, instance, or channel.
  • Spring Cloud Data Flow Task Applications: A detailed real-time performance report for all Task applications.

Here’s a preview of the Spring Cloud Data Flow and Skipper Server dashboard:

images/scdf_servers.png

Here’s a preview of the Spring Cloud Data Flow Stream Summary dashboard:

images/scdf_streams.png

Here’s a preview of the Spring Cloud Data Flow Stream applications dashboard:

images/scdf_applications.png

Here’s a preview of the Spring Cloud Data Flow Kafka Stream applications dashboard:

images/scdf_kafka_applications.png

Here’s a preview of the Spring Cloud Data Flow Task applications dashboard:

images/scdf_tasks.png

Spring Cloud Data Flow Metrics

Spring Cloud Data Flow’s generic performance metrics are based on Micrometer, and are registered in Micrometer’s registry with the spring.cloud.dataflow prefix.

The following table explains all the metrics in details:

Auto-configuration enables the instrumentation of requests handled by Spring MVC. When management.metrics.web.server.request.autotime.enabled is true, this instrumentation occurs for all requests.

Metric Name Description
spring.cloud.dataflow.server.* Metrics for all requests handled by SCDF’s Spring MVC application. Statistics: avg, count, max, sum.

Spring Integration Metrics

Spring Integration Metrics Documentation

Spring Integration registers micrometer timers for each MessageHandler and MessageChannel and a counter for each MessageSource. All metrics provided by the framework are registered in Micrometer’s global registry under the spring.integration prefix. The following table explains all the metrics in details:

Metric Name Description
spring.integration.receive (type=source) Message sources messages received.
spring.integration.receive (type=channel) Messages received on pollable message channels.
spring.integration.send (type=handler) Message handlers’ send processing time.
spring.integration.send (type=channel) Message channels’ send processing time.
spring.integration.channels Number of MessageChannels in the application.
spring.integration.handlers Number of MessageHandlers in the application.
spring.integration.sources Number of MessageSources in the application.

Kafka Client Metrics

Applicable for all Spring Cloud Stream (SCS) applications configured with Kafka binder. The Spring Kafka framework, used internally by SCS, provides micrometer Kafka Client metrics. Later expose Apache Kafka native Producers, Consumers and Streams metrics.

Kafka Records

The Record stand for a single Message exchanged between the Producer and the Consumer applications using the Kafka Brokers.

Metric Name Description
kafka.producer.record.send.rate The average number of records sent per second for a topic.
kafka.consumer.fetch.manager.records.consumed.rate Average number of records consumed per second for a specific topic or across all topics.
kafka.producer.record.size.* Size of the records sent per second for a topic: avg, max.
kafka.producer.record.error.rate Average record sends per second that result in errors.
kafka.producer.record.retry.rate Average number of re-tried record sends per-second.
kafka_consumer.fetch.manager.records.lag.* Number of messages consumer is behind producer, either for a specific partition or across all partitions on this client: avg, max.

Kafka Producer

Producers’ send request represents a single interaction between a Producer application and Kafka Broker. To exchange one Record (e.g. message) usually, multiple requests are performed between the producer and the brokers.

Metric Name Description
kafka.producer.request.rate The average number of requests sent per second to the broker.
kafka.producer.response.rate The average number of responses received per second.
kafka.producer.request.latency.* The request latency in ms: avg, max.
kafka.producer.io.wait.time.ns.avg The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds.
kafka.producer.io.wait.ratio The fraction of time the I/O thread spent waiting.
kafka.producer.network.io.rate The average number per second of network operations, reads or writes, on all connections.
kafka.producer.compression.rate.avg The ratio of data compression in the batches of data the producer sends to the broker. A higher compression rate indicates greater efficiency.
kafka.producer.batch.size.avg Average number of bytes sent per partition per request (e.g. data size send to different partition on the topic).
kafka.producer.outgoing.byte.rate The average number of outgoing bytes sent per second to all servers - e.g. the producer network throughput.
kafka.producer.requests.in.flight Current number of outstanding requests awaiting a response.
kafka.spring.cloud.stream.binder.kafka.offsetproducer.waiting.threads Number of user threads blocked waiting for buffer memory to enqueue their records.

Kafka Consumer

Consumer fetch request represents a single interaction between a Kafka Broker and a Consumer application. Retrieving a single Record (e.g. message) may involve multiple fetch requests.

Metric Name Description
kafka.consumer.fetch.manager.fetch.rate Number of fetch requests per second from the consumer.
kafka.consumer.fetch.manager.fetch.latency.* Time taken for any fetch request: avg, max.
kafka.consumer.fetch.manager.bytes.consumed.rate Average number of bytes consumed per second for a specific topic or across all topics.

Kafka Stream - Thread

Metric Name Description
kafka.stream.thread.[commit or poll or process or punctuate].rate The average number of respective operations per second across all tasks.
kafka.stream.thread.[commit or poll or process or punctuate].latency.avg The average execution time in ms, for the respective operation, across all running tasks of this thread.
kafka.stream.thread.task.created.rate The average number of newly created tasks per second.
kafka.stream.thread.task.closed.rate The average number of tasks closed per second.

Kafka Stream - Task & Process Node

The metrics are only available if the recording level (e.g. metrics.recording.level configuration option) is set to debug.

Metric Name Description
kafka.stream.task.[commit or process].rate The average number of respective operations per second across all tasks.
kafka.stream.task.[commit or process].latency.avg The average execution time in ns, for the respective operation for this task.
kafka.stream.task.dropped.records.rate The average number of records dropped within this task.
kafka.stream.task.record.lateness.* The observed lateness (stream time - record timestamp) for this task: avg, max.
kafka.stream.task.enforced.processing.rate The average number of enforced processings per second for this task.
kafka.stream.processor.node.process.rate The average number of records processed per second by a source node.

Spring Batch Metrics

Spring Batch Metrics Documentation

Spring Batch provides support for monitoring batch jobs using Micrometer. All metrics provided by the framework are registered in Micrometer’s global registry under the spring.batch prefix. The following table explains all the metrics in details:

Metric Name Description
spring.batch.job (tags: name, status) Duration of job execution.
spring.batch.job.active Currently active jobs.
spring.batch.step (tags: job.name, name, status) Duration of step execution.
spring.batch.item.read (tags: job.name, step.name, status) Duration of item reading.
spring.batch.item.process (tags: job.name, step.name, status) Duration of item processing.
spring.batch.chunk.write Duration of chunk writing.

Spring Cloud Task Metrics

Spring Cloud Task Monitoring Documentation

Spring Cloud Task provides support for monitoring batch jobs and short-lived task applications using Micrometer. All metrics provided by the framework are registered in Micrometer’s global registry under the spring.cloud.task prefix. The following table explains all the metrics in details:

Metric Name Description
spring.cloud.task Duration of Task execution.
spring.cloud.task.active Records the run-time status of long-time lasting tasks.

Spring Boot Metrics

Spring Boot Metrics Documentation

Spring Boot registers the following core metrics when applicable:

JVM metrics

Reports metrics for memory and buffer pools, garbage collection statistics, threads utilization, class loaders.

Metric Name Description
jvm.buffer.count Number of buffer pools.
jvm.buffer.memory.used Used buffer pools memory.
jvm.buffer.total.capacity Buffer pools total capacity.
jvm.classes.loaded Number of classes loaded.
jvm.classes.unloaded Number of classes unloaded.
jvm.gc.live.data.size The live data size is the size (in bytes) of the old generation after a major garbage collection.
jvm.gc.max.data.size The maximum size of long-lived heap memory pool for the old generation (in bytes).
jvm.gc.memory.allocated Increase in the size of the young heap memory pool after one garbage collection and before the next.
jvm.gc.memory.promoted Count of positive increases in the size of the old generation memory pool from before garbage collection to after garbage collection.
jvm.gc.pause.* Garbage collection pauses Statistics: avg, count, max, sum.
jvm.memory.committed JVM committed memory.
jvm.memory.max JMV max available memory.
jvm.memory.used JMV used available memory.
jvm.threads.daemon Current number of live daemon threads in this JVM.
jvm.threads.live Current number of live threads in this JVM.
jvm.threads.peak The peak number of threads in this JVM.
jvm.threads.states Reports threads states.

CPU metrics

Metric Name Description
process.cpu.usage Percentage of CPU usage.

File descriptor metrics

Metric Name Description
process.files.max Maximum allowed file descriptors count.
process.files.open Open file descriptors count.

Log4j2 and Logback metrics

Records the number of events logged to Log4j2 and Logback at each level.

Metric Name Description
logback.events Number of events logged to Log4j2 and Logback at each level.

Uptime metrics

Reports a gauge for uptime and a fixed gauge representing the application’s absolute start time.

Metric Name Description
process.start.time Fixed gauge representing the application’s absolute start time.
process.uptime Gauge representing the application’s uptime.

Spring MVC Metrics

Auto-configuration enables the instrumentation of requests handled by Spring MVC Metrics. When management.metrics.web.server.request.autotime.enabled is true, this instrumentation occurs for all requests.

Metric Name Description
http.server.requests.* Statistics: avg, count, max, sum.

Tomcat metrics

The server.tomcat.mbeanregistry.enabled must be set to true for all Tomcat metrics to be registered.

Metric Name Description
tomcat.sessions.active.current Number of Tomcat active sessions.
tomcat.sessions.active.max Maximum number of active Tomcat sessions.
tomcat.sessions.alive.max Duration of the maximum Tomcat active sessions.
tomcat.sessions.created Number of sessions created by Tomcat.
tomcat.sessions.expired Number of expired Tomcat sessions.
tomcat.sessions.rejected Number of sessions rejected after exceeding the maximum session configuration.