System Monitoring

Overview

There are common reasons for system monitoring:

• Performance optimization: monitoring of CPU and memory usage can help identify and tackle important performance issues. As an example, an abnormally high CPU may be a signal for a hardware upgrade or a software optimization.
• Resource management: analyze your system demands to plan upscales and performance spikes.
• System maintenance: get actionable data in time to ensure the system meets users requirements.
• Security: monitoring of network metrics can help identify and eliminate potential security threats.

TimeBase can be configured to enable application and standard JVM metrics to monitor your system - refer to Description of Metrics to learn more. You can use Prometheus to collect and store, and Grafana to visualize them.

Description of Metrics

TimeBase Application Metric	Description	Resources Affected
timebase_storage_io_read_count	The count of read operations TimeBase carries out at a particular moment in time.	Storage
timebase_storage_io_write_count	The count of write operations TimeBase carries out at a particular moment in time.	Storage
timebase_memory_available	JVM available memory in Heap.
timebase_connections_num_connections	Number of active connections to TimeBase.	Memory, CPU
timebase_connections_traffic_rate	Average throughput of TimeBase protocol measured in bytes per second.
timebase_num_loaders	Number of active loaders (writers).	Memory, CPU
timebase_num_cursors	Number of active cursors (readers).	Memory, CPU
timebase_storage_num_active_files	Number of files in memory.	Memory
timebase_storage_num_dirty_files	Number of files in write queue waiting to be stored on disk. Common pattern: active>=dirty	Memory
timebase.license.days_valid	Number of days before license expiration

List of usable JVM Metrics

JVM Metric	Description
jvm_memory_committed_bytes	The total memory (in bytes) in the Java virtual machine runtime.
jvm_memory_used_bytes	The free memory (in bytes) in the Java virtual machine runtime.
jvm_memory_max_bytes	The total memory (in bytes) in the Java virtual machine runtime.
process_cpu_utilization	The CPU Usage (in percent) of the Java virtual machine.
process_uptime_seconds	The amount of time (in seconds) that the Java virtual machine was running.
process_files_open_files	The open file descriptor count.
jvm_gc_total	The number of garbage collection calls.
jvm_gc_interval_time_seconds_total	The total time (in seconds) that elapsed between garbage collection calls.
jvm_gc_interval_total	The total number of intervals between garbage collections calls. Per server Counter.
jvm_gc_duration_seconds_total	The total time consumed, in seconds in garbage collection.
jvm_threads_daemon_threads	The current number of live daemon threads.
jvm_threads_live_threads	The current number of live threads including both daemon and non-daemon threads.
jvm_threads_peak_threads	The peak live thread count since the Java virtual machine started or peak was reset.
jvm_classes_loaded_classes	The number of classes that are currently loaded in the Java virtual machine.
jvm_classes_unloaded_classes_total	The total number of classes unloaded since the Java virtual machine has started execution.

Step 1: Enable Metrics

To enable application and JVM metrics, configure TimeBase via Java System Properties or TimeBase admin.properties configuration file. When configured, you can access metrics via a special endpoint /tb/metrics (ex. http://localhost:port/tb/metrics) in the format compatible with Prometheus.

Java System Properties

To enable system metrics, add these variables to your docker or docker-compose:

• Version 5.6
• Version 5.5

# docker-compose.yml example

environment:
  - JAVA_OPTS=
    -DTimeBase.metrics.enable=true              # enable metrics for TimeBase
    -DTimeBase.metrics.enableJvmMetrics=true    # enable JVM metrics (disabled by default)
    -DTimeBase.metrics.enableTomcatMetrics=true # enable Tomcat metrics (disabled by default)

admin.properties

To enable system metrics, specify the following parameters in admin.properties configuration file:

   TimeBase.metrics.enable=true              # enable metrics for TimeBase
   TimeBase.metrics.enableJvmMetrics=true    # enable JVM metrics (disabled by default)
   TimeBase.metrics.enableTomcatMetrics=true # enable Tomcat metrics (disabled by default)

info

Refer to admin properties to learn more.

Kubernetes Configuration

To enable all types of metrics in Kubernetes:

Enable serviceMonitor in TimeBase values.yaml. Note, that this service is enabled by default.

# include this in TimeBase values.yaml to enable system metrics monitoring
serviceMonitor:
  enabled: true
  namespace: monitoring
  interval: "30s"
  labels:
    monitoring: application

info

Refer to Deployment in Kubernetes to learn more.

Step 2: Get Metrics

You can use Prometheus to collect and store system metrics. To periodically scrape metrics from TimeBase, add the TimeBase endpoint /tb/metrics to metrics_path section in Prometheus configuration file:

# Example of configuration for Prometheus

global:
  scrape_interval: 1s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: "external-monitor"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "timebase"

    metrics_path: /tb/metrics

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 1s

    static_configs:
      - targets: ["localhost:8011"]

info

To learn how to install and configure Prometheus, refer to Prometheus Installation and Prometheus Getting Started.

Step 3: Use Metrics

You can visualize the collected system metrics in Grafana dashboards:

1. Install Grafana
2. Install and configure Prometheus data source

3. Add and configure Grafana dashboards

Download pre-configured dashboards for JVM and TimeBase metrics

This is a sample illustration of the JVM Actuator Grafana dashboard with TimeBase JVM metrics:

Disable JVM Metrics

There are two ways to disable JVM metrics, depending on the method you used to enable them:

Java System Properties

To disable JVM metrics, add these variables to your docker or docker-compose file:

# docker-compose.yml example

environment:
  - JAVA_OPTS=
    -DTimeBase.metrics.disableJvmMetrics=true # enable or disable JVM metrics

admin.properties

To disable JVM metrics, specify the following parameters in admin.properties configuration file:

TimeBase.metricsService=QSMetricsServiceInfo
TimeBase.metricsService.disableJvmMetrics=true # enable or disable JVM metrics