TimeBase functionality highlights: overview of all key functionality.
Click Tag to Display Pages: technology tbd

TickDb

TBD

TickStream

TBD

TickCursor

TBD

TickLoader

TBD

RecordClassDescriptor

TBD

Data Distribution

TBD

Live Distribution

TBD

Dynamic Subscription Change

TimeBase allows clients to subscribe to live data based on types, streams and symbols (entities) attributes, and change their subscription at any time.

There are three possible scenarios:

  • Basic

Subscription starts at some point of time (close to real-time) when the command reaches the server.

This scenario may be applied in case precise subscription timing and order is not critical.


// change entity entity
cursor.removeEntity(new ConstantIdentityKey("CLX20"));
cursor.addEntity(new ConstantIdentityKey("CLZ20"));
// add descriptor
cursor.addTypes("epam.timebase.samples.SensorType1");
// add stream
TickStream stream = Arrays.stream(tickDB.listStreams())
    .filter(item -> "SmartMetersOhio".equals(item.getKey()))
    .findFirst()
    .orElseThrow(() -> new IllegalStateException("Stream was not found"));
cursor.addStream(stream);

  • Precise Subscription Timing

This scenario may be applied in case it is critical to set the precise timing of a new subscription.

In this case, the subscribed data arrives precisely at the specified time but there is no guarantee as for the data ordering.


// specify time for newly subscribed data
cursor.setTimeForNewSubscriptions(Instant.now().toEpochMilli());
cursor.addEntity(new ConstantIdentityKey("TrafficLight-4419"));

  • Complete Reset

This scenario may be applied in case precise subscription timing and order are required.

It is slowest case due to the complete reset of all buffers.

Reset is performed at a specified time, starting from which you will receive data with the precise time and order.


// reset cursor
cursor.clearAllEntities();
String[] symbolsToAdd = new String[] {"Meter-1", "Meter-2", "Meter-3"};
cursor.addEntities(
    Arrays.stream(symbolsToAdd)
        .map(item -> new ConstantIdentityKey(item))
        .toArray(IdentityKey[]::new), 0, symbolsToAdd.length);
cursor.reset(Instant.parse("2020-08-04T10:12:35Z").toEpochMilli());

Internet Streaming

TimeBase is ideally suited for distributing data over the Internet.

When a TimeBase client connects to the server, it automatically performs a connection speed test (similar to a ping test, but over TCP).

By measuring the latency of a round-trip, TimeBase is able to detect whether the connection is over a LAN or a WAN/Internet.

For wide area networks, TimeBase will heavily compress non-live data before transmitting it in either direction.

Additionally, TimeBase supports multi-regional load balancing and failover.

Example

TimeBase servers with identical data are installed in multiple geographic regions.

TimeBase client supports a special mode, in which instead of pointing to a specific server, you can point it to a simple Website that provides a list of available servers.

When establishing connection, the TimeBase client will contact all available servers, and evaluate connection latency to each of them.

Based on this data, TimeBase client will detect which servers are located in the geographic region closest to the client, and, out of those servers, select the one that is least loaded with requests at connection time.

Replication

The goal of Replication is to incrementally propagate all changes from one stream (master) to the other (slave).

It is supported and quite common to have stream S1 be replicated from TimeBase instance A to instance B, while stream S2 is replicated from B to A.

Replication monitors all changes made to a specific stream (like purge, truncation) and modifies replication target inline with changes.

  • Replication is performed based on a per stream principle: it is not possible to replicate two streams into one and the other way around.
  • Symbol is a minimal replication unit. Symbol is a key that describes time-series (e.g. IoT sensor id)
  • Replication is realized in a form of a script written for TickDbShell with all the necessary logic and configurations.
  • Replication is supported for all symbols and streams
  • TimeBase replication propagates all types of changes, including schema migration.
  • Replication may be configured to dynamically support new symbols written to the source stream

Fixed and Dynamic Replication

  • In Fixed mode, replication addresses only a fixed list of keys (symbols) (replicate only symbol A and B)
  • In Dynamic mode, replication may dynamically pick-up and replicate new keys that are being written to the source stream

Fail-Overs Support

Replication supports fail-over and automatic connection restoring.

Backup and Restore

TimeBase Replication supports data Backup and Restore.

  • Stream may be replicated to data files - Backup
  • Data files can be imported back to TimeBase - Restore

Live Replication Mode

The Live mode maintains an open connection between master and slave, where all inserts, updates and deletes are propagated right away.

Live Replication mode applies to all Transient and Durable streams (e.g. from Durable to Transient, from Transient to Transient, from Durable to Durable, from Transient to Durable).

In case Transient stream is involved - it is a Live replication.

Batch Replication Mode

The Batch mode is used to replicate data during off hours.

When replication is run in the Batch mode, it downloads all available changes from the master, closes connection, and stops.

Batch Replication applies to Durable streams (e.g. from Durable to Durable)

Custom Replication

ClickHouse

TBD

S3

TBD

Timescale

TBD

Monitoring

ZBX

TimeBase metrics can be monitored via JMX using products like Zabbix/Prometheus

SNMP

To Be Removed??

TimeBase comes with an embedded SNMP agent. This agent is disabled by default, but can be easily enabled. It consumes little runtime resources, when enabled.

The **TimeBase SNMP Agent **allows server monitoring from a multitude of available standard SNMP monitoring tools, including NET-SNMP.

TimeBase comes with a MIB file, which can be viewed or downloaded from a running TimeBase instance simply by pointing a Web browser to the TimeBase host and port (and appending /QuantServer.mib to the URI.

TimeBase also comes with a simple SNMP monitoring and alerting application, called QuantServer Monitor. QuantServer Monitor can be easily used for monitoring TimeBase, but its use is not required. QuantServer Monitor provides browser-based user interface.

Grafana

Lear more about TimeBase Grafana Integration.

Security

  • UAC – User Authentication and Authorization
    • Users, Groups (LDAP/ActiveDirectory)
    • Permission Rules: { ALLOW/DENY, Principal, Operation, Resource }
  • DAC – Data Access Control (per-message)
    • Implemented for a few data vendors/clients to meet data distribution requirements (Bloomberg, Reuters)
  • SSL – Data in Transit Encryption (Considerable CPU penalty, recommended for WAN clients only)
  • SSO - Single Sign-On for Web Application
  • Development follows OWASP top 10 standard:
    • Spotbugs (with security plugin)
    • Dependency Check (scan of third-party dependencies against OWASP vulnerabilities database)

Query Language

TimeBase supports a query language, called QuantQuery Language (QQL).

QQL is designed to be, on one hand, similar to Standard SQL, and, on the other hand, to support queries on polymorphic time-series data.

QQL allows users to query data via the TimeBase API, the command-line interface, or the graphical user interface.

Usage Examples:

  • Create or modify streams in TimeBase
  • Query messages
  • Select specific data elements to query
  • Filter messages based on Boolean conditions (using the WHERE clause)
  • Select distinct values
  • Compute aggregate functions, such as MIN or MAX