Materialized Views

Overview

A materialized view is an object that contains the result of a QQL query. In TimeBase, we call it a View. The basic principle is that you can extract data via a QQL query from one stream and save it in a dedicated stream (or send via topic), which will get updated along with the data in the source stream.

You can use Views to have custom data selections or data transformations from TimeBase streams. By creating a View, you basically pre-process your query once and use it afterwards as a ready dataset, which may significantly reduce processing time, unlike running the same query over and over again every time you need it.

As a result, you can keep your data extract in a dedicated stream and work with it the same way you work with any other TimeBase stream in the Web Admin application or via API.

Design Principles

Lifecycle

Views acquire different states throughout their lifecycle.

State	Description
CREATED	A newly created View acquires this state after initialization.
PROCESSING	This state indicates that the query is currently being processed by the server.
IDLE	A View acquires this state when it is not in PROCESSING and is waiting for updates in the source stream.
FAILED	A View acquires this state when an error occurs in the source stream or while processing the QQL query. This is a final state; the View can leave it only after a restart.
COMPLETED	A View that does not draw data from a stream acquires this state once the QQL query execution is finalized. This is a final state; the View can leave it only after a restart.
RESTARTED	An intermediate state indicating that the View has been restarted. It is used to mark that the restart operation has occurred.
REMOVED	Deleted Views acquire this state.

Materialized View state transition diagram:

View Execution Environment

Starting from TimeBase 5.6.191, Materialized Views can be executed directly inside the TimeBase server.

In earlier versions, Views were executed only within the WebAdmin service. Beginning with this version, View processing can be handled by the TimeBase server itself.

To enable View processing in the TimeBase server use following:

• TimeBase server Java system property:

-DTimeBase.view.enable=true

• Or TimeBase server admin.properties file

TimeBase.views.enable=true

To avoid conflicts between TimeBase and WebAdmin processors, set TIMEBASE views processor property in WebAdmin:

• WebAdmin Java system property

-Dviews.processor=TIMEBASE

• Or WebAdmin application.yaml:

views:
  processor: "TIMEBASE"

• or WebAdmin env variable:

VIEWS_PROCESSOR=TIMEBASE

This option configures WebAdmin to delegate View processing to the TimeBase server instead of running View processors locally.

Data Handling

Views operate in a so-called live mode when they query data from TimeBase streams. In this case, new data (QQL query output) is added to the target stream of the View with every new message that comes to the source stream. Naming for target streams uses the following pattern: "VIEW_NAME#view#".

info

A View gets restarted and the target stream is rewritten completely every time a source stream is truncated or its schema changes.

A View operates in non-live mode when it processes only the historical data available in the source stream. In this mode, the QQL query reads existing messages from the source stream, writes the results to the target stream, and stops once all available data has been processed. After processing completes, the View transitions to the COMPLETED state.

View Restarts Handling

During a restart, the View attempts to preserve data integrity and continue processing from the point where it previously stopped. In supported cases, the View restores its state and resumes writing results from last written timestamp without truncation previously written data.

However, some query structures require special handling during restart:

• If the query contains an over time clause, the View resumes reading data from the beginning of the corresponding interval to ensure correct window calculations.
• If the query contains a group by expression that is not based on symbol, the View performs a truncate operation at the last processed timestamp in the target stream before continuing processing.

Queries Requiring Full Restart

Some query structures cannot guarantee that their results can be correctly restored after a restart. This applies to QQL queries that:

• use trigger over time
• compute aggregate functions over a window larger than the over time interval

Example: select sum{}(value) trigger over time (1m).

For such queries, it is recommended to create Materialized Views in auto-restart mode. In this mode, the View always restarts processing from the beginning of the source data to guarantee correct results.

Processing Views

Each View requires a dedicated set of TimeBase server resources in order to process data from the source stream and produce query results.

For reading input data, the View opens its own cursor on the source stream. For writing results, the View creates a loader for the target stream. In addition, each View requires a separate execution thread to run the QQL processing logic.

As a result, the number of active Views directly affects the number of open cursors, loaders, and processing threads used by the system.

tip

Keep in mind that the number of active views directly affects the performance of the entire application.

For Materialized Views running in TimeBase WebAdmin, you can configure the server to assign dedicated threads to process Views. Workers that process Views are evenly distributed among all the assigned threads.

There are two ways to configure dedicated threads to process Views:

• Add the JAVA system property: -Dviews.processor.thread-pool-size=8
• Configure the application.yaml:

views# Metadata Stream

TimeBase maintains a dedicated system stream named views# that stores metadata for all Materialized Views. Each record in this stream captures information about a View at a point in time, including its name, current status, output stream name, QQL query, and other properties.

A new message is written to views# every time a View, is created, removed, changes state (e.g., transitions from IDLE to PROCESSING) or any other metadata update.

The views# stream is configured as a unique stream, which means TimeBase maintains a current-state snapshot of the latest record for each View. In addition, periodic snapshots of the full current views metadata are written to views# at a configurable interval.

The change history is retained for a configurable period; older records are purged automatically.

Recovery After Restart

When the Views processor starts, it reads the views# stream to restore the last saved snapshot and resume all Views from their previous states.

If the schema of the views# stream has changed (e.g., after a TimeBase upgrade), the processor runs a migration procedure on startup: it updates the stream schema and preserves the old data in a backup stream named views#backup#.

If the migrated data appears corrupted, you can enable recovery from the backup stream by setting the following system property:

-DTimeBase.views.recoverFromBackup=true

With this option, the processor reads both views# and views#backup# when restoring the snapshot on startup.

Creating Views

Views can be created in two ways:

• QQL — use CREATE VIEW and related DDL statements directly in a query. See Materialized Views in the QQL reference.
• Web Admin — use the visual interface to create, manage, and monitor Views. See Working with Views in the Web Admin User Guide.