TimeBase Shell CLI overview and user tutorials
Click Tag to Display Pages: development tools

Launch

Execute tickdb script to open shell console.

Case Sensitivity

==> set STREAM Coinbase
set STREAM: unrecognized option. (Type ? for usage)
==> SET stream Coinbase
Stream 'Coinbase' was not found.
==> SET stream COINBASE
COINBASE

Commands used by TimeBase Shell are case-insensitive. However names of parameters and stream names, symbols used by streams are case-sensitive:

Help

==> ?
tickdb Shell Commands:
*******************************************************************************
Note: it is often convenient to specify the set db and open commands
on the command line:
…

Enter question mark ? command to display multiple screens of TimeBase shell help.

Exit

You can exit TimeBase Shell using exit or quit command.

Set Commands

TimeBase Shell has SET command. This command controls shell operating options described in the sections below.

Global Options


> Max Number of Messages 

set max <n>

set max 20

> Time Zones 

set tz <time zone>

set tz GMT
set tz America/New_York

> Time Format 

set timeformat <format>

set timeformat yyyy-MM-dd HH:mm:ss

  • Set maximum number of messages printed at once (page size).
  • Set time zone for parsing and displaying time.
  • Set time format for parsing and displaying time stamps.

DB Management Commands

Set location of TimeBase using URL or path.

Normally in most cases you connect by passing TimeBase server URL.

TimeBase URL must start with dxtick:// followed by host and port.

Example: set db dxtick://localhost:8011

This command has extended form that allows to specify user credentials when TimeBase has user access control enabled.

See also set user command: set db dxtick://user:password@localhost:8011.

Here is an example:

set db dxtick://localhost:8011
open

If TimeBase server is not running, TimeBase Shell can also open TimeBase data storage directory (and launch mini-TimeBase server inside this tool).

In this case instead of URL you need to specify TimeBase directory path:

set db /deltix/timebase
open

User Credentials (UAC)

This command is used to pass user credentials to TimeBase connection (when User Access Control is enabled): set user <user> <password>

Here is an example: set user jdoe WTdRE*rvDU*C

Main Commands

Open Database


> Open Database

open //open database

open ro //read-only connection

set db dxtick://localhost:8011 //full example 
open ro

This command opens connection to TimeBase server.

If you want to open a read-only connection add ro parameter.

List Streams

You can see a list of streams defined in TimeBase using streams command.

View Stream Schema

> Describe Command Example 


==> desc balances
DURABLE STREAM "balances" (
    CLASS "deltix.samples.uhf.balance.BalanceMessage" (
        "account" VARCHAR,
        "amount" FLOAT,
        "destination" VARCHAR
    );
)
OPTIONS (FIXEDTYPE; PERIODICITY = 'IRREGULAR'; DF = 1; HIGHAVAILABILITY = FALSE)
COMMENT 'Auto-created by Ember'

You can inspect data schema of currently selected stream using desc (describe) command.

Close Database

Close command gracefully closes connection from shell to TimeBase server: close.

Select Current Stream

Once you opened TimeBase connection the next task is usually selecting which streams you will be working with:

set stream bars

Use stream keys to identify streams.

You can select multiple streams. Use separated by spaces or commas or semicolons:

set stream bbo, trades

View Data

==> set stream balances
balances
==> select
InstrumentMessage,CUSTOM,BTC,2020-01-07 14:45:33.063,1578426333063
InstrumentMessage,CUSTOM,USD,2020-01-07 14:45:33.063,1578426333063
InstrumentMessage,CUSTOM,LTC,2020-01-07 14:45:33.063,1578426333063
InstrumentMessage,CUSTOM,ETH,2020-01-07 14:45:33.065,1578426333065


> Monitor 

==> set stream COINBASE
COINBASE
==> monitor
Monitoring COINBASE; hit <Enter> to abort ...
PackageHeader,FX,XRP/BTC,2020-03-12 08:50:24.481,1584017424481,999
PackageHeader,FX,BTC/USD,2020-03-12 08:50:24.481,1584017424481,999
PackageHeader,FX,BTC/USD,2020-03-12 08:50:24.481,1584017424481,999
PackageHeader,FX,XRP/USD,2020-03-12 08:50:24.483,1584017424483,999
PackageHeader,FX,LTC/USD,2020-03-12 08:50:24.483,1584017424483,999

STOP CURRENT OPERATION (y,n) ? ==> y

You can view instruments collected by selected stream using entities command.

You can view data inside selected stream using select command. This command reads data from the beginning of stream.

If you want to see live messages recorded into the stream at current time use monitor command: monitor

Monitor command continuously displays newly recorded messages. You can hit Enter key to interrupt it.

Data Copy


> Data Copy Example 

set stream apple-bars
symbols add EQUITY AAPL
copyfrom bars

This command copies all data from the specified streams to the current stream, using the currently set time range and current filter.

Filters

Filters allow filtering data by symbol and time.

Symbol Filters

> Examples:

symbols add EQUITY AAPL
symbols add EQUITY MSF
symbols remove FUTURE CLZ12

Symbol filter commands:

  • symbols all - Subscribe for all symbols
  • symbols show - Show subscribed (filtered) symbols
  • symbols clear - Clear subscribed symbol from the filter
  • symbols add <type> <symbol> - Add specified symbol to the filter
  • symbols remove <type> <symbol> - Remove the specified symbol from the filter

Time Filters

> Example:

set stream bbo
set tz GMT
set timeformat yyyy-MM-dd HH:mm
set time 2009-01-02 15:00
set endtime 2009-01-02 16:00

Set start and end/time for data selection operations.

Default timestamp format is yyyy-MM-dd HH:mm(can be changed using timestamp option):

  • set time <text> - Set selection time by parsing text (text must match the current format precisely)
  • set endtime <text> - Set selection end time by parsing text (text must match the current format precisely)

Import From Data File Into Defined Stream

> Example:

open rw
set src c:\dailyBars.qsmsg.gz
set stream dailyBars
set tz GMT
import

You can import data from data file into the currently selected stream using import command.

Special set parameter src is used to select path to source data file (usually in *.qsmsg or *.qsmsg.gz format)

Export Data To Data File


set db dxtick://localhost:8011
open
set stream trades
export d:\temp\exp.qsmsg.gz


set stream dailyBars
symbols all
symbols add FUTURE CL12 
symbols add FUTURE NG12
symbols show
export c:\future100.qsmsg.gz

You can export data from selected streams into data file using export command.

Destination file is passed as an argument of export command.

Export will respect selected filters. The attached example exports data only for futures CL12 and NG12.

Delete Stream

set stream ticks
delete

TimeBase shell has delete command that completely removes currently selected stream.

In this and similar cases database must be open in read-write mode.

Purge Stream (Delete Before)

set stream ticks
purge 2009-01-01 09:30:00.00

Purge operation removes data before specified timestamp (not including the specified time).

Truncate Stream (Delete After)

set stream ticks
truncate 2009-01-01 09:30:00.00

Truncate operation removes data after specified timestamp.

In this and similar cases database must be open in read-write mode.

Trim Stream

For each stream TimeBase may pre-allocate space for future growth.

For example, truncate operation simply moves last used position marker while keeping space occupied by truncated data.

Sometimes this behavior is undesirable. For example, you may need to free as much space as possible for other streams.

TimeBase shell has special trim command to trim currently selected stream to actually used size.

In this and similar cases database must be open in read-write mode.

Delete Data In The Middle


delete [start];[end]   

> Delete time range from selected streams:

delete 2009-01-02 09:31:00.0;2009-01-02 09:59:00.0

> Example: Delete all data before specified date:

delete ;2009-01-02 09:59:00.0

> Example: Delete all data older than specified date

delete 2009-01-02 09:31:00.0;

It is possible to remove data in the middle of the stream.

This command takes start and end of time range to be deleted as an argument (inclusive).

Rename Stream

set stream GDAX
rename COINBASE

This command allows to rename currently selected stream.

Lock/Unlock Stream


> Example 1: 

set stream securities
lock
import
unlock

> Example 2:

set stream ticks
lock r
select
unlock

TimeBase supports per-stream read/write locks.

At any moment of time only one client connection can acquire a write lock.

Multiple connections can acquire read locks (and prevent a concurrent write).

Syntax of locking commands is the following:

  • lock [r|w] - Acquire lock on the active stream. If read/write mode is unspecified, write lock is acquired.
  • unlock - Release active lock.

Playback Commands

set stream ticks
play ticks2

Playback allows to play data from one or several streams in real-time and store it into another stream.

The data is stored undistorted, except the timestamp of every message is adjusted to current time.

Main commands:

set playlog off|time|data

Control the output while playing data:

  • off - no output
  • time - log # of messages and historical time
  • data - log all played messages entirely

play <into stream>

Starts playing the currently selected data into the specified stream.

Most general Data Selection commands can be used to control the start time and filtering of data, if desired.

You may find at least set time ... to be useful, e.g. set time to market open.

The playback process starts in the background. The commands to control playback:

  • pause - Suspends playback
  • resume - Resumes playback
  • next - Immediately skips to the next message. This command is useful to skip over a time gap in the data.
  • stop - Completely terminates playback

Backup/Replicate/Restore Stream

Main Commands


> Example 1: 

==> set bkpath C:\backups\trade
==> set stream trade
==> backup

> Example 2: 

==> set bkpath C:\backups\trade
==> set stream trade
==> set reload allow
==> restore bars

> Example 3: replicate using specified symbols 

==> set srcdb URI
==> set srcstream trades
==> set reload truncate
==> symbols add EQUITY ORCL
==> replicate trade

> Example 4: replicate all stream data

==> set srcdb URI
==> set srcstream trades
==> set reload allow
==> replicate trade

> Example 5:

==> set timeoffset 1D // 1day

==> set bkfilesize 50

==> set bkcommitsize 500000



  • backup – start backup.
  • set bkpath <path> – Set path to backup folder.
  • set timeoffset <interval> – Set selection start time as offset from end time (in time interval).
  • set endtimeoffset <interval> - Set selection start time as offset from end time (in time interval).
  • set bkcommitsize <count> - Set number of messages for the single transaction when writing into backup file.
  • set bkfilesize <size> - Set size in MB of the single backup file, 100MB by default.
  • set cpmode <live|batch> - Enable/disables live replication.
  • set retries <count> - Set number of attempts to retry replication/backup/restore procedure in case of any error. Default value is 0.
  • set retrytimeout <count> - Period of time in ms between reconnect attempts for replication/backup/restore procedures in case of any error. Default value is 5 seconds.
  • set – display defined settings
  • set reload <allow/truncate/prohibit> - Set reload mode for replication/backup in prohibit mode, copy processes will NEVER change target data, new data can be appended. In truncate mode, when data replicated or restored, target stream can be truncated. In reload mode, target data can (and will) be wiped out due to schema incompatibility, no matter what caused it.
  • backup [format] - In case [format] is specified, clear previous saved data in , otherwise incremental backup will be done.
  • restore <stream> - Restore from or into specified stream. if does not exists, it will be created. All data from is restored, no filters applied.
  • set srcstream <stream> - Set source stream to replicate/restore from.
  • set srcdb <uri> - Set source database to replicate from.
  • replicate <stream> - Replicate from source db and source stream into specified . If does not exist, it will be created.

Backup Stream

Use backup function replicate from the selected stream into external files to the specified path (back-up folder).

TimeBase supports online and offline backup of streams between multiple TimeBase instances.

The goal of backup is to incrementally store all changes made in a set of streams, or entire TimeBase instance, into a set of compressed files.

Later, backup files can be used to restore data.

Backup Any Stream From Source Embedded Tickdb

tickdb
set db URI\tickdb
open
set cpmode  batch
set bkpath PATH
set stream stream_name
backup
close
quit

Backup any Stream From Source Remote Tickdb

tickdb
set db URI
open
set cpmode  batch
set bkpath PATH
set stream stream_name
backup
close
quit

Backup Stream With Specified Filter (symbol)

tickdb
set db URI\tickdb
open
set cpmode  batch
set bkpath PATH
set stream stream_name
symbols add symbol_name 
backup
close
quit

Only filtered data from stream will be back upped.

Backup Stream With Specified Time Range

tickdb
set db URI\tickdb
open
set cpmode batch
set bkpath PATH
set stream stream_name
set tz America/New_York
set timeformat yyyy-MM-dd HH:mm
set time 2020-06-15 07:15
set endtime  2020-06-15 07:50
backup
close
quit

The data of defined time range will be back upped.

Backup Stream With Specified Timeoffset or Endoffset Filter

tickdb
set db URI\tickdb
open
set cpmode  batch
set bkpath PATH
set stream stream_name
set timeoffset 1D //one day
backup
close
quit

The filtered data will be back upped.

Live Backup Any Stream From Source Remote Tickdb

tickdb
set db URI
open
set cpmode live
set bkpath PATH
set stream stream_name
backup

Backup Format

tickdb
set db URI
open
set cpmode  batch
set bkpath PATH
set stream stream_name
backup
set stream stream_name
backup
set stream stream_name
backup format
close
quit


Use to clear previous backup in folder and do stream backup.

Restore Stream

Restore backup files into [targetstream].

If target stream does not exist in the target tickdb, new stream is created to store data.

Before restoring, it is recommended to backup stream files.

Restore From Backup Folder To Embedded Tickdb

tickdb
set db URI\tickdb
open
set cpmode  batch
set reload prohibit
set bkpath PATH
set srcstream stream_name
restore  stream_name
close
quit

Restore From Backup Folder To Remote Tickdb

tickdb
set db URI
open
set reload prohibit
set cpmode  batch
set bkpath PATH
set srcstream stream_name
restore  stream_name
close
quit

Restore if Target Stream Same as Source Schema


> Prohibit Example 

tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload prohibit
restore  stream_name
close
quit


> Allow Example 

tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload allow
restore  stream_name
close
quit

> Truncate Example 

tickdb
set db URI
open
set bkpath PATH
set srcstream  stream_name
set reload truncate
restore  stream_name
close
quit

Set reload mode = prohibit

Restore will only append new data into target stream due to prohibit mode.

Target data is never changed, new data can be appended.

Set reload mode = allow

All data from backup are restored (re-written).

Initial data of target stream will be lost (clearing).

Set reload mode = truncate

Target stream will be truncated.

Restore if Target Stream and Source Schema Are Different

tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload prohibit
restore  stream_name
close
quit

tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload allow
restore  stream_name
close
quit

Restore data in case target existing stream and source stream schema (backup and target) are different and set reload mode = prohibit or truncate.

Data will not be restored.

User will get the WARNING: “Target stream schema is not compatible with backup. Cannot restore stream into different schema”.

Restore data in case target existing stream and stream schema (source and target) are different and set reload mode= allow.

Data will be reloaded and stream schema re-written.

Initial data of target stream is lost (clearing).

Replicate Stream

TimeBase supports online and offline replication of streams between multiple TimeBase instances.

The goal of replication is to incrementally propagate all changes made in one stream (master) to one or more streams in additional TimeBase instances (slaves).

Replicate from source db and source stream into specified <stream>.

The online mode maintains an open connection from the slave instance to the master instance, and all inserts, updates and deletes are propagated right away.

The offline mode is used to replicate data in batch mode, for example, during the off hours.

When replication is run in offline mode, it downloads all available changes from the master, closes connection, and stops.

It is also possible to replicate stream from source tickdb where Server run in read-only mode.

Replicate From Remote Source DB Into Specified Stream In Tickdb


tickdb
set db  URI
open
set srcdb URI
set cpmode batch
set reload prohibit
set srcstream stream_name
replicate stream_name

Use in case target stream did not exist in the target tickdb.

Target stream will be created. Data will be replicated.

Replicate Data With Filter


tickdb
set db  URI
open
set cpmode batch
set reload prohibit
set srcdb URI
set srcstream stream_name
symbols add symbol_name
replicate stream2_name

Target stream will be created. Filtered Data will be replicated only.

Replicate With Time Range


tickdb
set db URI
open
set cpmode batch
set reload prohibit
set srcdb URI
set srcstream stream_name
set tz America/New_York
set timeformat yyyy-MM-dd HH:mm
set time 2009-06-15 07:15
set endtime  2009-06-15 07:50
replicate stream_name
close
quit


Target stream will be created.

Data of specified interval will be replicated only.

Replicate With Timeoffset/Endoffset Filter

Example1:

tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set endtimeoffset 1H
replicate stream_name
close
quit

Example 2:

tickdb
set db URI\tickdb
open
set srcdb URI
set srcstream stream_name
set timeoffset 1D
replicate stream_name
close
quit

Target stream will be created.

Data of defined filter will be replicated only.

Replicate Target Stream Schema Incompatible With Source


> Prohibit 

tickdb
set db  URI
open
set srcdb URI
set srcstream ticks
set cpmode batch
set reload prohibit
replicate stream_name


> Truncate

tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set cpmode batch
set reload truncate
replicate stream_name

> Allow

Example:
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload allow
replicate stream_name

Use in case target stream schema is not compatible with source.

Set reload mode =prohibit.

Operation is not possible.

Cannot replicate stream into different schema. Target stream should not be changed.

User will get a warning: “Target stream [stream_name] schema in not compatible with [stream_name]. Cannot replicate stream.”

Set reload mode= truncate

Operation is not possible.

Cannot replicate stream trade into different schema. Target stream should not be changed.

Set reload mode= allow

All data and target stream schema get reloaded.

Clearing data and changing target [stream_name] schema due to rewrite mode. Initial data of target stream is lost (clearing).

Replicate Target Stream Schema Compatible With Source

> Allow 


tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload allow
replicate stream_name


> Truncate 

tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload truncate
replicate stream_name

> Prohibit 

tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload prohibit
replicate stream_name


Use in case target stream schema is compatible with source.

Set reload mode = allow

All data and target stream schema get reloaded.

Initial data of target stream is lost (clearing).

Set reload mode = truncate

In case data is replicated, target stream is truncated.

Set reload mode = prohibit

Target data is never changed, new data can be appended.

Live Replication


tickdb
set db  URI
open
set srcdb URI
set cpmode live
set reload prohibit
set srcstream stream_name
start replicate stream_name

Source data recordings from the aggregator are replicated to other remote tickdb.

The online mode maintains an open connection from the slave instance to the master instance, and all updates are propagated right away.

Note:

  • jobs - Display list of currently executing background jobs
  • kill <job_id> - Interrupts job with given <job_id>

Live Replication With Changed Schema


> Example 1:

tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set cpmode live
set reload prohibit
replicate stream_name

> Example 2: Target stream schema was changed. Reloading all data. Data will be replicated.

tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set cpmode live
set reload allow
replicate stream_name

Use in case a source schema got changed in the middle of the process of live replication.

Can not replicate.

Cannot change schema due to restrictions of reload mode [prohibit] (if set reload= prohibit) or Reload data (if set reload= allow)

Benchmarks


set warmup 20 
set duration 100
set iterations 3
set rate 100000
set payloadsize 100
benchmark thr

TimeBase shell has built-in capability to run throughput and latency tests.

Benchmark command allows getting general idea on channel performance between the client located on the machine with the shell installed and the selected tickdb instance.

  • benchmark thr- Runs throughput benchmark using current settings
  • benchmark lat - Runs latency benchmark using current settings
  • set warmup <value> - Sets warmup time (in seconds) before making measurements (before each iteration)
  • set duration <value> - Sets duration (in seconds) for a single measurement iteration
  • set iterations <value> - Sets a number of benchmarks iterations
  • set rate <value> - Sets target message rate (number of messages per second), affects only latency benchmarks
  • set payloadsize <value> - Sets the size of an additional payload (in bytes). Default value = 0.
  • set channeltype <value1,value2> - Sets channel type for benchmark. Valid values:
    • dstream - durable stream
    • tstream - transient stream
    • ipctopic - topic IPC
    • updsingletopic - topic UPD single producer

Command Line Options

It is possible to run TimeBase shell in non-interactive mode using -exec command line argument.

This argument can be used multiple times.

tickdb -db dxtick://localhost:8011 -exec open -exec streams -exit

tickdb -db /deltix/timebase -time "2004-01-01 12:45:00.0" -exec open ro -exec select -exit

It is possible to place commands in a text file and supply the file name as an @ argument to TimeBase Shell:

tickdb @/deltix/temp/myscript.txt