Skip to main content

Shell CLI

Launch​

Execute tickdb script to open shell console.

tip

You can find all scripts in the bin folder in your TimeBase installation.

Case Sensitivity​

Commands used by TimeBase Shell are case-insensitive. However, parameters, stream names, and symbols are case-sensitive: 

==> set STREAM Coinbase
set STREAM: unrecognized option. (Type ? for usage)
==> SET stream Coinbase
Stream 'Coinbase' was not found.
==> SET stream COINBASE
COINBASE

Help​

Run question mark ? command to display Shell help.

==> ?
tickdb Shell Commands:
********************************************************************
Note: it is often convenient to specify the set db and open commands
on the command line:
…

Exit​

You can exit TimeBase Shell by running exit or quit command.

Set Commands​

TimeBase Shell SET command controls operating options described in the sections below.

Global Options​

  • Set maximum number of messages printed at once (page size).
  • Set time zone for parsing and displaying time.
  • Set time format for parsing and displaying timestamps.
# Max Number of Messages 
set max <n>
set max 20

# Time Zones 
set tz <time zone>
set tz GMT
set tz America/New_York

# Time Format 
set timeformat <format>
set timeformat yyyy-MM-dd HH:mm:ss

DB Management Commands​

Set TimeBase location using URL or path. In most cases you connect by passing TimeBase server URL. set command starts with dxtick:// followed by host and port.

set db dxtick://localhost:8011

This command allows specifying user's credentials when TimeBase has user access control enabled:

set db dxtick://user:password@localhost:8011

set command usually followed by the open command that performs the connection to TimeBase:

set db dxtick://localhost:8011
open

If TimeBase server is not running, TimeBase Shell can also open TimeBase data storage directory (and launch mini-TimeBase server inside this tool). In this case specify the TimeBase directory path instead of the URL:

set db /deltix/timebase
open

User Credentials (UAC)​

Run set user to pass user's credentials to TimeBase connection (when User Access Control is enabled):

set user <user> <password>
set user jdoe WTdRE*rvDU*C
tip

A more secure option would be to specify hashed password value. Ask support for instructions.

Main Commands​

Open Database​

Run open to connect to TimeBase server. Add ro parameter to open a read-only connection .

Open database
open //open database
open ro //read-only connection
set db dxtick://localhost:8011 //full example 
open ro

List Streams​

Run streams command to get a list of streams.

View Stream Schema​

Run desc (describe) command to get a stream schema.

Describe command example
==> desc balances
DURABLE STREAM "balances" (
    CLASS "deltix.samples.uhf.balance.BalanceMessage" (
        "account" VARCHAR,
        "amount" FLOAT,
        "destination" VARCHAR
    );
)
OPTIONS (FIXEDTYPE; PERIODICITY = 'IRREGULAR'; DF = 1; HIGHAVAILABILITY = FALSE)
COMMENT 'Auto-created by Ember'

Close Database​

Run close command to gracefully close the connection.

Select Current Stream​

Use this command to set the stream you will be working with after establishing the connection. Use stream keys to identify streams.

Selecting streams
set stream bars

# select multiple streams
set stream bbo, trades

View Data​

  • Run entities command to get instruments collected by selected stream.
  • Run select command to view data in the selected stream. This command reads data from the beginning of stream.
  • Run monitor command to see live messages written into the stream. This command continuously displays newly recorded messages. You can hit Enter key to interrupt it.
Viewing stream data
==> set stream balances
balances
==> select
InstrumentMessage,CUSTOM,BTC,2020-01-07 14:45:33.063,1578426333063
InstrumentMessage,CUSTOM,USD,2020-01-07 14:45:33.063,1578426333063
InstrumentMessage,CUSTOM,LTC,2020-01-07 14:45:33.063,1578426333063
InstrumentMessage,CUSTOM,ETH,2020-01-07 14:45:33.065,1578426333065

# Monitor 
==> set stream COINBASE
COINBASE
==> monitor
Monitoring COINBASE; hit <Enter> to abort ...
PackageHeader,FX,XRP/BTC,2020-03-12 08:50:24.481,1584017424481,999
PackageHeader,FX,BTC/USD,2020-03-12 08:50:24.481,1584017424481,999
PackageHeader,FX,BTC/USD,2020-03-12 08:50:24.481,1584017424481,999
PackageHeader,FX,XRP/USD,2020-03-12 08:50:24.483,1584017424483,999
PackageHeader,FX,LTC/USD,2020-03-12 08:50:24.483,1584017424483,999

STOP CURRENT OPERATION (y,n) ? ==> y

Data Copy​

Run copyfrom command to copy all data from the specified streams to the current stream, using the current time range and filter.

Copying data between streams
set stream apple-bars
symbols add EQUITY AAPL
copyfrom bars

Filters​

Filters allow filtering data by symbol and timestamp.

Symbol Filters​

Symbol filter commands:

  • symbols all - Subscribe all symbols.
  • symbols show - Show subscribed (filtered) symbols.
  • symbols clear - Clear subscribed symbols from the filter.
  • symbols add <type> <symbol> - Add specified symbols to the filter.
  • symbols remove <type> <symbol> - Remove the specified symbols from the filter.
Filtering by symbols
symbols add EQUITY AAPL
symbols add EQUITY MSF
symbols remove FUTURE CLZ12

Time Filters​

Set time and endtime for data selection operations. The Default timestamp format is yyyy-MM-dd HH:mm(can be changed using timestamp option):

  • set time <text> - Set selection time by parsing text (text must match the current format precisely).
  • set endtime <text> - Set selection end time by parsing text (text must match the current format precisely).
Filtering by time
set stream bbo
set tz GMT
set timeformat yyyy-MM-dd HH:mm
set time 2009-01-02 15:00
set endtime 2009-01-02 16:00

Import From Data File Into Defined Stream​

Run import command to import data from the file into the selected stream. Parameter set src is used to select source file path (usually in .qsmsg or .qsmsg.gz format)

Importing from a file
open rw
set src c:\dailyBars.qsmsg.gz
set stream dailyBars
set tz GMT
import

Export Data To Data File​

Run export command to export stream data into a file. The destination file name is passed as an export argument. Export respects the selected filters.

Exporting to a file
# exports data only for futures CL12 and NG12
set db dxtick://localhost:8011
open
set stream trades
export d:\temp\exp.qsmsg.gz

set stream dailyBars
symbols all
symbols add FUTURE CL12 
symbols add FUTURE NG12
symbols show
export c:\future100.qsmsg.gz

Delete Stream​

Run delete to remove completely the selected stream. In this and similar cases database must be open in read-write mode.

Deleting stream
set stream ticks
delete
info

Refer to TickStream API to read more about deleting streams.

Purge Stream​

Run purge to remove all stream data before the specified timestamp (not including the specified time). 

set stream ticks
purge 2009-01-01 09:30:00.00
info

Refer to TickStream API to read more about purging streams.

Truncate Stream​

Run truncate to remove all stream data after the specified timestamp. In this and similar cases database must be open in read-write mode.

set stream ticks
truncate 2009-01-01 09:30:00.00
info

Refer to TickStream API to read more about truncating streams.

Trim Stream (Enterprise)​

TimeBase 4.0 allocates all data in so-called M-files. Each file has a preset size. In case we truncate a stream, we remove all the data after the specified timestamp, but the size of the corresponding M-file may still remain intact. Run trim to resize the M-file to match the size of the data that is actually stored in it. In this and similar cases database must be open in read-write mode.

Delete Data In The Middle​

It is possible to remove data in the middle of the stream. This command takes as arguments (inclusive) start time and end time of the time range to be deleted.

tip

Either start or end time range can be omitted.

In this case this command is similar to purge and truncate.

delete [start];[end]   

# Delete time range from selected streams
delete 2009-01-02 09:31:00.0;2009-01-02 09:59:00.0

# Delete all data before specified date
delete ;2009-01-02 09:59:00.0

# Delete all data older than specified date
delete 2009-01-02 09:31:00.0;

Rename Stream​

Run rename to rename the selected stream.

set stream GDAX
rename COINBASE
info

Refer to TickStream API to read more about renaming streams.

Lock/Unlock Stream​

TimeBase supports per-stream read/write locks. Only one client connection can acquire a write lock at a specific time. Multiple connections can acquire read locks and prevent a concurrent writing.

  • lock [r|w] - Acquire lock on the active stream. Write lock is acquired if read/write mode is not specified.
  • unlock - Release active lock.
# Example 1: 
set stream securities
lock
import
unlock

# Example 2:
set stream ticks
lock r
select
unlock
info

Refer to TickStream API to read more about locking streams.

Playback Commands​

Playback allows playing data from one or several streams in real-time and storing it in other streams. The data is stored intact, except the timestamps, that get adjusted to the current time.

Control the output while playing data:

  • off - no output.
  • time - log # of messages and historical time.
  • data - log all played messages in-full.
set playlog off|time|data

Run data selection commands to control the playback start time and data filtering. 

set time to market   open

Run these commands to control playback on the background:

  • pause - Suspends playback.
  • resume - Resumes playback.
  • next - Immediately skips to the next message. Skip over a time gap in the data.
  • stop - Completely terminate playback.
# template
play <into stream>

set stream ticks
play ticks2

Backup/Replicate/Restore Stream​

Main Commands​

Common

  • set – Display the defined settings.
  • set timeoffset <interval> – Set selection start time as offset from end time (in time interval).
  • set endtimeoffset <interval> - Set selection start time as offset from end time (in time interval).
  • set retries <count> - Set number of attempts to retry replication/backup/restore procedure in case of an error. Default value is 0.
  • set retrytimeout <count> - Period of time in ms between reconnect attempts for replication/backup/restore procedures in case of an error. Default value is 5 seconds.
  • set reload <allow/truncate/prohibit> - Set reload mode for replication/backup/restore to define how the system propagates changes in the source stream to the target stream. 
      + `prohibit` - target stream remains unchanged. 
      + `truncate` - in case the source stream is truncated, changes are propagated to the target stream. 
      + `allow` - in case the source stream schema is changed, changes are propagated to the target stream.

Backup

  • backup – Start backup.
  • set bkpath <path> – Set a backup folder path.
  • set bkcommitsize <count> - Set number of messages for a single transaction when writing into a backup file.
  • set bkfilesize <size> - Set a single backup file size in MB. Default size is 100MB.
  • backup [format] - In case [format] is specified, clear previously saved data in <bkpath>; otherwise, incremental backup will be applied.
  • restore <stream> - Restore from <bkpath> or <sourcedb> into a specified stream. New stream is created in case <stream> does not exist. All data from <bkpath> is restored, no filters apply.

Replication

  • set cpmode <live|batch> - Enable/disables live replication.
  • set srcstream <stream> - Set source stream to replicate/restore from.
  • set srcdb <uri> - Set source database to replicate from.
  • replicate <stream> - Replicate from source db and source stream into a specified <stream>. New stream is created in case <stream> does not exist.
# Example 1: 
==> set bkpath C:\backups\trade
==> set stream trade
==> backup

# Example 2: 
==> set bkpath C:\backups\trade
==> set stream trade
==> set reload allow
==> restore bars

# Example 3: replicate using specified symbols 
==> set srcdb URI
==> set srcstream trades
==> set reload truncate
==> symbols add EQUITY ORCL
==> replicate trade

# Example 4: replicate all stream data
==> set srcdb URI
==> set srcstream trades
==> set reload allow
==> replicate trade

# Example 5:
==> set timeoffset 1D // 1day
==> set bkfilesize 50
==> set bkcommitsize 500000

Backup Stream​

Run backup to replicate the selected stream into external files in the preset backup folder. The backup goal is to incrementally store in compressed files al the changes made in a set of streams, or entire TimeBase instance. Backup files can be then used to restore data. TimeBase supports online and offline streams backup between multiple TimeBase instances.

Backup Any Stream From Source Embedded Tickdb​

tickdb
set db URI\tickdb
open
set cpmode  batch
set bkpath PATH
set stream stream_name
backup
close
quit

Backup any Stream From Source Remote Tickdb​

tickdb
set db URI
open
set cpmode  batch
set bkpath PATH
set stream stream_name
backup
close
quit

Backup Stream With Specified Filter​

# Only filtered data from stream will be backed up.
tickdb
set db URI\tickdb
open
set cpmode  batch
set bkpath PATH
set stream stream_name
symbols add symbol_name 
backup
close
quit

Backup Stream With Specified Time Range​

# The data of the defined time range will be backed up.
tickdb
set db URI\tickdb
open
set cpmode batch
set bkpath PATH
set stream stream_name
set tz America/New_York
set timeformat yyyy-MM-dd HH:mm
set time 2020-06-15 07:15
set endtime  2020-06-15 07:50
backup
close
quit

Backup Stream With Specified Timeoffset or Endoffset Filter​

# The filtered data will be backed up.
tickdb
set db URI\tickdb
open
set cpmode  batch
set bkpath PATH
set stream stream_name
set timeoffset 1D //one day
backup
close
quit

Live Backup Any Stream From Source Remote Tickdb​

tickdb
set db URI
open
set cpmode live
set bkpath PATH
set stream stream_name
backup

Backup Format​

# Use to clear previous backup in a folder and do stream backup.
tickdb
set db URI
open
set cpmode  batch
set bkpath PATH
set stream stream_name
backup
set stream stream_name
backup
set stream stream_name
backup format
close
quit

Restore Stream​

Run restore to backup files into [targetstream]. New stream is created to store data if a target stream does not exist in the target tickdb. It is recommended to backup stream files before restoring.

Restore From Backup Folder To Embedded Tickdb​

tickdb
set db URI\tickdb
open
set cpmode  batch
set reload prohibit
set bkpath PATH
set srcstream stream_name
restore  stream_name
close
quit

Restore From Backup Folder To Remote Tickdb​

tickdb
set db URI
open
set reload prohibit
set cpmode  batch
set bkpath PATH
set srcstream stream_name
restore  stream_name
close
quit

Restore if Target Stream Same as Source Schema​

Set set reload <allow/truncate/prohibit> mode for restore to define how the system propagates changes in the source stream to the target stream.

  • prohibit - target stream remains unchanged. Restore is stopped in this case.
  • truncate - in case the source stream is truncated, changes are propagated to the target stream.
  • allow - in case the source stream schema is changed, changes are propagated to the target stream.
# Prohibit Example 
tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload prohibit
restore  stream_name
close
quit


# Allow Example 
tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload allow
restore  stream_name
close
quit

# Truncate Example 
tickdb
set db URI
open
set bkpath PATH
set srcstream  stream_name
set reload truncate
restore  stream_name
close
quit

Restore if Target Stream and Source Schema Are Different​

Set set reload <allow/truncate/prohibit> mode for restore to define how the system propagates changes in the source stream to the target stream.

  • prohibit - target stream remains unchanged. Restore is stopped in this case.
  • truncate - in case the source stream is truncated, changes are propagated to the target stream.
  • allow - in case the source stream schema is changed, changes are propagated to the target stream.
tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload prohibit
restore  stream_name
close
quit

tickdb
set db URI
open
set bkpath PATH
set srcstream stream_name
set reload allow
restore  stream_name
close
quit

Replicate Stream​

TimeBase supports online and offline streams replication between multiple TimeBase instances. The goal of replication is to incrementally propagate all changes made in one stream (master) to one or more streams in additional TimeBase instances (slaves).

tip

Note, that entire TimeBase instances are not configured as masters or slaves. This is made on a stream level. It is supported and quite common to have stream S1 be replicated from TimeBase instance A to instance B, while stream S2 to be replicated from B to A.

Replicate from source db and source stream into a specified <stream>.

  • The online mode maintains an open connection from the slave instance to the master instance, and all inserts, updates and deletes are propagated right away.
  • The offline mode is used to replicate data in batch mode, for example, during the off-hours. When data is replicated offline, TimeBase downloads all the available changes from the master, closes connection, and stops.

It is also possible to replicate stream from source tickdb where Server runs in read-only mode.

info

Refer to Replication to read more about replication and replication options.

Replicate From Remote Source DB Into Specified Stream In Tickdb​

Use in case target stream does not exist in the target tickdb. Target stream will be created and all the data will be replicated.

tickdb
set db  URI
open
set srcdb URI
set cpmode batch
set reload prohibit
set srcstream stream_name
replicate stream_name

Replicate Data With Filter​

Target stream will be created. Only filtered data will be replicated.

tickdb
set db  URI
open
set cpmode batch
set reload prohibit
set srcdb URI
set srcstream stream_name
symbols add symbol_name
replicate stream2_name

Replicate With Time Range​

Target stream will be created. Only the specified interval data will be replicated.

tickdb
set db URI
open
set cpmode batch
set reload prohibit
set srcdb URI
set srcstream stream_name
set tz America/New_York
set timeformat yyyy-MM-dd HH:mm
set time 2009-06-15 07:15
set endtime  2009-06-15 07:50
replicate stream_name
close
quit

Replicate With Timeoffset/Endoffset Filter​

Target stream will be created. Only the defined filter data will be replicated.

# Example1:
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set endtimeoffset 1H
replicate stream_name
close
quit

# Example 2:
tickdb
set db URI\tickdb
open
set srcdb URI
set srcstream stream_name
set timeoffset 1D
replicate stream_name
close
quit

Replicate Target Stream Schema Incompatible With Source​

In case target stream schema is not compatible with the source stream schema, you can set set reload <allow> to replicate changes to the target stream schema. 

# Allow
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload allow
replicate stream_name

Replicate Target Stream Schema Compatible With Source​

Use in case a target stream schema is compatible with the source stream schema. Set set reload <allow/truncate/prohibit> mode for replication to define how the system propagates changes in the source stream to the target stream.

  • prohibit - target stream remains unchanged. Replication is stopped in this case.
  • truncate - in case the source stream is truncated, changes are propagated to the target stream.
  • allow - in case the source stream schema is changed, changes are propagated to the target stream.
# Allow 
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload allow
replicate stream_name

# Truncate 
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload truncate
replicate stream_name

# Prohibit 
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set reload prohibit
replicate stream_name

Live Replication​

Source data from the aggregator is replicated to the other remote tickdb. The online mode maintains an open connection from the slave instance to the master instance and all updates are replicated right away.

  • jobs - Display list of currently executing background jobs.
  • kill <job_id> - Interrupts job with a given <job_id>.
tickdb
set db  URI
open
set srcdb URI
set cpmode live
set reload prohibit
set srcstream stream_name
start replicate stream_name

Live Replication With Changed Schema​

Use in case a source stream schema has been changed in the middle of the live replication process. It is not possible to change stream schema in the prohibit reload mode. Set reload= allow to restart replication and propagate changes to the target stream.

# Example 1: replication is not possible.
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set cpmode live
set reload prohibit
replicate stream_name

# Example 2: Target stream schema was changed. Reloading all data. Data will be replicated.
tickdb
set db  URI
open
set srcdb URI
set srcstream stream_name
set cpmode live
set reload allow
replicate stream_name

Benchmarks​

TimeBase Shell can run throughput and latency tests. benchmark command allows getting channel performance general information between the client located on the machine with the Shell installed and the selected tickdb instance.

tip

Please keep in mind, that this benchmark is quite rough and may not represent the actual behavior correctly.

  • benchmark thr- Runs throughput benchmark using current settings.
  • benchmark lat - Runs latency benchmark using current settings.
  • set warmup <value> - Sets warmup time (in seconds) before making measurements (before each iteration).
  • set duration <value> - Sets duration (in seconds) for a single measurement iteration.
  • set iterations <value> - Sets a number of benchmarks iterations.
  • set rate <value> - Sets target message rate (number of messages per second), affects only latency benchmarks.
  • set payloadsize <value> - Sets the size of an additional payload (in bytes). Default value = 0.
  • set channeltype <value1,value2> - Sets channel type for benchmark. Valid values:
    • dstream - durable stream.
    • tstream - transient stream.
    • ipctopic - topic IPC.
    • updsingletopic - topic UPD single producer. 
set warmup 20 
set duration 100
set iterations 3
set rate 100000
set payloadsize 100
benchmark thr

Command Line Options​

It is possible to run TimeBase Shell in non-interactive mode using -exec command line argument. This argument can be used multiple times.

tickdb -db dxtick://localhost:8011 -exec open -exec streams -exec quit
tickdb -db /deltix/timebase -time "2004-01-01 12:45:00.0" -exec open ro -exec select -exec quit

It is possible to place commands in a text file and supply the file name as an @ argument to TimeBase Shell:

tickdb @/deltix/temp/myscript.txt