Live Configurable Feed

This tutorial takes the reader through the basic steps of setting up a live configurable feed process using the Kraken data connector. Kraken data connector aggregates data from the KRAKEN exchange.

Preparation

If you have not done so yet, you must install Deltix QuantServer.

info

Installation instructions.

Setting up QuantServer Home

Starting QuantServer Architect

Launch QuantServer Architect (QSA). If you are using Windows with User Access Control (UAC) enabled, you usually want to run QSA as Administrator. This allows you to configure the QuantServer processes as system services. However, this step-by-step guide will show you how to run QuantServer processes in command windows, which does not require administrative privilege.

tip

The same exact process will work on Linux.

Having installed QuantServer, the installer will suggest that you launch QuantServer Architect. It is recommended to select this option. If QuantServer is already installed, you can launch the QuantServer Architect client from the Windows desktop shortcut.

If you are using Linux, or you do not want to use the desktop shortcut, you can simply launch {installation root}/QuantServer/bin/qsadmin.sh or {installation root}\QuantServer\bin\qsadmin.cmd (Windows).

You might want to set up your executable search path to include the QuantServer\bin folder, so you can run commands by name in command shell windows.

QuantServer Architect remembers the last used QuantServer configuration. However, when you run it the first time, it may not have one set, or may default to C:\Program Files\Deltix\QuantServerHome. It is not a good idea to create a QuantServer Home in the installation folder for several reasons:

• It is usually located on the system partition, which should not be used for database files.
• Administrative access is usually required to create files there.
• If you are reading this guide, you are probably creating a test environment, as opposed to a centralized production environment. You should create it in a temporary area to which you have access.

To create a QuantServer Home in a private folder, just navigate to where you would like to create it in the Open dialog and click the Create New Folder button.

Configuring Services

After you create a QuantServer Home, the QuantServer Administrator will notice that it is new and automatically enter the "Configuration Edit" mode.

Configure as follows:

1. Enable Aggregate and/or Maintain Market Data.
2. Change TimeBase and Aggregator Service mode to CMD. This will cause all services to run as simple console commands, which make them easier to manage in a test environment, and avoid the requirement for administrative privileges on the system.

Here is a list of properties for advanced service configuration:

Property	Description
Embedded	Embed this service in main QuantServer process. In this mode only Java-based data connectors (if Aggregator) or strategies (if UHF) can be used. The advantage of embedded mode is the elimination of inter-process communication between this service and TimeBase. The disadvantages are: 1. You cannot run .Net-based connectors. 2. Possibile malfunction of data connectors (if Aggregator) or Algorithms (if UHF) and resource consumption that affects TimeBase.
Service	Determines whether this component will be installed as a system service: AUTO Install as automatically started system service. MANUAL Install as a system service, but do not start by default. User will start the service from the service control panel, or from QuantServer Administrator. CMD Do not create a system service. User will start the component by launching a command file. This can be done by right-clicking the component on the diagram, when in Management Mode.
Failure Action	Specifies action to execute if service terminated abnormally. RESTART Restart the service. Special Watchdog service will be used to watch over the running service. Please install DXWatchdog service if you haven't done so already.
Port	The port number, on which this process will listen for connections.
Enable SSL	Activate if you want to protect your connection using SSL.
Remote	Do not install service in this Home. Instead, reference a separately installed service by specifying its host name or IP address and port number.
Enable Remote Access	Allow access to management web interface for remote http clients.
Host	The host name or IP address for accessing the service.
id	Identifies Aggregator instance when multiple aggregators share the same TimeBase.
Memory Size	Initial The amount of memory allocated to QuantServer on startup. If QuantServer requires more memory, it will grab additional memory from the operating system, up to the Max amount. Ideally, the amount of Initial memory will coincide with the amount of Max memory, so that QuantServer does not have to grab additional memory from the operating system in the middle of doing something. Also, in some cases, it is possible that QuantServer may be unable to grab additional memory due to virtual memory fragmentation. However, if memory is at a premium, you can start QuantServer with a lower amount of Initial memory. This parameter corresponds to the Java Virtual Machine option -Xms. Max The maximum amount of memory QuantServer will allocate for its heap. Most of the objects used by QuantServer reside in the so-called heap. Max memory must exceed the Cache size by a reasonable amount, to allow room for QuantServer to allocate additional objects on top of the memory reserved for caching persistent data. Please see the sizing guide for details. This parameter corresponds to the Java Virtual Machine option -Xmx. Physical Physical memory available on this system.
Reconnect Settings	Aggregator reconnect settings. These settings control reconnect interval calculation (between two sequential reconnect attempts): NextAttemptInterval = InitialInterval * FactorN - 1 where N - reconnect attempt number [1..MaxAttemptCount] If MaxInterval > 0 then NextAttemptInterval = Min(NextAttemptInterval, MaxInterval)
Max Attempt Count	Max reconnect attempt count. Aggregator stops reconnecting if this number is reached. When set to 0, Aggregator retries infinitely.
Max Interval	Max interval between two sequential reconnect attempts. If set to 0, then interval is not restricted.
Initial Interval	Initial reconnect interval.
Factor	The factor, by which reconnect interval increases.
Additional JVM args	Additional tune up parameters for JVM when Aggregator runs as Java process.
Platform	Manual platform switching for Aggregator. This property overrides platform value automatically allocated by QuantServer Architect.
Single Option	Turn on this flag to subscribe for single option symbol. By default, Aggregator subscribes only for option roots (e.g. for the entire option chain).

Your configuration should look as follows:

Click Apply, Start, and then Done. Congratulations, you have just created a QuantServer home!

Configuring Securities Metadata Stream (SMD)

The SMD stream will contain a list of financial instruments (also called securities, tickers, symbols, etc.). This stream will be durable, which means it will be permanently stored on disk. In this, durable streams are like traditional database tables.

The unique feature of TimeBase is transparent integration of the concepts of database tables and message topics, in the form of durable and transient streams. By configuring a stream as transient or durable, the user can control whether the data is asynchronously stored for future back testing or auditing, without impacting any other aspect of system configuration.

Start TimeBase: right-click the TimeBase box and choose Start in Console. A console window comes up, and TimeBase will initialize quickly. You can minimize this window to the taskbar (on Windows).

1. If your TimeBase is empty, we will need to create the SMD stream through Timebase Administrator. Right-click Storage and make sure the Read-Only Mode option is unchecked. Then, launch TimeBase Admin from the same menu.
2. Select Data > New Stream from the top-level UI. A new stream dialog box will appear. Enable Security Metadata checkbox. This action will automatically populate the Key field with required "securities" key. Accept all other default values and select Next and then Create on the following panel.

Start Aggregator: right-click the Aggregator box and choose Start in Console. A console window comes up, and Aggregator will initialize quickly. You can minimize this window to the taskbar (on Windows).

1. The last step before we move to creating the configurable feed process is to add a set of instruments to the securities stream.
2. In TimeBase Administrator, right-click securities and choose Edit. The securities editor opens.
3. Click Add, then Currency. Click in the Symbol cell and type in XBT/USD. Click Accept. You have configured a Currency with the symbol of XBT/USD. Repeat the process and add as many symbols as you would like. For this example, all DJIA components will be added to the SMD stream. TimeBase Administrator should now look as follows:

tip

Do not close TimeBase Administrator yet, we will need it to confirm that the data feed is working once the configurable feed process is created.

Accessing the Data Connector Wizard

The data connector wizard is where you will define your configurable feed process. You can access the wizard as follows:

1. Edit Aggregator Processes: right-click the Aggregator box and select Edit Processes.
2. New Configurable Feed Process: Select New... drop-down menu and select New Configurable Feed. This will launch the Data Connector Wizard that is used to define a configurable feed process.

Configure General Process Properties

The General Process panel contains a list of properties that are common across all data provider configurations.

Property	Description
Aggregator Process Key	Required field that must be unique.
Start Process Automatically	When on, the process is started automatically upon aggregator startup or process update. When off, the process must be started manually.
Control Field Key	Key should point to a field within Timebase Security Metadata stream that contains feed specific symbol format. This property controls symbol selection as follows: - If this property is empty, all symbols present in the Security Metadata stream are requested from the feed provider. - If the property is not null, and the field is of VARCHAR data type, then the field is considered to contain the symbol to query from the feed provider. This allows to map symbols on-the-fly. Market messages stored into TimeBase will contain the main TimeBase symbol, and not the feed provider's symbol. - If this property is not empty, and the field is of BOOLEAN data type, then aggregator selects symbols with controlField == true. - If this property is not empty, and the field is of any other data type, then aggregator selects symbols with controlField == is not empty.
Schedule	Scheduling pattern in the cron scheduling pattern language for starting the process. For indefinitely running processes, such as indefinite live data collection, this can be left unset (or set to @reboot), in which case the process will launch on aggregator startup. If value is empty or set to @manual, then process activity should be started manually.
Max Duration	Maximum process duration. If specified, the process will be terminated after reaching this limit.
Schedule Time Zone	If specified, overrides local time zone.
QQL Filter Expression	Filters out connector messages, which don't satisfy the specified QQL condition expression. Examples: - size > 1 and this is deltix.qsrv.hf.pub.TradeMessage - bidSize >100 and this is deltix.qsrv.hf.pub.BestBidOfferMessage - tradeType =='BUY' and this is deltix.qsrv.hf.plugins.data.lime.types.LimeTradeMessage
Message Transform Type	Price Multiplier Transform: Property points to a field within the SMD which contains a multiplier value. The field should be numeric data type. Compound Transform: Custom transformation.
Loader Performance Type	Minimize Latency: Special mode where loader flushes messages immediately rather than buffer them. This option reduces internal latency but may require an additional CPU power.

Steps:

1. Define Aggregator Process Key.
2. Select Next.

Choose a Data Connector

The Data Connector panel will list all the data connectors that are currently certified and available for installation.

Steps:

1. Enable Kraken and select Next to install the Kraken data adapter. This process will download the connector and install it into your local QuantServer installation within the custom folder.

Configure the Data Connector

Configuration of the data connector consists of defining connection URL(s) and Advanced data connector parameters.

Steps:

1. In this example, we will define connection with Kraken WebSocket URL
2. In this example, we will use default Level2 Parameters for Kraken Data Connector
3. Select Next to complete data connector configuration.

Configure the Data Model

The data model tree presents all messages available via the Kraken data connector. Italicized messages represent abstract classes that are automatically selected when enabling a message type that inherits their properties. Input data is organized as a package. Package is a header format, containing a header field that describes message type, and message body.

info

More information about Universal Format.

In this example, we will setup aggregator process to record Best Bid Offer, Level 2, and Trade Messages. The relationship is expressed by "->" to parent class.

Steps:

1. Enable Kraken Package Header -> Package. Notice that Abstract classes are automatically enabled.
2. Expand Entries under Kraken Package Header.
   • Market trades are presented as TradeEntry type;
   • BBO is supplied as L1Entry;
   • L2 data (like FIX IncrementalRefresh/FullRefresh, CME.SBE, FAST, etc) is converted into: L2EntryNew (for new price level) and L2EntryUpdate (for updating or deleting price levels);
   • BookResetEntry - enforces market data provider to drop state of a particular book.
3. (Optional): Enable Connection Status Change message to receiving notification of connection changes.
4. Select Next to complete the data model configuration.

Test-Drive the Data Connector

caution

Optional
Only Supported in 4.3

The test drive feature initializes an embedded aggregator and established data feed connection, creates and starts a process and redirects messages to the monitoring table. Symbol mappings, transformations, and the other process properties (except schedule and max duration) affect the output in monitoring table. In this example, we will test that we receive data for symbols configured in securities stream.

Steps:

1. Activate the data connector by selecting the below icon. Verify connection has been successfully made by viewing output in monitoring table.

2. Verify that you begin to receive messages for your data model.
3. Disconnect the adapter by selecting the below icon before choosing a destination stream.

4. Select Next.

Choose the Destination Stream in TimeBase

It is recommended to create a new stream, when possible, to avoid any conflicts between the data model you defined and existing DB models. If an existing stream is used, a comparison between data and DB models will be made and you will have an opportunity to resolve any non-fatal conflicts while reviewing encodings in the next step in the process definition.

Steps:

1. Select New.
2. Select Next.

Review Encoding

The next step in the process will allow you to review the schema model that will be used to create the new stream in TimeBase. Each class node can be expanded and encodings for all of the class attributes can be reviewed/modified.

Steps:

1. No changes are necessary if we selected to create a new stream so that data model and database model are identical.
2. Select Next.

note

In some cases, use of an existing stream will be required. The following table highlights some conflicts that may arise and possible options on resolving them through the Review Encoding panel. Fatal in this table represents a situation where it is not possible to complete the feed process configuration successfully without altering the DB model through the TimeBase Administrator.

Diagnostic	Action	Comment
Class in Model but not in DB.	Add class to schema.	TimeBase change.
Field in Model but not in DB.	Add field to class.	TimeBase change. The field is always added to the end of class fields list to avoid any data conversion. When a new field belongs to the parent class, it is added to the concrete classes, which are inherited from the parent.
Field in DB but not in Model (nullable)	Set field to NULL in every message.	Warning
Non-nullable field in DB but not in Model.	None	Fatal
Encoding may be too narrow.	None	Warning
Static field in DB but non-static field in Model.	None	Fatal
Different field type.	None	Fatal
Different enum descriptors.	None	Fatal

Creating New Stream

Define the stream configuration that will be used to store the data that is consumed by the feed process.

Property	Description
Key	Required field that must be unique.
Name	Name of stream.
Description	Description of stream.
Scope	Specifies the scope of the stream to be one of the following: DURABLE: This stream will be persisted on durable storage (hard disk/SSD/SAN). TRANSIENT: This stream will not be persisted and can be thought of as a queue for passing messages from the data connector to the UHF server and other real-time clients. TOPIC: Data will be written to TimeBase Topic. Suitable for low-latency feeds.
Distribution	Determines how the stream is distributed. This is a critical tuning parameter. The choices are: DISTRIBUTED: Data for every entity ("symbol") is stored in a separate M-File. CONSOLIDATED: All data in this stream is stored in a single M-File. HYBRID: Data in this stream is distributed among the specified number of M-Files. Here are some rules to follow when choosing the distribution model: In all cases, avoid having more than 400 or so M-Files. There is no hard limit, and the performance curve is highly hardware-dependent, especially CPU cache size. As the number of M-Files increases, CPU cache misses will start progressively impacting the read performance. Also, having a very large number of files in a single folder exposes OS inefficiencies. If you have thousands of symbols, you should NOT choose the DISTRIBUTED model. Let's say you have a large number of symbols (thousands), and your performance goal is the ability to quickly retrieve very small subsets of those symbols (about 50-100 symbols at once). In this case, choose the HYBRID model with the number of files at least 3 times the number of symbols you typically expect to read (but not too large, according to rule #1). However, if you typically read all, or even close to half of the entities, the hybrid system will not bring any benefits. Or, if you need to read more than 200 symbols at a time, you would have to have the # of M-FIles in the 600+ range in order to achieve the selectivity benefit, which would be immediately negated by CPU cache misses. In these cases, choose the CONSOLIDATED model for the best performance.

Steps:

1. Define stream Key.
2. Define the appropriate Scope and Distribution based on your particular use case.
3. Select Next.

Complete the process configuration by selecting Finish and Apply. Verify that the feed process has been activated and that symbol subscription has occurred by reviewing aggregator logs. Verify that you are receiving data by monitoring data through Timebase Administrator stream monitor feature.