Skip to main content

Network Configuration

This guide explains how to tune TimeBase for networks with relatively high latency (10 ms and more).

Summary

Tuning TimeBase for High Latency Network​

If you intend to use TimeBase over a network with relatively high latency (10 ms and more), you are likely to need extra tuning to avoid a bottleneck in the transport layer. Such a bottleneck not only penalizes throughput, but is also likely to trigger data loss and bad latency spikes.

Select Target Percentile​

Before starting the tuning, determine the percentile of messages that you optimize for. An attempt to optimize for 100% of cases may mean suboptimal values for the majority of cases.

For example, if median network latency is 50 ms, but 99% of messages are delivered within 100 ms, 99.9% within 200 ms, and 99.99% within 1000 ms, then if your target is the 99.9% percentile, you may not need to optimize for 1000 ms latency, and can use 200 ms latency in calculations.

Computing Buffer Sizes​

TimeBase uses a dedicated data channel buffer for each loader and cursor. The required buffer size can be calculated as follows:

network_rtt = 2 * one_way_network_latency
accumulation_period = network_rtt + other_jitter
data_rate_bytes_per_second = message_size_bytes * message_rate_per_second
target_buffer_size = accumulation_period * data_rate_bytes_per_second

So:

target_buffer_size = (2 * one_way_network_latency + other_jitter) * message_size_bytes * message_rate_per_second

Where:

  • other_jitter is the maximum expected variance in processing time, possibly related to thread context switches, garbage collection, and similar factors, depending on the application and whether the environment was tuned for low latency.
    • Usually, it is OK to assume other_jitter to be 10ms.
  • one_way_network_latency is the one-way network latency in seconds at the target percentile.
  • message_size_bytes is the average message size in bytes. For market data feeds, the message size is often in the range of 200–300 bytes, but should be measured for your specific case.
  • message_rate_per_second is the message rate in messages per second at the given percentile.
    • Formally, if your target percentile is P, then for P% of messages the number of immediately preceding messages over 1 second may not exceed message_rate_per_second.

Example:

  • Message rate: 20 000 messages per second
  • Message size: 300 bytes
  • Network latency: 70 ms (one way, at target percentile)
  • Other jitter: 10 ms
  • target_buffer_size = (2 * one_way_network_latency + other_jitter) * message_size_bytes * message_rate_per_second = (2 * 0.070s + 0.010s) * 300 bytes * 20000 msg/s = 0.150s * 6 000 000 bytes/s = 900 000 bytes ~= 900Kb

Tuning TimeBase​

There are two buffer sizes that may need adjustment according to the computed target_buffer_size:

  • Socket buffer size — size of the socket buffer on the OS layer
  • Channel buffer size — buffer on the "logical" channel (cursor or loader) in TimeBase

Algorithm:

  • If your target_buffer_size is less than 64Kb, then you may not need to change the default values.
  • If your target_buffer_size is less than 512Kb, then you need to increase the socket buffer size and OS max buffer size.
  • If your target_buffer_size is greater than 512Kb, then you need to increase the socket buffer size, OS max buffer size, and channel buffer size.
  • If your data_rate_bytes_per_second is greater than 4Mb, then you may need to additionally disable compression to avoid excessive CPU usage. However, that should be carefully measured for your specific setup and may require an additional increase of the socket buffer size to accommodate uncompressed data.

Socket Buffer Size​

  • The default socket buffer size is 64Kb. If your computed target_buffer_size is less than 64Kb, you may not need to change the default value.
  • Socket buffer size should be set to at least target_buffer_size / compression_factor to avoid a network bottleneck.
    • Where compression_factor is the compression ratio, which is usually around 2 for market data feeds.
    • If compression is disabled, compression_factor is 1.
  • Round up the result to the full 8 Kb page. It is common to use powers of 2 for values. Can be set to 2–4x of the computed value to allow for future growth.
  • For loaders, send socket buffer size should be set on the client side and receive socket buffer size should be set on the server side.
  • For cursors, send socket buffer size should be set on the server side and receive socket buffer size should be set on the client side.
  • Each connected client uses 3 sockets by default. So if you have 100 connected clients and increased socket buffer size to 2 Mb, then just socket buffers will consume 3*100*2Mb = 600Mb of memory on the server.

Setting Socket Buffer Size in TimeBase​

Socket buffer size has to be set on both client and server sides.

For the server side, "Socket Receive Buffer Size" can be set in the admin.properties file. In all other cases, a JVM option should be used.

NameJVM optionadmin.properties
Socket Send Buffer Size-DTimeBase.network.socket.sendBufferSize=BYTES(not yet supported)
Socket Receive Buffer Size-DTimeBase.network.socket.receiveBufferSize=BYTESTimeBase.network.socket.receiveBufferSize=BYTES

OS-Level Maximum Socket Buffer Size​

The maximum socket buffer size is limited by the OS settings. Setting the socket buffer size in TimeBase alone is not sufficient. You have to set net.core.rmem_max and net.core.wmem_max to the desired value on the OS level:

  • net.core.wmem_max must be not less than the "Socket Send Buffer Size" value set for TimeBase.
  • net.core.rmem_max must be not less than the "Socket Receive Buffer Size" value set for TimeBase.
  • Do not change OS settings if they already have higher values.
  • The exact way to set these values depends on the OS:

Channel Buffer Size​

  • The default channel_buffer_size is 512Kb, which is usually enough for average workloads. So if your computed target_buffer_size is less than 512Kb, you may not need to change the default value.
  • channel_buffer_size should be set to the same value as the computed target_buffer_size, or possibly up to 2–4x more if the data rate or message size is expected to increase in the future.
  • Usually should be a power of 2, but it is not strictly required.

Setting Channel Buffer Size in TimeBase​

  • Channel buffer size has to be set on the client only. It is not needed on the server side.
  • Channel buffer size can be set for each loader and cursor individually from the code.
    • Usually this is the recommended way, as an application may want to set different values for different loaders/cursors.
NameProgrammatic (Java)JVM optionAggregator Data Connector Process Config
Loader Channel Buffer SizeLoadingOptions.channelBufferSize-DTimeBase.transport.channel.remote.defaultSize=BYTES<channelBufferSize>BYTES</channelBufferSize>
Cursor Channel Buffer SizeSelectionOptions.channelBufferSize-DTimeBase.transport.channel.remote.defaultSize=BYTES(not applicable)

Examples​

Increasing Socket Buffer Size to 512Kb​

On both TimeBase client and server sides:

  • Add JVM options: -DTimeBase.network.socket.sendBufferSize=524288 -DTimeBase.network.socket.receiveBufferSize=524288
  • Check current OS settings for net.core.wmem_max and net.core.rmem_max:
    • sysctl net.core.wmem_max (may differ depending on the OS)
    • sysctl net.core.rmem_max (may differ depending on the OS)
  • Increase net.core.wmem_max and net.core.rmem_max to 524288 on the OS level (only if the current value is lower!):
    • sysctl -w net.core.wmem_max=524288 (may differ depending on the OS)
    • sysctl -w net.core.rmem_max=524288 (may differ depending on the OS)