Skip to main content

Streaming Throughput

This article examines the "durable" streaming performance of TimeBase, specifically how overall throughput behaves as number of producer and consumer pairs grow.

Summary​

For described data flow TimeBase throughput goes up as we add more producer-consumer pairs, but levels off around 11–12M msg/s once we reach 16 producer-consumer pairs.

Environment​

  • CPU: Intel(R) Xeon(R) Platinum 8488C, 64 vCPUs (m7i.16xlarge)
  • RAM: 256G
  • Disk: GP3 EBS (3000 IOPS)
  • Network: 25 Gigabit
  • OS: Amazon Linux 2023
  • OpenJDK 17.0.16
  • TimeBase 5.6.161 (default settings)
  • TimeBase.network.socket.receiveBufferSize=4194304
  • TimeBase.ramCacheSize=10737418240 (10G)
  • Storage format: 5.0

The following diagram illustrate this test setup:

Server:

java -Xms60G -Xmx60G -XX:+AlwaysPreTouch $ADD_OPENS -cp "/home/ec2-user/timebase-home/lib/*" deltix.qsrv.comm.cat.TBServerCmd -tb -home /home/ec2-user/QSHome -port 8011

Client:

java -DTimeBase.network.socket.bufferSize=4194304 $ADD_OPENS -cp "/home/ec2-user/jars-bench/*" deltix.qsrv.hf.tickdb.tool.perf.thr.Benchmark_LiveThroughput -url dxtick://172.31.17.217:8011 -duration 60 -cooldown 60 -channelType streams -producers 16 -consumers 1 -stream thr

Data​

In this test we were streaming trades (average payload size encoded in 36 bytes).

Results​

PairsTotal_Msg_s_MedianPer_Consumer_Msg_s_Median
11.67 M1.67 M
44.74 M1.19 M
88.82 M1.10 M
1611.48 M0.72 M
2411.39 M0.47 M

  • The results show that total throughput scales well as the number of producer-consumer pairs increases, but the gains taper off after about 16 pairs, where throughput begins to saturate around 11–12 million messages per second.

  • At the same time, the per-consumer throughput declines steadily, dropping from ~1.7M msg/s for a single pair to under 0.5M msg/s with 24 pairs.

Appendix: Raw results​

Trades, Live + .find fix:
 
1Px1C:
===== 
Messages per second (total): 1 763 302
Messages per consumer per second: 1 763 302
 
Messages per second (total): 1 552 984
Messages per consumer per second: 1 552 984

Messages per second (total): 1 673 764
Messages per consumer per second: 1 673 764
 
 
4Px1C:
===== 
Messages per second (total): 4 741 933
Messages per consumer per second: 1 185 483

Messages per second (total): 4 890 772
Messages per consumer per second: 1 222 693
 
Messages per second (total): 4 539 211
Messages per consumer per second: 1 134 802

8Px1C:
===== 
Messages per second (total): 9 187 492
Messages per consumer per second: 1 148 436

Messages per second (total): 8 821 003
Messages per consumer per second: 1 102 625
 
Messages per second (total): 8 124 136
Messages per consumer per second: 1 015 517

16Px1C:
======= 
Messages per second (total): 11 479 381
Messages per consumer per second: 717 461

Messages per second (total): 11 479 381
Messages per consumer per second: 717 461

Messages per second (total): 11 380 460
Messages per consumer per second: 711278


24Px1C:
====== 
Messages per second (total): 11 394 100
Messages per consumer per second: 474 754
 
Messages per second (total): 11 394100
Messages per consumer per second: 474 754
 
Messages per second (total): 11 249 920
Messages per consumer per second: 468 746