Deployment
Setup
To setup TimeBase Cluster:
- Choose a Cluster layout by deciding on the following attributes:
- How to identify cluster nodes (static IP address vs. fixed hostname)
- Desired cluster size (minimum size = 3)
- Cluster member naming pattern
For more information, seeTimeBase.cluster.memberKey
in the Settings section.
- On each cluster machine, setup or copy standalone TimeBase (versions 7+).
You can use regular TimeBase Docker images with versions 7 and onward. - For each TimeBase instance, apply cluster-specific changes to
admin.properties
.
Note that some settings are instance specific, making it is impossible to copy the sameadmin.properties
to each cluster node.
Settings
The following table lists the Cluster-specific settings in admin.properties
:
Name | Required/ Optional | Default Value | Description |
---|---|---|---|
TimeBase.host | Required (if not IP is not static) | Host (static IP or hostname) of the cluster node that will be used for client connections. Used in combination with Timebase.port . | |
TimeBase.cluster.enabled | False | (boolean) Enables cluster mode. Must be true to enable cluster functionality. | |
TimeBase.cluster.memberKey | Required | Logical unique ID of cluster node. Must be unique on each cluster node. Cannot be changed after the initial setup. | |
TimeBase.cluster.memberAddress | Required | Host (static IP or hostname) and port to a cluster node for cluster-specific internal protocol. The port must be different to Timebase.port . Must be unique on each cluster node. | |
TimeBase.cluster.allServers | Required | Comma separated list of hosts and ports for all known cluster nodes. The the value from TimeBase.cluster.memberAddress must be in the list. Data from that list is used only when cluster gets started for the first time. | |
TimeBase.cluster.group | Required | Cluster group id. Must be same for all cluster nodes. | |
TimeBase.cluster.stream.replicationFactor | Optional | 2 for multi-node cluster | A positive integer number determines the default replication factor for all new streams nodes a stream is written to. In the special case of 1, node cluster defaults to 1. |
TimeBase.cluster.blockAllocatorType | Optional | DEFAULT | Determines block allocation strategy. This option should not be changed in production. Possible values: - DEFAULT - PREDICTABLE - MEMBER_KEY |
The block allocation strategy types for TimeBase.cluster.blockAllocatorType
are as follows.
DEFAULT
: Main allocator implementation. Should be used in all production setups.PREDICTABLE
: Similar to the default allocator but has RNG turned off. More predictable and can be used for some tests.MEMBER_KEY
: Uses a trivial allocator that only considersmemberKey
for allocation priority. Suitable for tests only.
caution
After the initial setup, all cluster-specific settings, excluding TimeBase.cluster.stream.replicationFactor
, cannot be changed.
Examples
This section provides examples of various admin.properties
setups.
A configuration with a static IP address:
TimeBase.cluster.enabled=true
TimeBase.cluster.group=tbcluster
TimeBase.cluster.memberKey=member1
TimeBase.cluster.memberAddress=127.0.0.1:7081
TimeBase.cluster.allServers=127.0.0.1:7081,127.0.0.1:7082,127.0.0.1:7083
A configuration with a static hostname:
TimeBase.cluster.enabled=true
TimeBase.cluster.group=tbcluster
TimeBase.cluster.memberKey=member1
TimeBase.cluster.memberAddress=member1.cluster1.localnet:7080
TimeBase.cluster.allServers=member1.cluster1.localnet:7080,member2.cluster1.localnet:7080,member3.cluster1.localnet:7080
An example with additional options:
TimeBase.cluster.stream.replicationFactor=2
TimeBase.cluster.blockAllocatorType=DEFAULT
Connecting to the Cluster
To connect to the TimeBase cluster, specify the connection url according to the following:
scheme "://" [[user:password]"@"] host1 [":" port1] | host2 [":" port2] | ... | hostN [":" portN]
The scheme can be dxctick
or dsctick
(SSL communication).
To create a TimeBase client in Java Code, use the following code fragment:
DXTickDB client = TickDBFactory.createFromUrl("dxctick://host1:9000|host2:9000|host3:9000");
Aggregator
To connect to a cluster, update the TimeBase aggregator settings to:
TimeBase.host=<any_cluster_node_hostname>
TimeBase.port=8011
TimeBase.cluster.enabled=true
Maintenance
Adding New Nodes (Scale-up)
To add new nodes to cluster, prepare the new servers with the same TimeBase configuration as the existing cluster members, but with the following changes:
- For each new member, set a unique value for
TimeBase.cluster.memberKey
. - Modify
TimeBase.cluster.memberAddress
to contain the full new member list.
All old and new cluster members members must be in the list. - Start all the new members.
- Unless you have the
TimeBase.cluster.autoRegisterAsPeer
option set, for each newMEMBER_ADDRESS
, execute the following REST request to any old live cluster members:/tb/cluster/add?memberPeerId=MEMBER_ADDRESS
.
To ensure that the new members appear in the Peer list, check the cluster web stats page.
Removing Nodes (Scale-down)
This functionality will be available with TimeBase version 7.3.
You can remove nodes from cluster under the following conditions:
- All streams have a replication factor of 2 or more.
- Temporary data unavailability is acceptable. The duration of the period when data is unavailable is proportional to the time that is needed to copy all the data from the servers that will be removed.
- There is enough disk space on other cluster members to store all the data from the members that are going to be removed.
- Enough cluster members will remain to keep up with the defined replication factor for all streams.
- At least 3 members will remain in the cluster.
If all of the conditions are met, you can use the following steps to remove nodes:
- Enable a "Maintenance State" on all the servers you want to remove by sending the following REST request to any live cluster member:
/tb/cluster/turn-on/maintenance?memberKey=MEMBER_KEY&duration=86400000&turnOffOnNodeRestart=false&cause=Decomission&overridesExisting=true
- On each server you want to remove, execute the following:
- Turn off the server.
- Wait for failover procedures to complete. All offline replications have to stop.
You can check the status on the Web UI.
Recommend Configuration
The minimum cluster size is 3 servers. The recommended cluster size depends on the number of expected active connections and throughput.
It's recommended to have an odd cluster size (3,5,7...) because with an even numbered cluster member (4,6,8...), there is no additional gain on the number of failures that the cluster can tolerate. However, this is not important for clusters larger than 8 servers.
Throughput
Throughput = (Single Node Throughput Node Count) / (Replication Factor K), where K can be estimated as 2.
For example, if you want to get 2x the throughput of a standalone TB server and you plan to use RF=2, the cluster size you need is 8.
Replication Factor
The Replication Factor (RF) should be at least 2 and a maximum of 4.
The advantage of having a higher replication factor is that it provides resilience for your system. If the replication factor is N, up to N-1 nodes may fail without impacting availability.
The disadvantages of having a higher replication factor include:
- Higher latency experienced by the producers, as the data needs to be replicated to all the replica brokers before an ack.
- More disk space required on your system because each message is replicated N times.