Scylla’s Journey Towards Being an Elastic Cloud Native Database

Editor's Notes

1. Hi, Thanks for joining my session on the advances we have done toward making scylla and Elastic Cloud Native Database
2. My name is Shlomi and I manage the development team at Scylla, lets get to it
3. Elasticity - is the ability to grow and shrink the cluster. Our goal is to make the process of adding or removing a node as fast as possible while keeping the impact on the ongoing workload cluster minimal
4. Lets look at an example The two pictures show the 95%, 99% latency of an ongoing read workload running on a 6 node cluster as its beeing resized to 9 nodes. We can see that in Cluster 1 as the cluster is grown there is an effect on latency while on Cluster 2 under the same exact workload the impact as its being resized is smaller At times the need to resize a cluster is known in advance and can be pre planned one common example is the resize many retailers do toward BlackFriday. Yet success (and load accompanying it) can not always be foreseen - in such cases scaling must be done under load while striving to have as little effect as possible on the ongoing workload. We consider the ability to resize a cluster fast while assuring impact of the ongoing workload is minimal mandatory to enable auto scaling.
5. Ltes look at what needs to be done after a new node has been added to a cluster Existing nodes in the cluster transfer parts of the data they own to the new node
6. As this data is beeing transferred ongoing writes to the cluster to transferred parts must also be replicated onto the new node
7. Once the data has been fully transferred the new node can be transitioned to being a regular node and start beeing used to serve reads
8. So a new node beeing added into the cluster has two states In the joining state its receiving the data it will own (and updated to that data) Once it has all the data it switches state to normal
9. Going back to Cluster 1, that is using Scylla 4.2.2 - latest open source release, the green and yellow lines are the 95% and 99% latency of the ongoing workload the additional Orange and Light Blue lines mark transitions over the cluster life times The Orange lines split the timeline of the cluster based on the cluster total size, total node count, 6,7,8,9 nodes (joining_normal) The Light Blue line splits the timeline in which the cluster had 6,7,8,9 nodes in normal state → serving reads As can be seen There is high correlation between a node switching state and an increase in read latency. Lets look at overall optimization we can do in adding a new node
10. To summarize Existing nodes: Handle ongoing traffic Transfer ranges to new node Writes to transfered ranges need to be written to new node New node (state joining) DOES NOT SERVE READS Manage the transfer process Receive transferred ranges Receive “regular” writes
11. The first property we can exploit is that the node in joining state is not used for reads - as such sstables created from the received regular writes can be compacted less “aggressivly” In addition the data transferred from existsing nodes does not have overwrites and will not be used for reads in joining state - there is no need to compact those sstables. Overall we can reduce compaction and use the saved Disk I/O and CPU for “faster” completion of the bootstrap process.
12. The new node controls the process of data transfer - it requests other nodes to send it ranges of data. As we have freed the CPU and DISK I/O the new node can now increase the pace in requesting data transfer to a level that it has an effect on the ongoing workload. We need to rate limit this process to assure the impact to the ongoing workload on the existing nodes will be minimal. All data transfer management operations (including bootstrap) have been moved to be repair based - this assures us better correctness for some operations and in the case of adding a node - enables resumable bootstraps. So far we have looked at the data transfer aspects - lets see what can be done when the node switches from joining to normal state
13. As we move the node to normal state it will be used to serve read requests. However, the new node is at loss compared to other nodes in the cluster, its cache is cold. We have introduced a few years ago heat weighted load balanceing that uses the ache hit ratio as measure to be used in splitting the ongoing work load As we have saw in cluster 1 that is not enough - the cache miss reads from disk of the new node are slower compared to the other nodes. The reason being the data is not shaped aligning to the compaction strategy. Lets fix that.
14. We will introduce an additional state after all the data has been transferred in which the data will be reshaped. Only once the data has been reshaped will we switch the state
15. Employing all of this we reach to a state in which elasticity may be faster and the impact on the ongoing workload of adding the node is smaller
16. So far I have covered items we have done to improve elasticity that have been done and are beeing rolled out to the next Scylla Open Source Release While auto scaling will now be possible - we are not done, We are pklanning to further improve elasiticy. The first enhancement i’ll discuss is multiple node topology changes - that is the ability to shrink / grow the cluster while adding removing multiple nodes.
17. Lets look at an example of what needs to be done today to double a 3 node cluster to 6 nodes Today we need to iteratively add one node till we reach the desired size. So a new node (node 4) that is added into the cluster will receive information from the existing nodes
18. Once it moved to normal state another node may be added (node 5) And node 5 will receive information from all the old nodes and some data from node 4 that was just added
19. Once it moved to normal state another node may be added (node 6) And node 6 will receive information from all the old nodes and some data from the last 2 nodes that were “just’ added
20. So if we are looking at a specific range that ended up on node 6 it may have had a “long route” by beeing first moved to node 4 and then moved to node 5. And that was later moved to node 6 (e.g.) -
21. If we had known that we will be adding 3 nodes we could have precomputed the end data location and have the data transferred directly to the 6 node
22. We are planning to add the capability of adding multiple nodes at the same time that will optimize the data movment and will allow “faster” scaling.
23. Another enhnacmenet a bit further in our roadmap is to move to tablets Data in tables today that is spread across the cluster is split for all tables in the same manner - the split is based on token ranges. Tablets will allow breaking tables based on their data density - so that small tables will split into fewer parts (and be placed potentially on fewer nodes) . Tablets will also enable more control over the resize process - after a tablet is moved from existing node to a new node it can start to be used for reads.
24. Thanks for attending the session