A Closer Look at Apache Kudu

Editor's Notes

1. Structured storage in the Hadoop ecosystem has typically been achieved in two ways: for static data sets, data is typically stored on HDFS using binary data formats such as Apache Avro[1] or Apache Parquet[3]. However, neither HDFS nor these formats has any provision for updating indi- vidual records, or for efficient random access. Mutable data sets are typically stored in semi-structured stores such as Apache HBase[2] or Apache Cassandra[21]. These systems allow for low-latency record-level reads and writes, but lag far behind the static file formats in terms of sequential read throughput for applications such as SQL-based analytics or machine learning.
2. Following design of BigTable Kudu relies on: A single Master server, responsible for metadata Can be replicated for fault tolerance An arbitrary number of Tablet Servers, responsible for data
3. When READ_LATEST is specified the server will always return committed writes at the time the request was received. This type of read does not return a snapshot timestamp and is not repeatable. In ACID terms this corresponds to Isolation mode: "Read Committed" This is the default mode. Monotonic reads [19] is a guarantee that this kind of anomaly does not happen. It’s a lesser guarantee than strong consistency, but a stronger guarantee than eventual con‐ sistency. When you read data, you may see an old value; monotonic reads only means that if one user makes several reads in sequence, they will not see time go backwards, i.e. they will not read older data after having previously read newer data. In this situation, we need read-after-write consistency, also known as read-your-writes consistency [20]. This is a guarantee that if the user reloads the page, they will always see any updates they submitted themselves. It makes no promises about other users: other users’ updates may not be visible until some later time. However, it reassures the user that their own input has been saved correctly.
4. When READ_LATEST is specified the server will always return committed writes at the time the request was received. This type of read does not return a snapshot timestamp and is not repeatable. In ACID terms this corresponds to Isolation mode: "Read Committed" This is the default mode.
5. By default, Kudu does not provide an external consistency guarantee. That is to say, if a client performs a write, then communicates with a di↵erent client via an external mecha- nism (e.g. a message bus) and the other performs a write, the causal dependence between the two writes is not captured. A third reader may see a snapshot which contains the second write without the first However, for users who require a stronger guarantee, Kudu o↵ers the option to man- ually propagate timestamps between clients: after performing a write, the user may ask the client library for a timestamp to- ken. This token may be propagated to another client through the external channel, and passed to the Kudu API on the other side, thus preserving the causal relationship between writes made across the two clients. In this situation, we need read-after-write consistency, also known as read-your-writes consistency [20]. This is a guarantee that if the user reloads the page, they will always see any updates they submitted themselves. It makes no promises about other users: other users’ updates may not be visible until some later time. However, it reassures the user that their own input has been saved correctly.
6. By default, Kudu does not provide an external consistency guarantee. That is to say, if a client performs a write, then communicates with a di↵erent client via an external mecha- nism (e.g. a message bus) and the other performs a write, the causal dependence between the two writes is not captured. A third reader may see a snapshot which contains the second write without the first However, for users who require a stronger guarantee, Kudu o↵ers the option to man- ually propagate timestamps between clients: after performing a write, the user may ask the client library for a timestamp to- ken. This token may be propagated to another client through the external channel, and passed to the Kudu API on the other side, thus preserving the causal relationship between writes made across the two clients.
7. By default, Kudu does not provide an external consistency guarantee. That is to say, if a client performs a write, then communicates with a di↵erent client via an external mecha- nism (e.g. a message bus) and the other performs a write, the causal dependence between the two writes is not captured. A third reader may see a snapshot which contains the second write without the first However, for users who require a stronger guarantee, Kudu o↵ers the option to man- ually propagate timestamps between clients: after performing a write, the user may ask the client library for a timestamp to- ken. This token may be propagated to another client through the external channel, and passed to the Kudu API on the other side, thus preserving the causal relationship between writes made across the two clients.