Kirana Club is experiencing rapid growth, and while it brings a lot of excitement, it also presents certain challenges, one of which has already been discussed here. The challenge we are going to discuss about today is saving the impressions of different kind of items Today, we’re addressing another challenge related to storing impression data of different type of content shown to users. This data is crucial for our recommendation systems, guiding the personalized content we deliver to each user. We’ll explore why Kirana Club chose Scylla DB to tackle this aspect.
A little background about Kirana Club (the application).
Kirana Club is as a community application tailored for Kirana Store owners. Upon logging in, users gain access to valuable information shared by other users and our quality researchers. This information comes in various forms such as text posts, video posts, banners/infographics, surveys, and more. Internally, these are all referred to as “posts,” each identified by a varchar postID.
When I mention storing impressions of these posts, I’m referring to recording which user viewed which post at a specific time. As mentioned earlier, this data plays a crucial role in guiding our recommendation systems to deliver personalized content to users.
What is the scale?
First things first, we are not just building to handle the current scale, we are building to handle the scale we expect in the coming 2-3 years. Currently, the peak we have received is 520 ops/sec . While it can be managed with Mysql, with the exponential growth we have seen , we need to be future safe.
Why not MYSQL?
MySQL has been our go-to primary database, handling various types of data frequently queried by our backend systems. However, when it came to impressions, we encountered a few challenges.
The most prominent issue was the occurrence of spikes in write latency. While latency was typically predictable for the majority of the day, during peak hours, we experienced unpredictable spikes in write latency. Specifically, when our team sent notifications or ran campaigns, causing a substantial increase in the number of users on the application, the load on the MySQL database surged correspondingly.
Second was increased I/O , which further impacted the read time of other queries.
There were some big hints, while it was still managable on Mysql, we had to migrate to a dedicated data storage to save our world from an apocalypse.
Why Scylla?
Scylla DB is based on Cassandra architecture that is tailor made for write-heavy systems.
Let us discuss about what architectural choices makes Scylla a better option for write heavy system.
- Log-structured storage engine:
Scylla uses log structured storage engine, it means new data is appended to a write-ahead log (WAL) in a sequential manner before being committed to storage. While MySql first search for empty space big enough for the new data and adds the data there. Thus causing an overhead in I/O time. - Asynchronous Commit Log:
Scylla uses an asynchronous commit log for durability. Write acknowledgments sent to the client are very quick, as the system doesn’t wait for data to be fully written to disk before confirming.
MySQL on the other hand typically uses synchronous commit, waiting for confirmation that data is safely written to disk before acknowledging the write operation. - Write Amplification Reduction:
Scylla minimizes write amplification by appending only new data to the log-structured segments. This approach significantly reduces the overall volume of writes.
MySQL, by its nature, might rewrite entire data pages during updates, leading to higher write amplification, more disk I/O, and slower write operations. - Scaling:
When it comes to MySql, you can only scale up, that is increase the compute capacity (RAM, Storage, CPU, etc ) of the machine, which has physical limits and cannot be increased beyond a certain limit.
Scylla on the other hand can be scaled out, you can very easily add one or two new VMs as nodes.
Journey of a write operation in Scylla
The aforementioned points can be a bit difficult to understand , so let us now try to look at the complete journey of a write operation in Scylla DB. Please note, we will not be going in depth of architecture of Scylla DB. The official Scylla Website has really good documentation about it.
Here’s a step-by-step explanation of how a write operation is performed in ScyllaDB:
1. Client Sends a Write Request:
The write process begins when a client sends a write request to a node in the ScyllaDB cluster. Clients can connect to any node in the cluster, as ScyllaDB uses a peer-to-peer architecture where all nodes are equal.
2. Coordinator Node Determination:
The receiving node becomes the coordinator for the write operation. The coordinator is responsible for coordinating the write across the nodes in the cluster.
3. Partition Key Extraction:
The coordinator extracts the partition key from the write request. The partition key is a crucial component of ScyllaDB’s data distribution mechanism, as data is sharded or partitioned based on this key.
4. Consistency Level Configuration:
The client specifies the consistency level for the write operation. The consistency level determines how many replicas must acknowledge the write for it to be considered successful. ScyllaDB supports various consistency levels, such as “ONE,” “QUORUM,” “LOCAL_QUORUM,” etc.
5. Node Selection:
The coordinator determines which nodes in the cluster are responsible for the specified partition key. This is done through the use of a partitioner and the cluster’s token ring. The token ring helps distribute data across nodes.
6. Replication Factor Consideration:
Based on the replication factor (the number of replicas configured for the keyspace), the coordinator selects the appropriate number of nodes that need to acknowledge the write to achieve the specified consistency level.
7. Write to Commit Log:
The coordinator initiates the write by first appending the data to the commit log. The commit log is an append-only log file that ensures durability. This write to the commit log provides a record of the write operation.
8. Write to MemTable:
Simultaneously, the data is added to an in-memory data structure called the MemTable. This in-memory structure allows for fast write operations, providing low-latency responses.
9. Acknowledgment to Client:
Once the write has been successfully written to the commit log and MemTable on the coordinator node, an acknowledgment is sent back to the client. This acknowledgment does not wait for the data to be flushed to disk (asynchronously committed).
10. Replication to Replica Nodes:
The coordinator then forwards the write request to the selected replica nodes responsible for other copies of the data. These replica nodes follow a similar process of writing to their commit logs and MemTables.
11. Commit Log Flush and SSTable Creation:
In the background, ScyllaDB periodically flushes the MemTable to disk, creating an SSTable (Sorted String Table). This process ensures that data is durable and persisted on disk.
As you can see here, Acknowledgement to Client is sent before the replication, this is makes Scylla Eventually Consistent and not Strongly Consistent. But again, given the nature of our data, Eventual Consistency is all we need.
Complete System Design
Storing impressions in a very high frequency, but not very realtime operation. To streamline it, we use a worker queue. The design is very straight forward and simple to understand.
Let us see the components first:
Components:
- Client [Mobile App]
- Worker Queue
- API to insert to queue
- Workers
- Scylla DB
There are some configurable parameters of these components, like the number of workers that listen to the queue, the retention of the queue, etc . We keep playing with these numbers to get the optimum performace that we need. Now that we know the components, let us look at the flow that binds these components.
Flow:
- The client maintains an in memory list of all the posts that have come on the screen. When the length of list is more than 10, or when the user closes the application, it hits an API with the list of posts.
- The API just enqueues the request data into a worker queue and returns 200 to the Client.
- A bunch of workers keep listening to the queue, and as soon as any data comes into the queue, the workers validate, transform and store the data in Scylla DB.
And this is how and why Kirana Club chose Scylla DB at Kirana Club.
0 Comments