The Quix SDK allows you to do manual checkpointing when you read data from a Topic. This gives you the ability to inform the Message Broker that you have already processed messages up to one point, usually called a checkpoint.
This is a very important concept when you are developing high-performance, streaming applications, processing tons of data in memory. You don’t want to persist a state for each message received because it would cause an unaffordable processing cost, slowing down your streaming speeds and performance.
Checkpointing lets you do some of this costly processing at a very low frequency, without having to worry about losing data. If, for some reason, your process is restarted or crashes and you haven’t saved all the in-memory data you are processing, the Message Broker will resend all the messages from the last Checkpoint when you reconnect to the topic.
Refer to the Committing / checkpointing section of this documentation to find out how to do Checkpointing when reading data with the Quix Sdk.
Let’s explain the checkpointing concept and its benefits with an easy example.
One process is reading and processing data, without saving its state after each input message. This allows good performance and high throughtput of the service but, without checkpointing, risks data loss in the case of failure.
The process reads the first four messages, keeping its state in memory.Figure 1. Reading first four messages
The process commits the messages of the topic (checkpointing) just after reading the first four and saves the in-memory state to the database.Figure 2. Checkpointing marking messages already processed
The process reads the next four messages, but it crashes just after that, without time to commit the messages. This will not result in data loss because it will begin from the last checkpoint after it restarts.Figure 3. Process crashes without commiting last messages
The process restarts and reopens the input topic. It will start reading messages from the last checkpoint resulting in no data loss from the previous crash.Figure 4. Process recover itself after restart
The process resumes reading the next five messages, keeping its state in memory.Figure 5. Process resumes reading next five messages
The process commits the messages of the topic just after reading the previous five messages and saves the in-memory state to the database.Figure 6. Checkpointing 6