Vitess messaging gives the application an easy way to schedule and manage work
that needs to be performed asynchronously. Under the covers, messages are
stored in a traditional MySQL table and therefore enjoy the following
Scalable: Because of vitess’s sharding abilities, messages can scale to
very large QPS or sizes.
Guaranteed delivery: A message will be indefinitely retried until a
successful ack is received.
Non-blocking: If the sending is backlogged, new messages continue to be
accepted for eventual delivery.
Adaptive: Messages that fail delivery are backed off exponentially.
Analytics: The retention period for messages is dictated by the
application. One could potentially choose to never delete any messages and
use the data for performing analytics.
Transactional: Messages can be created or acked as part of an existing
transaction. The action will complete only if the commit succeeds.
The properties of a message are chosen by the application. However, every
message needs a uniquely identifiable key. If the messages are stored in a
sharded table, the key must also be the primary vindex of the table.
Although messages will generally be delivered in the order they’re created,
this is not an explicit guarantee of the system. The focus is more on keeping
track of the work that needs to be done and ensuring that it was performed.
Messages are good for:
Handing off work to another system.
Recording potentially time-consuming work that needs to be done
Scheduling for future delivery.
Accumulating work that could be done during off-peak hours.
Messages are not a good fit for the following use cases:
These inserts can be part of a regular transaction. Multiple messages can be
inserted to different tables. Avoid accumulating too many big messages within a
transaction as it consumes memory on the VTTablet side. At the time of commit,
memory permitting, all messages are instantly enqueued to be sent.
Messages can also be created to be sent in the future:
Processes can subscribe to receive messages by sending a MessageStream
request to VTGate. If there are multiple subscribers, the messages will be
delivered in a round-robin fashion. Note that this is not a broadcast; Each
message will be sent to at most one subscriber.
The format for messages is the same as a vitess Result. This means that
standard database tools that understand query results can also be message
recipients. Currently, there is no SQL format for subscribing to messages, but
one will be provided soon.
It’s possible that you may want to subscribe to specific shards or groups of
shards while requesting messages. This is useful for partitioning or load
balancing. The MessageStream API allows you to specify these constraints. The
request parameters are as follows:
Name: Name of the message table.
Keyspace: Keyspace where the message table is present.
Shard: For unsharded keyspaces, this is usually “0”. However, an empty
shard will also work. For sharded keyspaces, a specific shard name can be
KeyRange: If the keyspace is sharded, streaming will be performed only from
the shards that match the range. This must be an exact match.
A message that was successfully sent will wait for the specified ack wait time.
If no ack is received by then, it will be resent. The next attempt will be 2x
the previous wait, and this delay is doubled for every attempt.
The MessageAck functionality is currently an API call and cannot be used
inside a transaction. However, you can ack messages using a regular DML. It
should look like this:
update my_message set time_acked = :time_acked, time_next = nullwhere id in ::ids and time_acked isnull
You can manually change the schedule of existing messages with a statement like
update my_message set time_next = :time_next, epoch = :epoch where id in ::ids and time_acked isnull
This comes in handy if a bunch of messages had chronic failures and got
postponed to the distant future. If the root cause of the problem was fixed,
the application could reschedule them to be delivered immediately. You can also
optionally change the epoch. Lower epoch values increase the priority of the
message and the back-off is less aggressive.
You can also view messages using regular select queries.