Smooth scaling: Slack’s journey toward a new database
Slack has experienced tremendous growth for a young company, serving over nine million weekly active customers. But with great growth comes greater growth pains. Slack’s rapid growth over the last few years outpaced the scaling capacity of its original sharded MySQL database, which negatively impacted the company’s customers and engineers.
Ameet Kotian explains how a small team of engineers embarked on a journey for the right database solution, which eventually led them to Vitess, a powerful open source database cluster solution for MySQL. Vitess combines the features of MySQL with the scalability of a NoSQL database. It has been serving Youtube’s traffic for numerous years and has a strong community.
Although Vitess meets a lot of Slack’s needs, it’s not an out-of-the-box solution. Ameet shares how the journey to Vitess was planned and executed, with little customer impact, in the face of piling operational challenges, such as AWS issues, MySQL replication, automatic failovers, deployments strategies, and so forth. Ameet also covers Vitess’s architecture, trade-offs, and what the future of Vitess looks like at Slack.
Ameet Kotkian, senior storage operations engineer at Slack, shows us how Slack uses Vitess.
Slack is embarking on a major migration of the MySQL infrastructure at the core of our service to use Vitess' flexible sharding and management instead of our simple application-based shard routing and manual administration. This effort is driven by the need for an architecture that scales to meet the growing demands of our largest customers and features under the pressure to maintain a stable and performant service that executes billions of MySQL transactions per hour. This talk will present the driving motivations behind the change, why Vitess won out as the best option, and how we went about laying the groundwork for the switch. Finally, we will discuss some challenges and surprises (both good and bad) found during our initial migration efforts, and suggest some ways in which the Vitess ecosystem can improve that will aid future migration efforts.
Sugu and Anthony showed what it looks like to use Vitess now that Keyspace IDs can be completely hidden from the application. They gave a live demo of resharding the Guestbook sample app, which now knows nothing about shards, and explained how new features in VTGate make all of this possible.
Vitess team member Anthony Yeh’s talk at
the January 2016 CoreOS Meetup
discussed challenges and techniques for running distributed databases
within Kubernetes, followed by a deep dive into the design trade-offs
of the Vitess on Kubernetes deployment templates.
Please note the Vitess on Kubernetes deployment templates were removed as of February 27, 2020.
Vitess team member Anthony Yeh’s talk at Oracle OpenWorld 2015 focused on what the Cloud Native Computing paradigm means when applied to MySQL in the cloud. The talk also included a deep dive into transparent, live resharding, one of the key
features of Vitess that makes it well-adapted for a Cloud Native environment.
Vitess team member Anthony Yeh’s talk at Percona Live 2015 provided an overview of Vitess as well as an explanation of how Vitess has evolved to live in a containerized world with Kubernetes and Docker.
Google I/O 2014 - Scaling with Go: YouTube’s Vitess #
In this talk, Sugu Sougoumarane from the Vitess team talks about how Vitess solved YouTube’s scalability problems as well as about tips and techniques used to scale with Go.