Backup and Restore

Backup and Restore are integrated features provided by tablets managed by Vitess. As well as using backups for data integrity, Vitess will also create and restore backups for provisioning new tablets in an existing shard.

Concepts #

Vitess supports pluggable interfaces for both Backup Storage Services and Backup Engines.

Before backing up or restoring a tablet, you need to ensure that the tablet is aware of the Backup Storage system and Backup engine that you are using. To do so, use the following command-line flags when starting a vttablet or vtctld that has access to the location where you are storing backups.

Backup Storage Services #

Currently, Vitess has plugins for:

  • File (using a path on shared storage, e.g. an NFS mount)
  • Google Cloud Storage
  • Amazon S3
  • Ceph

Backup Engines #

The engine is the techology used for generating the backup. Currently Vitess has plugins for:

  • Builtin: Shutdown an instance and copy all the database files (default)
  • XtraBackup: An online backup using Percona’s XtraBackup

VTTablet and Vtctld configuration #

The following options can be used to configure VTTablet and Vtctld for backups:

Flags
backup_storage_implementationSpecifies the implementation of the Backup Storage interface to use.

Current plugin options available are:
  • file: Using shared storage (e.g. NFS).
  • gcs: Google Cloud Storage.
  • s3: Amazon S3.
  • ceph: Ceph Object Gateway S3 API.
backup_engine_implementationSpecifies the implementation of the Backup Engine to use.

Current options available are:
  • builtin: Copy all the database files into specified storage. This is the default.
  • xtrabackup: Percona XtraBackup.
backup_storage_hookIf set, the content of every file to backup is sent to a hook. The hook receives the data for each file on stdin. It should echo the transformed data to stdout. Anything the hook prints to stderr will be printed in the vttablet logs.
Hooks should be located in the vthook subdirectory of the VTROOT directory.
The hook receives a -operation write or a -operation read parameter depending on the direction of the data processing. For instance, write would be for encryption, and read would be for decryption.
backup_storage_compressThis flag controls if the backups are compressed by the Vitess code. By default it is set to true. Use -backup_storage_compress=false to disable.
This is meant to be used with a -backup_storage_hook hook that already compresses the data, to avoid compressing the data twice.
file_backup_storage_rootFor the file plugin, this identifies the root directory for backups. This path must exist on shared storage to provide a global backup view for all vtctlds and vttablets.
gcs_backup_storage_bucketFor the gcs plugin, this identifies the bucket to use.
s3_backup_aws_regionFor the s3 plugin, this identifies the AWS region.
s3_backup_storage_bucketFor the s3 plugin, this identifies the AWS S3 bucket.
ceph_backup_storage_configFor the ceph plugin, this identifies the path to a text file with a JSON object as configuration. The JSON object requires the following keys: accessKey, secretKey, endPoint and useSSL. Bucket name is computed from keyspace name and shard name is separated for different keyspaces / shards.
restore_from_backupIndicates that, when started with an empty MySQL instance, the tablet should restore the most recent backup from the specified storage plugin.
restore_from_backup_tsIf set, restore the latest backup taken at or before this timestamp rather than using the most recent one. Example: β€˜2021-04-29.133050’. (Vitess 12.0+)
xbstream_restore_flagsThe flags to pass to the xbstream command during restore. These should be space separated and will be added to the end of the command. These need to match the ones used for backup e.g. --compress / --decompress, --encrypt / --decrypt
xtrabackup_root_pathFor the xtrabackup backup engine, directory location of the xtrabackup executable, e.g., `/usr/bin`
xtrabackup_backup_flagsFor the xtrabackup backup engine, flags to pass to the backup command. These should be space separated and will be added to the end of the command.
xtrabackup_stream_modeFor the xtrabackup backup engine, which mode to use if streaming, valid values are tar and xbstream. Defaults to tar.
xtrabackup_userFor the xtrabackup backup engine, required user that xtrabackup will use to connect to the database server. This user must have all necessary privileges. For details, please refer to xtrabackup documentation.
xtrabackup_stripesFor the xtrabackup backup engine, if greater than 0, use data striping across this many destination files to parallelize data transfer and decompression.
xtrabackup_stripe_block_sizeFor the xtrabackup backup engine, size in bytes of each block that gets sent to a given stripe before rotating to the next stripe. Defaults to 102400.
xtrabackup_prepare_flagsFlags to pass to the prepare command. These should be space separated and will be added to the end of the command.

Authentication #

Note that for the Google Cloud Storage plugin, we currently only support Application Default Credentials. This means that access to Google Cloud Storage (GCS) is automatically granted by virtue of the fact that you’re already running within Google Compute Engine (GCE) or Google Kubernetes Engine (GKE).

For this to work, the GCE instances must have been created with the scope that grants read-write access to GCS. When using GKE, you can do this for all the instances it creates by adding --scopes storage-rw to the gcloud container clusters create command.

Backup Frequency #

We recommend to take backups regularly – e.g. you should set up a cron job for it.

To determine the proper frequency for creating backups, consider the amount of time that you keep replication logs (see the binlog_expire_logs variables) and allow enough time to investigate and fix problems in the event that a backup operation fails.

For example, suppose you typically keep four days of replication logs and you create daily backups. In that case, even if a backup fails, you have at least a couple of days from the time of the failure to investigate and fix the problem.

Concurrency #

The backup and restore processes simultaneously copy and either compress or decompress multiple files to increase throughput. You can control the concurrency using command-line flags:

  • The vtctl Backup command uses the -concurrency flag.
  • vttablet uses the -restore_concurrency flag.

If the network link is fast enough, the concurrency matches the CPU usage of the process during the backup or restore process.