Thoughts and Updates from the Hedvig Blog

Subscribe to the blog
Get updates sent directly to your inbox

Hedvig integration with ClusterHQ Flocker

by Gaurav Yadav on August 26, 2015

We're excited about today's announcement of the partnership between Hedvig and ClusterHQ, resulting in our jointly developed Hedvig storage driver for Flocker. Given the massive uptake of Dockerized applications and growing demands for the ability to persist and migrate data along with container movement, we collaborated with ClusterHQ to integrate Hedvig’s software-defined storage solution with Flocker. Flocker provides an unique capability for volumes created inside docker containers. It enables volumes to move between Docker hosts along with containers and also provides multi-host access to containerized applications.

Using the Hedvig-Flocker driver, users can not only achieve persistent storage for Docker applications, they can also take advantage of other Hedvig storage features such as deduplication, compression, and replication. Hedvig provides a raw virtual block disk, which can be formatted using any file system and used for storing any kind of data.

Hedvig-Flocker Docker driver

How does the Hedvig-Flocker driver work?

Steps to use a Hedvig Virtual Disk (vdisk) inside Docker containers:

  1. Flocker requests a new volume from Hedvig, which in turn creates a new Hedvig block vdisk.

  2. Different parameters like deduplication, compression, replication factor and block size can be set along with each volume, using flocker config.

  3. This vdisk can then be attached to any host. Flocker then creates a filesystem on this vdisk and mounts it using the path provided by user. The filesystem type can also be configured by the user.

  4. All I/O to the volume will now go through Hedvig vdisk/storage

  5. As the container moves in the environment, the vdisk will be automatically detached and reattached to any host and data will be persisted using the policies chosen during volume creation.

 

What Hedvig commands are mapped to Flocker?

Description of api calls implemented as part of hedvig driver

  1. create_volume: Creates a new hedvig block vdisk. This is equivalent to calling mkvdisk from the Hedvig CLI or creating a new vdisk using the web interface

  2. attach_volume: Attaches a vdisk to a Docker host and container. This is accomplished in two steps:

    1. addlun: Creates a new block vdisk LUN and specifies which “controller” to use for the vdisk. The Hedvig “controller” is simply used to convert block, file and object operations to unified Hedvig calls. This enables a single platform to be used to store data for different storage types and avoids storage silos. It also provides the flexibility to add new storage types in future without any underlying architecture disruption.

    2. addaccess: Provides I/O access for any given vdisk to a node.

  3. get_device_path: Returns the device path in the current node on which the vdisk is mounted. This is simply done using the “iscsiadm“ command.

  4. detach_volume: Detaches a volume from the node. This simply removes vdisk access from the node but data is still persisted and can be used again in future by using ‘attach_volume’ to any node.

  5. destroy_volume: Deletes the vdisk so that corresponding data can not be used anymore. This is instantaneous and due to the way we physically store the data, there is no meaningful way of fetching the data once a vdisk is deleted.

  6. list_volumes: Lists all volumes already attached to this node. This will simply return all vdisks to which this node has been given access using “addaccess.”

Why use the Hedvig storage driver for Flocker

As companies push Docker and containers into production environments, they will need persistent data storage for stateful applications. In today’s Docker environment this is difficult. The Hedvig and ClusterHQ joint solution solves this problem, with three unique differentiators:

  1. Dynamic storage provisioning: Volume/vdisk creation is instantaneous, even if you want to create volumes of the order of terabytes or petabytes. We never pre-allocate any physical storage for a vdisk and hence any vdisk size update is simply a metadata update. This enables users to increase their volume size at any point.

  2. Flexible volume configuration: Flocker can leverage data storage policy settings at the volume level to set vdisk features (dedupe, etc.) and allow containers with different storage requirements on the same host. You might have an email server and image server running as two different containerized applications on the same host, and now you have the flexibility of enabling dedupe and compression for the email server, but disable these sames services for for the image server.

  3. Automatic data migration: As containers move to new nodes, Flocker automatically manages detaching data volumes from source nodes and attaching to target nodes. There is no physical movement of data and hence there is no additional downtime associated with data movement!

What’s next with Hedvig and ClusterHQ joint development? We plan to continue to enhance the driver adding things like support for NFS and object-based volumes. If you'd like to learn more about the solution, download our solution brief.

Download now

Gaurav Yadav

Gaurav Yadav

Gaurav Yadav is a software engineer at Hedvig and a self-proclaimed expert in FIFA (the game). Prior to Hedvig Gaurav worked at Oracle for four years and has also worked at a number of companies including CA, NetLogic and a search engine startup. Gaurav graduated (somehow) from Texas A&M (Gig'em aggies).
See all authored articlesÚ