How does it work?

cat data/data.xml.dvc

md5sum data/data.xml

DVC created a tracking file with the extension .dvc.

The MD5 hash of the data file data.xml is kept in data.xml.dvc. Since the copy of data.xml saved in the cache is named after its MD5 hash, the hash is effectively a pointer to the cached copy.

If the content of data.xml is changed, its MD5 hash will change. DVC is able to detect the change by comparing the new MD5 hash to the MD5 hash kept in data.xml.dvc.

tree .dvc/cache

DVC has also saved a copy of data/data.xml to .dvc/cache/ using its MD5 hash as a file name.

To save storage space and time when dealing with large files, DVC can use symlinks, hardlinks, or reflinks, depending on the file system and DVC settings. Read more in the user guide for Large Dataset Optimization.

Data remotes

Using dvc push and dvc pull, you can save and retrieve DVC tracked data files from the .dvc/cache to and from a data remote — similar to how you use git push and git pull to save and retrieve commits to and from Github.

You can use DVC data remotes to give other team members or systems access to your work and to create backups.

To set up a data remote, you would typically use Amazon S3, Google Cloud, or another supported storage type like this:

dvc remote add --default \
    my-storage s3://my-bucket/dvc-storage

To keep this scenario simple and not dependent on a specific storage type, we'll use local "remote" storage instead. A local data remote can also be used to back up data to a locally-mounted volume.

dvc remote add --default \
    mystorage /tmp/data-storage

dvc remote list

git status -s

git diff .dvc/config

git commit .dvc/config \
    -m "Configure data storage"

DVC supports the following remote storage types: Google Drive, Amazon S3, Azure Blob Storage, Google Cloud Storage, Aliyun OSS, SSH, HDFS, and HTTP. Please refer to dvc remote add for more details and examples.

Saving and retrieving data

The command dvc status compares the data on the workspace with the data saved on the cache.

dvc status

In this case it shows that everything is up to date. However with the option -c, --cloud it compares instead the cache with the data remote storage:

dvc status --cloud

In this case it shows that there is a new file on the cache that has not been uploaded yet to the data storage. Let's upload it:

dvc push

dvc status --cloud

tree /tmp/data-storage/

tree .dvc/cache/

Now the cached file is also saved on the data storage.

Let's now remove everything and retrieve it again:

rm -rf .dvc/cache

rm -f data/data.xml

dvc pull

tree data/

tree .dvc/cache/

dvc pull is usually used after git clone, git pull, or git checkout to synchronize the data with the code. Along with dvc push, it provides a basic collaboration workflow, similar to git push and git pull, that facilitates sharing of data.

Making changes

Let's make the dataset smaller. It's not what would you usually do (usually we want more data), but since there are some RAM limitations to this online platform we have to do this to being able to run training later:

head -n 12000 data/data.xml > data/data.xml.1

mv data/data.xml.1 data/data.xml

dvc status

When the file data/data.xml is changed, DVC can detect it by analyzing the corresponding .dvc file, run dvc add to save its new version to cache and update data/data.xml.dvc to match the new hash of the data file:

dvc add data/data.xml

git diff

git commit -a -m "Dataset updates"

dvc push

So, each version of data/data.xml.dvc in Git is linked to a version of the data file in the DVC cache. This linking or connection is done through the MD5 hash of the cached file, which is also its name on the cache.

Switching between versions

Yes, DVC is technically even not a version control system! .dvc files (and dvc.lock, but we'll get to that later) content defines data file versions. Git itself serves as the version control system. DVC in turn creates these .dvc files, updates them, and synchronizes DVC-tracked data in the workspace efficiently to match them.

Let's get get the previous version of the dataset data/data.xml:

git checkout HEAD^1 data/data.xml.dvc

git diff --staged

md5sum data/data.xml

dvc checkout

dvc checkout command synchronizes data files in the workspace to match the .dvc files content:

md5sum data/data.xml

Alternatively, dvc pull can be used again. The difference is that dvc pull also downloads missing data into cache, while dvc checkout only can restore data that already in cache.