How does it work?

Let's take a look at inside the .dvc file to learn how it keeps track of our data:

cat data/data.xml.dvc

This is a valid YAML 1.2 file. md5 field is used to keep track changes in data.xml. Let's check whether the hash calculated by md5sum is identical with the hash calculated by DVC.

md5sum data/data.xml 

grep 'md5' data/data.xml.dvc

DVC uses the hash to address the file in the cache. DVC detects the change by comparing the new MD5 hash to the hash kept in data.xml.dvc.

Let's see how the cache is organized:

tree .dvc/cache

DVC saved a copy of data/data.xml to .dvc/cache/ using the first two characters as directory and the rest as file name.

To save storage space and time when dealing with large files, DVC can use symlinks, hardlinks, or reflinks depending on the file system. Read more in the user guide for Large Dataset Optimization.

Data Remotes

DVC allows to set up remotes that are accessible by other systems or team members.

To set up a data remote, you would typically use Amazon S3, Google Cloud, or another supported storage type like:

dvc remote add --default \
    my-storage s3://my-bucket/dvc-storage

Note that DVC remotes are unlike Git remotes. DVC remotes are data storage locations without a commit history. History of data and model files are tracked in *.dvc text files, and these are typically shared with other team members as if they were code/text files.

It's possible to use a local directory in the same file system as a remote as well. This allows fast backups. We'll use this feature to keep this scenario simple.

dvc remote add --default \
    mystorage /tmp/data-storage

You can get a list of configured remotes using:

dvc remote list

The configuration is typically stored in Git:

git diff .dvc/config

Let's commit the changes to Git:

git commit .dvc/config \
    -m "Configure data storage"

Saving and Retrieving Data

The command dvc status can compare the data on the workspace with the data saved on the cache.

dvc status --cloud

Without the -c, --cloud option

dvc status

checks whether workspace files are up to date with the local cache.

As dvc status --cloud shows, we have a missing file in mystorage.

We manage data and model files in remotes with dvc push and dvc pull. These are used to save and retrieve DVC tracked files from local .dvc/cache to and from a data remote.

Now we are ready to push data.xml to mystorage remote we configured in the previous step:

dvc push

Note that dvc push uses default remote. There can be multiple remotes for a project, and if you want to push to a particular one, you can use the --remote option for dvc push.

Let's check the status again:

dvc status --cloud

Now, let's take a look at the content of /tmp/data-storage (location of our remote) to see what happens when we push a file to the cache.

tree /tmp/data-storage/

The structure of .dvc/cache and /tmp/data-storage is identical. Both of them contains the same files addressed by the same hash values.

tree .dvc/cache

Suppose we deleted data.xml from the workspace and from the cache.

rm -rf .dvc/cache

rm -f data/data.xml

We deleted all local versions of data.xml. Neither .dvc/cache, nor data/ contains our data file.

ls -R | grep 'data.xml'

This is also reported by:

dvc status

In order to retrieve missing data and model files from remotes, we can use:

dvc fetch

This command copies files from remote to local cache, but it doesn't update the workspace with the data file.

dvc status

Note that data.xml is still missing from the workspace, but we can get it from the local cache:

dvc checkout

dvc checkout links (or copies) from local cache files to the workspace.

dvc status

Instead of a two-step operation with fetch and checkout, we can use pull. Let's delete the data file and the cache again.

rm -rf .dvc/cache 
rm -f data/data.xml

We can retrieve the file from remote with a single command:

dvc pull

dvc pull is usually used after git clone, git pull, or git checkout to synchronize the data with the code. Along with dvc push, it provides a basic collaboration workflow, similar to git push and git pull.

Making Changes

Let's make the dataset smaller. It's not what we'd usually do. In general, we want more data, but since there are limitations in this platform, we need to do this to run the training later:

head -n 12000 data/data.xml > data/data.xml.1
mv data/data.xml.1 data/data.xml

We check the status of the project:

dvc status

Run dvc add to save its new version to cache and update data/data.xml.dvc.

dvc add data/data.xml

It updates the hash value in the corresponding dvc file:

git diff

Commit the changed hash value to track the version of the file:

git commit -a -m "Dataset updates"

Push the new version of the file to the default remote:

dvc push

So, each version of data/data.xml.dvc in the Git history is linked to a version of the data file in the DVC cache.

Switching between Versions

Technically speaking, DVC is not a version control system.

Git serves as the VCS for text and code files, although DVC can work without it.

Let's get the previous version of data/data.xml:

git checkout HEAD~ data/data.xml.dvc

When we check what changed using:

git diff --staged

We see that data.xml.dvc now contains the previous hash value of data.xml.

md5sum data/data.xml

To synchronize data.xml with the version addressed in data.xml.dvc:

dvc checkout

Now we can see that the value in data.xml.dvc and the hash value of data.xml` are identical.

grep 'md5' data/data.xml.dvc
md5sum data/data.xml

Instead of dvc checkout, we could use dvc pull again. Pulling also downloads missing data from the remote storage, whereas dvc checkout can only restore that's already in the local cache.