Discovering files

If you look at the repository, you won't see data/data.xml or model.pkl, or any DVC-tracked files. They are not stored in Git. We can dvc get them, but how do we even know what data is tracked in a remote DVC repo before accessing it?

If dvc get is the analog of wget or curl, then dvc list is the analog of ls or aws s3 ls and similar commands:

dvc list \
  https://github.com/iterative/example-get-started \
  data/

The only difference is that we pass a Git URL. Same interface as dvc get. Now we can see data.xml as well as regular Git files.

Python API

Besides using DVC commands in the command line, we can also access any DVC-tracked artifact "natively" from Python with the API. Please click the below link to open the Python script:

process.py

The script downloads the data like dvc get and counts the number of lines in it:

python3 /root/process.py

Note that the script doesn't download the data to a file before counting the lines.

The interface of dvc.api.open is similar to the one we've seen already. It receives Git repo URL and path as arguments, and works the same way. There are also a few important differences:

• It's Python "native", we don't have to combine CLI scripts with Python code to process data or deploy a model.

• It doesn't consume space for a file on the file system - open() doesn't consume space in the file system - it loads the data into the memory as needed. If you want to process a large dataset or deploy a huge model you don't have to keep it on the disk.

• It reads data lazily - it doesn't allocate a huge array internally to hold the data, instead it streams it from the remote storage. This means you can process a huge dataset with a very low memory footprint.

• It unifies storage access - it doesn't matter if actual data is stored on S3, or Google Cloud, or SSH - the interface is the same.

Reusing (importing) data or models

Modern programming languages have a way to package, distribute, and reuse code as libraries (in the Python world - PyPI, pip, conda are some well-known examples). It's an important feature that significantly simplifies managing the complexity of the development process.

What about datasets and ML models?

A DVC repository and the dvc import command are enough to export data and models, reuse them, track upstream changes, etc. Let's give it a try:

mkdir data
dvc import \
  https://github.com/iterative/dataset-registry \
  get-started/data.xml -o data/data.xml

dvc import command creates data/data.xml.dvc to track the dependency. You can view this file in the editor:

example-get-started/data/data.xml.dvc

The url and rev_lock subfields under repo are used to save the origin and the version of the dependency, respectively:

The effect of using dvc import is similar to running dvc get + dvc add, but the resulting .dvc file includes metadata to track changes in the source repository. This allows you to bring in changes from the data source later, using:

dvc update data/data.xml.dvc

In this case, everything is up to date, but if someone creates a new version of data.xml in the data registry, this command can detect the change and update the data/data.xml file accordingly.

Congratulations!

In this scenario, we learned how to access data registries via DVC. We can download model and data files with dvc get or import them to DVC repositories with dvc import. DVC also has an API that streams large files directly into the memory with dvc.api.open.

DVC allows to have a central registry for all the data and model files and using them in different projects. It's based on Git, and provides flexibility without requiring additional infrastructure.

If you want to read more about the workflow, refer to the Data Registries use case.