Queueing experiments

For the MLP version of the project, we have two parameters that change the number of hidden units and the activation function.

example-get-started/params.yaml

Let's setup experiments with different hyperparameters. We can define all the combinations we want to try without executing anything, by using the --queue flag:

dvc exp run --queue -n exp-1 -S model.mlp.units=32
dvc exp run --queue -n exp-2 -S model.mlp.units=64
dvc exp run --queue -n exp-3 -S model.mlp.units=128
dvc exp run --queue -n exp-4 -S model.mlp.units=256

The -n option is used to label the experiments. If it's not specified, DVC provides a name. We set the names for easier reference here.

Next, run all queued experiments using --run-all:

dvc exp run --run-all

Comparing many experiments

To compare all of these experiments, we need more than dvc exp diff:

dvc exp show compares any number of experiments in one table:

dvc exp show --no-timestamp \
             --include-params model.mlp.units \
             --no-pager

As we have the most hidden units in MLP for exp-4, it has the highest categorical_accuracy.

Persisting experiments

Now that we know the best parameters, let's keep that experiment and ignore the rest.

dvc exp apply rolls back the workspace to the specified experiment:

dvc exp apply exp-4

dvc exp apply is similar to dvc checkout, but it works with experiments. DVC tracks everything in the pipeline for each experiment (parameters, metrics, dependencies, and outputs) and can later retrieve it as needed.

Check that metrics.json reflects the metrics in the table above:

example-get-started/metrics.json

Once an experiment has been applied to the workspace, it is no different from reproducing the result without dvc exp run. Let's make it persistent in our regular pipeline by committing it in our Git branch:

git add dvc.lock params.yaml metrics.json train.log.csv
git commit -m "Preserve best Avg. Prec. experiment"

dvc push only uploads persistent experiments that have been committed to Git. The other experiments will not be pushed to the remote. See dvc exp push and dvc exp pull for how to share other experiments.

Cleaning up

After committing the best experiment to Git, let's take another look at the experiments table:

dvc exp show --no-timestamp \
             --include-params model.mlp.units \
             --no-pager

Where did all the experiments go? By default, dvc exp show only shows experiments since the last commit, but don't worry. The experiments remain cached and can be shown or applied. For example, use -n to show experiments from the previous n commits:

dvc exp show -n 2 --no-timestamp \
                  --include-params model.mlp.units \
                  --no-pager

Eventually, old experiments may clutter the experiments table.

dvc exp gc removes all references to old experiments:

dvc exp gc  --workspace 
dvc exp show -n 2 --no-timestamp \
                  --include-params model.mlp.units \ 
                  --no-pager

dvc exp gc only removes references to the experiments, not the cached objects associated to them. To clean up the cache, use dvc gc.

Congratulations!

You have completed Experiments scenario. We learned how to reproduce pipelines while modifying their parameters through the command line via dvc exp run.

We created a queue of experiments using dvc exp run --queue and run all of them at once dvc exp run --run-all.

We reviewed the experiment results in a table with dvc exp show and brought in the best one to the workspace dvc exp apply.

We cleaned up the remaining experiments with dvc exp gc.

For more information about experiment management, you can read the section in the user guide.

This completes the Getting Started course for DVC on Katacoda. Thank you for joining us.