Smart Search (Azure)

If a validation set is passed manually then only those validation samples will be reflected in the confusion matrix.

However, if CV (custom validation) is enabled, then we will train on k-1 folds and compute a confusion matrix based on the left out fold (for each fold). At the end of leaving out all k folds we will aggregate the confusion matrices from each fold and that is the final validation metric you see for your run.

When using Azure AutoML runs you configure the experiment/train with the AutoMLConfig class. When doing so you have the following options for your dataset splits and cross-validation:

– A. You explicitly provide a validation dataset in the validation_data parameter: In this case, the validation and metrics will be calculated with that validation set (AML Dataset or Pandas DataFrame) which is not used for training the model. Only the training set will be used for training.

– B. You simply provide the validation_size parameter which is the fraction of the data to hold out for validation when user validation data is not specified. This should be between 0.0 and 1.0 non-inclusive (i.e. 0.2 means 20% of the data hold out for validation data).

– C. You explicitly set n_cross_validations parameter to a value: This means you want to do cross-validation. This parameter sets the How many cross validations (based on the same number of folds) to perform.

 – D. If you didn’t explicitly specify neither a validation set or the cross-validation parameter, then AutoML will use a train/validation split based approach or will use cross validation with default values depending on the number of rows in the single dataset provided:

 – D.a: If the training dataset is larger than 20,000 rows, it’ll do a train/validation split (by default the validation set is 10%) and use that validation set for the metrics calculation.

– D.b: If the training dataset is smaller than 20,000 rows, it’ll do a cross-validation approach. The default number of folds again depends on the number of rows. If the dataset is small (less than 1,000 rows) it’ll use 10 folds. If the rows are between 1,000 and 20,000, then it’ll use 3 folds.

As a more advanced custom scenario, you can also provide you own cross-validation data folds through the ‘cv_splits_indices’ parameter with the indices where to split training data for cross validation, as explained in the AutoMLConfig class reference page.

Note that as of Azure ML SDK v1.4.0, Azure AutoML (SDK and UI) doesn’t use any Test dataset (user provided or internal split) for automatic metrics calculation. However, it’s in the Azure AutoML roadmap.

Afterward, the model retrained on the entire dataset from scratch.