Practical Aspects Of Testing and Debugging in Machine Learning

This post tries to give you a sense of humor on how to work with debugging when you are calling yourself a ML engineer or Data scientist. This post reduces your task as a ML Engineer and gives you the ability to implement the understanding of basic life’s rules which is go slow , grow more.

Keluro

What We Are Discussing Here?

Google

Once you had finished of framing your idea and wasted 3-4 months trying copy paste other’s work to gain your reputation in your organization when everything fails as your organazation has a unique collection of data though organization is pretty much same as the next block of our office you have two option left either switch to other organization or actually trying to solve .

If you are in first category means you are actually switching then here is a treat for you don’t read my post further but if you are motivated enough to work on the data read further

The treat of 1 hour

Why You Debug ML Models?

You debug your ML model to make the model work. Once your model is working, you optimize the model’s quality for production-readiness. If you think debugging is easy read the article here by Google Expert . We want to achieve debugging because:

• Interpretable and explainable ML
• Formal methods and program verification
• Visualization and human factors
• Security and adversarial examples in ML 

As explained in the paper

Advertisements

How is ML Debugging Different?

Before diving into ML debugging, let’s understand what differentiates debugging ML models from debugging typical programs. Unlike typical programs, poor quality in an ML model does not imply the presence of a bug. Instead, to debug poor performance in a model, you investigate a broader range of causes than you would in traditional programming.

For example, here are a few causes for poor model performance:

  • Features lack predictive power.
  • Hyperparameters are set to nonoptimal values.
  • Data contains errors and anomalies.
  • Feature engineering code contains bugs.

Debugging ML models is complicated by the time it takes to run your experiments. Given the longer iteration cycles, and the larger error space, debugging ML models is uniquely challenging.

How to basically search for debugging in data :

if you have ever worked with mnist data set comes preloaded with keras implementation you will notice first if there is any imbalance in dataset a couple of line of code will give you a result like this

classes in x and frequency in y axis

The preceding graph shows that the 10 classes are roughly equally represented .Now comes one more important thought to you what if your data has some issues that is unique like you want to do feature scaling and one of your feature becomes nan if you cant debug, you will pay later .

Advertisements

Data Schema

To monitor your data, you should continuously check your data against expected statistical values by writing rules that the data must satisfy. This collection of rules is called a data schema. Unwritten rules of schema

  1. For your feature data, understand the range and distribution. For categorical features, understand the set of possible values.
  2. Encode your understanding into rules defined in the schema.
  3. Test your data against the data schema

Ensure Splits are Good Quality

Your test and training splits must be equally representative of your input data. If the test and training splits are statistically different, then training data will not help predict the test data. If you are a continuous visitor of my blog you will remember me speaking of the importance of splits in Baby Weight Prediction app building which can be found here.

Model Debugging

Once done with data schema come to your model

Check that the Model can Predict Labels

Before debugging your model, try to determine whether your features encode predictive signals. You can find linear correlations between individual features and labels by using correlation matrices. For an example of using correlation matrices .

However, correlation matrices will not detect nonlinear correlations between features and labels. Instead, choose 10 examples from your dataset that your model can easily learn from. Alternatively, use synthetic data that is easily learnable. 

Using a few examples that are easily learnable simplifies debugging by reducing the opportunities for bugs. You can further simplify your model by switching to the simpler gradient descent algorithm instead of a more advanced optimization algorithm.

you can check this code link here .

Model Debugging

The first step in debugging your model is Data Debugging. After debugging your data, follow these steps to continue debugging your model, detailed in the following sections:

  1. Check that the data can predict the labels.
  2. Establish a baseline.
  3. Write and run tests.
  4. Adjust your hyperparameter values.

Check that the Model can Predict Labels

Before debugging your model, try to determine whether your features encode predictive signals. You can find linear correlations between individual features and labels by using correlation matrices. For an example of using correlation matrices, see this Colab.

However, correlation matrices will not detect nonlinear correlations between features and labels. Instead, choose 10 examples from your dataset that your model can easily learn from. Alternatively, use synthetic data that is easily learnable. For instance, a classifier can easily learn linearly-separable examples while a regressor can easily learn labels that correlate highly with a feature cross. Then, ensure your model can achieve very small loss on these 10 easily-learnable examples.

Using a few examples that are easily learnable simplifies debugging by reducing the opportunities for bugs. You can further simplify your model by switching to the simpler gradient descent algorithm instead of a more advanced optimization algorithm.

Establish a Baseline

Comparing your model against a baseline is a quick test of the model’s quality. When developing a new model, define a baseline by using a simple heuristic to predict the label. If your trained model performs worse than its baseline, you need to improve your model.

Examples of baselines are:

  • Using a linear model trained solely on the most predictive feature.
  • In classification, always predicting the most common label.
  • In regression, always predicting the mean value.

Once you validate a version of your model in production, you can use that model version as a baseline for newer model versions. Therefore, you can have multiple baselines of different complexities. Testing against baselines helps justify adding complexity to your model. A more complex model should always perform better than a less complex model or baseline.

Implement Tests for ML Code

The testing process to catch bugs in ML code is similar to the testing process in traditional debugging. You’ll write unit tests to detect bugs. Examples of code bugs in ML are:

  • Hidden layers that are configured incorrectly.
  • Data normalization code that returns NaNs.

A sanity check for the presence of code bugs is to include your label in your features and train your model. If your model does not work, then it definitely has a bug.

Adjust Hyperparameter Values

check the colab code here .

TensorFlow Debugger

tfdbg is a specialized debugger for TensorFlow. It lets you view the internal structure and states of running TensorFlow graphs during training and inference, which is difficult to debug with general-purpose debuggers such as Python’s pdbdue to TensorFlow’s computation-graph paradigm.

This guide focuses on the command-line interface (CLI) of tfdbg. For guide on how to use the graphical user interface (GUI) of tfdbg, i.e., the TensorBoard Debugger Plugin, please visit its README.

python -m tensorflow.python.debug.examples.debug_mnist

This code means code we use previous the debugger will get you a result like this:

Accuracy at step 0: 0.1113 Accuracy at step 1: 0.3183 Accuracy at step 2: 0.098 Accuracy at step 3: 0.098 Accuracy at step 4: 0.098

learn more about tf debugger if you just use tf here.

Running the codes in this blog post

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s