[New Book] Click to get The Beginner's Guide to Data Science!
Use the offer code 20offearlybird to get 20% off. Hurry, sale ends soon!

Add Binary Flags for Missing Values for Machine Learning

Missing values can cause problems when modeling classification and regression prediction problems with machine learning algorithms.

A common approach is to replace missing values with a calculated statistic, such as the mean of the column. This allows the dataset to be modeled as per normal but gives no indication to the model that the row original contained missing values.

One approach to address this issue is to include additional binary flag input features that indicate whether a row or a column contained a missing value that was imputed. This additional information may or may not be helpful to the model in predicting the target value.

In this tutorial, you will discover how to add binary flags for missing values for modeling.

After completing this tutorial, you will know:

  • How to load and evaluate models with statistical imputation on a classification dataset with missing values.
  • How to add a flag that indicates if a row has one more missing values and evaluate models with this new feature.
  • How to add a flag for each input variable that has missing values and evaluate models with these new features.

Kick-start your project with my new book Data Preparation for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

  • Updated Jul/2020: Fixed bug in the creation of the flag variable.
Add Binary Flags for Missing Values for Machine Learning

Add Binary Flags for Missing Values for Machine Learning
Photo by keith o connell, some rights reserved.

Tutorial Overview

This tutorial is divided into three parts; they are:

  1. Imputing the Horse Colic Dataset
  2. Model With a Binary Flag for Missing Values
  3. Model With Indicators of All Missing Values

Imputing the Horse Colic Dataset

The horse colic dataset describes medical characteristics of horses with colic and whether they lived or died.

There are 300 rows and 26 input variables with one output variable. It is a binary classification prediction task that involves predicting 1 if the horse lived and 2 if the horse died.

There are many fields we could select to predict in this dataset. In this case, we will predict whether the problem was surgical or not (column index 23), making it a binary classification problem.

The dataset has numerous missing values for many of the columns where each missing value is marked with a question mark character (“?”).

Below provides an example of rows from the dataset with marked missing values.

You can learn more about the dataset here:

No need to download the dataset as we will download it automatically in the worked examples.

Marking missing values with a NaN (not a number) value in a loaded dataset using Python is a best practice.

We can load the dataset using the read_csv() Pandas function and specify the “na_values” to load values of ‘?’ as missing, marked with a NaN value.

The example below downloads the dataset, marks “?” values as NaN (missing) and summarizes the shape of the dataset.

Running the example downloads the dataset and reports the number of rows and columns, matching our expectations.

Next, we can evaluate a model on this dataset.

We can use the SimpleImputer class to perform statistical imputation and replace the missing values with the mean of each column. We can then fit a random forest model on the dataset.

For more on how to use the SimpleImputer class, see the tutorial:

To achieve this, we will define a pipeline that first performs imputation, then fits the model and evaluates this modeling pipeline using repeated stratified k-fold cross-validation with three repeats and 10 folds.

The complete example is listed below.

Running the example evaluates the random forest with mean statistical imputation on the horse colic dataset.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, the pipeline achieved an estimated classification accuracy of about 86.2 percent.

Next, let’s see if we can improve the performance of the model by providing more information about missing values.

Want to Get Started With Data Preparation?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Model With a Binary Flag for Missing Values

In the previous section, we replaced missing values with a calculated statistic.

The model is unaware that missing values were replaced.

It is possible that knowledge of whether a row contains a missing value or not will be useful to the model when making a prediction.

One approach to exposing the model to this knowledge is by providing an additional column that is a binary flag indicating whether the row had a missing value or not.

  • 0: Row does not contain a missing value.
  • 1: Row contains a missing value (which was/will be imputed).

This can be achieved directly on the loaded dataset. First, we can sum the values for each row to create a new column where if the row contains at least one NaN, then the sum will be a NaN.

We can then mark all values in the new column as 1 if they contain a NaN, or 0 otherwise.

Finally, we can add this column to the loaded dataset.

Tying this together, the complete example of adding a binary flag to indicate one or more missing values in each row is listed below.

Running the example first downloads the dataset and reports the number of rows and columns, as expected.

Then the new binary variable indicating whether a row contains a missing value is created and added to the end of the input variables. The shape of the input data is then reported, confirming the addition of the feature, from 27 to 28 columns.

We can then evaluate the model as we did in the previous section with the additional binary flag and see if it impacts model performance.

The complete example is listed below.

Running the example reports the mean and standard deviation classification accuracy on the horse colic dataset with the additional feature and imputation.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we see a modest lift in performance from 86.2 percent to 86.3 percent. The difference is small and may not be statistically significant.

Most rows in this dataset have a missing value, and this approach might be more beneficial on datasets with fewer missing values.

Next, let’s see if we can provide even more information about the missing values to the model.

Model With Indicators of All Missing Values

In the previous section, we added one additional column to indicate whether a row contains a missing value or not.

One step further is to indicate whether each input value was missing and imputed or not. This effectively adds one additional column for each input variable that contains missing values and may offer benefit to the model.

This can be achieved by setting the “add_indicator” argument to True when defining the SimpleImputer instance.

We can demonstrate this with a worked example.

The example below loads the horse colic dataset as before, then imputes the missing values on the entire dataset and adds indicators variables for each input variable that has missing values

Running the example first downloads and summarizes the shape of the dataset as expected, then applies the imputation and adds the binary (1 and 0 values) columns indicating whether each row contains a missing value for a given input variable.

We can see that the number of input variables has increased from 27 to 48, indicating the addition of 21 binary input variables, and in turn, that 21 of the 27 input variables must contain at least one missing value.

Next, we can evaluate the model with this additional information.

The complete example below demonstrates this.

Running the example reports the mean and standard deviation classification accuracy on the horse colic dataset with the additional indicators features and imputation.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we see a nice lift in performance from 86.3 percent in the previous section to 86.7 percent.

This may provide strong evidence that adding one flag per column that was inputted is a better strategy on this dataset and chosen model.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Related Tutorials

Summary

In this tutorial, you discovered how to add binary flags for missing values for modeling.

Specifically, you learned:

  • How to load and evaluate models with statistical imputation on a classification dataset with missing values.
  • How to add a flag that indicates if a row has one more missing values and evaluate models with this new feature.
  • How to add a flag for each input variable that has missing values and evaluate models with these new features.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Get a Handle on Modern Data Preparation!

Data Preparation for Machine Learning

Prepare Your Machine Learning Data in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Data Preparation for Machine Learning

It provides self-study tutorials with full working code on:
Feature Selection, RFE, Data Cleaning, Data Transforms, Scaling, Dimensionality Reduction, and much more...

Bring Modern Data Preparation Techniques to
Your Machine Learning Projects


See What's Inside

6 Responses to Add Binary Flags for Missing Values for Machine Learning

  1. Avatar
    Alexander Prokofyev July 21, 2020 at 3:25 am #

    İ beleive you should swap lines:

    # mark all non-nan as 0
    a[~isnan(a)] = 0
    # mark all nan as 1
    a[isnan(a)] = 1

    Thanks!

  2. Avatar
    Yoel Shoshan July 24, 2020 at 8:14 pm #

    Thanks for sharing!
    I understand it’s a technical and focused post, but still,
    one thing worth mentioning, is that it’s good to check if missing / not missing in itself is a strong indicator of (for example) the target class (for example, in univariate analysis), because many times it means there was a bias in the data acquisition process.
    For example, in binary classification, if 80% of positive class samples have a missing value, and
    30% of the negative class samples have a missing value, there’s a good chance it’s an issue.

  3. Avatar
    Houston Muzamhindo September 7, 2022 at 1:09 pm #

    Great tutorial. I have seen many talk about doing this for numerical variables. We impute categorical as well. Why is this process not used for categories but only numerical variables?

Leave a Reply