{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "np.set_printoptions(precision=3, linewidth=100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Introducing the `ColumnTransformer`: applying different transformations to different features in a scikit-learn pipeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*This work is supported by the Université Paris-Saclay Center for Data Science*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "

\n", "Short summary: the ColumnTransformer, which allows to apply different transformers to different features, has landed in scikit-learn (the PR has been merged in master and this will be included in the upcoming release 0.20). \n", "

\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "Real-world data often contains heterogeneous data types. When processing the data before applying the final prediction model, we typically want to use different preprocessing steps and transformations for those different types of columns. \n", "A simple example: we may want to scale the numerical features and one-hot encode the categorical features. \n", "\n", "Up to now, scikit-learn did not provide a good solution to do this out of the box. You can do the preprocessing beforehand using eg pandas, or you can select subsets of columns and apply different transformers on them manually. But, that does not easily allow to put those preprocessing steps in a scikit-learn `Pipeline`, which can be important to avoid data leakage or to do a grid search over preprocessing parameters.\n", "\n", "There are third-party projects that try to address this. For example, the [`sklearn_pandas`](https://github.com/scikit-learn-contrib/sklearn-pandas) package has a `DataFrameMapper` that maps subsets of a DataFrame's columns to a specific transformation. Many thanks to the authors of this library, as such \"contrib\" packages are essential in extending the functionality of scikit-learn, and to explore things that would take a long time in scikit-learn itself. \n", "The `ColumnTransformer` aims to bring this functionality into the core scikit-learn library, with support for numpy arrays and sparse matrices, and good integration with the rest of scikit-learn.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Basic example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To illustrate the basic usage of the `ColumnTransformer`, let's load the titanic survival dataset:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "titanic = pd.read_csv(\"https://raw.githubusercontent.com/amueller/scipy-2017-sklearn/master/notebooks/datasets/titanic3.csv\")\n", "# there is still a small problem with using the OneHotEncoder and missing values,\n", "# so for now I am going to assume there are no missing values by dropping them\n", "titanic2 = titanic.dropna(subset=['pclass', 'sex', 'age', 'sibsp', 'parch', 'fare', 'embarked'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Selecting some of the features and target:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "target = titanic2.survived.values\n", "features = titanic2[['pclass', 'sex', 'age', 'fare', 'embarked']]" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
pclasssexagefareembarked
01female29.0000211.3375S
11male0.9167151.5500S
21female2.0000151.5500S
31male30.0000151.5500S
41female25.0000151.5500S
\n", "
" ], "text/plain": [ " pclass sex age fare embarked\n", "0 1 female 29.0000 211.3375 S\n", "1 1 male 0.9167 151.5500 S\n", "2 1 female 2.0000 151.5500 S\n", "3 1 male 30.0000 151.5500 S\n", "4 1 female 25.0000 151.5500 S" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This dataset contains some categorical variables (\"pclass\", \"sex\" and \"embarked\"), and some numerical variables (\"age\" and \"fare\"). Note that the \"pclass\", although categorical, is already encoded as integers in the dataset. \n", "So let's use the `ColumnTransformer` to **combine transformers** for those two types of features:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n", "from sklearn.compose import ColumnTransformer, make_column_transformer" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "preprocess = make_column_transformer(\n", " (['age', 'fare'], StandardScaler()),\n", " (['pclass', 'sex', 'embarked'], OneHotEncoder())\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above creates a simple preprocessing pipeline (that will be combined in a full prediction pipeline below) to scale the numerical features and one-hot encode the categorical features. \n", "We can check this is indeed working as expected by transforming the input data" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[-0.057, 3.136, 1. , 0. , 0. , 1. , 0. , 0. , 0. , 1. ],\n", " [-2.012, 2.063, 1. , 0. , 0. , 0. , 1. , 0. , 0. , 1. ],\n", " [-1.937, 2.063, 1. , 0. , 0. , 1. , 0. , 0. , 0. , 1. ],\n", " [ 0.013, 2.063, 1. , 0. , 0. , 0. , 1. , 0. , 0. , 1. ],\n", " [-0.335, 2.063, 1. , 0. , 0. , 1. , 0. , 0. , 0. , 1. ]])" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preprocess.fit_transform(features).toarray()[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the above, we specified the subsets of columns as lists. We can also use boolean masks (eg to make a selection of the columns based on the data types), integer positions and slices. Further, the `ColumnTransformer` allows you to specify whether to drop or pass through other columns that were not specified. See the [development docs](http://scikit-learn.org/dev/modules/generated/sklearn.compose.ColumnTransformer.html) for more details.\n", "\n", "**This is new functionality in scikit-learn, so you are very welcome to try out the development version, experiment with it in your use cases, and provide feedback!** I am sure there are ways to further improve this functionality (the [PR](https://github.com/scikit-learn/scikit-learn/pull/9012))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The rest of the post shows a more complete example of using the ColumnTransformer in a scikit-learn pipeline.\n", "\n", "---\n", "\n", "### Integrating in a full pipeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's show a full example where we integrate the `ColumnTransformer` in a prediction pipeline." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.impute import SimpleImputer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Redefining `target` and `features` to take the full dataset this time including the missing values:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "target = titanic.survived.values\n", "features = titanic[['pclass', 'sex', 'age', 'fare', 'embarked']].copy()\n", "# I still fill the missing values for the embarked column, because we cannot (yet) easily handle categorical missing values\n", "features['embarked'].fillna(features['embarked'].value_counts().index[0], inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then, let's split the data in training and testing data:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "X_train, X_test, y_train, y_test = train_test_split(features, target, random_state=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here we will create again a ColumnTransformer, but now using more advanced features: we use a mask to select the column subsets based on the dtypes, and we use another pipeline to combine imputation and scaling for the numerical features:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "numerical_features = features.dtypes == 'float'\n", "categorical_features = ~numerical_features" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "preprocess = make_column_transformer(\n", " (numerical_features, make_pipeline(SimpleImputer(), StandardScaler())),\n", " (categorical_features, OneHotEncoder()))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can combine this preprocessing step based on the ColumnTransformer with a classifier in a Pipeline to predict whether passengers of the Titanic survived or not:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "model = make_pipeline(\n", " preprocess,\n", " LogisticRegression())" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "logistic regression score: 0.786585\n" ] } ], "source": [ "model.fit(X_train, y_train)\n", "print(\"logistic regression score: %f\" % model.score(X_test, y_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using our pipeline in a grid search\n", "\n", "The ColumnTransformer integrates well with the rest of scikit-learn. For example, we can now also do a grid search on the parameters of the different preprocessing steps." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import GridSearchCV" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Defining a simple search grid, where I search for both the Imputer strategy of the numerical preprocessing step as for the regularization parameter of the logistic regression step:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "param_grid = {\n", " 'columntransformer__pipeline__simpleimputer__strategy': ['mean', 'median'],\n", " 'logisticregression__C': [0.1, 1.0, 1.0],\n", " }" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Performing the grid search:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "grid_clf = GridSearchCV(model, param_grid, cv=10, iid=False)\n", "grid_clf.fit(X_train, y_train);" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'columntransformer__pipeline__simpleimputer__strategy': 'mean',\n", " 'logisticregression__C': 0.1}" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grid_clf.best_params_" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "best logistic regression from grid search: 0.792683\n" ] } ], "source": [ "print(\"best logistic regression from grid search: %f\" % grid_clf.best_estimator_.score(X_test, y_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "*This post was written in a Jupyter notebook. You can [download](http://jorisvandenbossche.github.io/downloads/notebooks/scikit-learn-columntransformer.ipynb) this notebook, or see a static view [on nbviewer](http://nbviewer.jupyter.org/url/jorisvandenbossche.github.io/downloads/notebooks/scikit-learn-columntransformer.ipynb).*" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (dev)", "language": "python", "name": "dev" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.5" }, "nav_menu": {}, "toc": { "navigate_menu": true, "number_sections": true, "sideBar": true, "threshold": 6, "toc_cell": false, "toc_section_display": "block", "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }