I am building a machine learning model using sklearn Pipeline which includes a ColumnTransformer as a preprocessor before the actual model. Below is the code how the pipeline is created.

transformers = []
num_pipe = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
transformers.append(('numerical', num_pipe, num_cols))
cat_pipe = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('ohe', OneHotEncoder(handle_unknown='ignore'))
transformers.append(('categorical', cat_pipe, cat_cols))
preprocessor = ColumnTransformer(transformers, remainder='passthrough')
model = Pipeline([
  ('prep', preprocessor),
  ('clf', XGBClassifier())
])

I am using Mlflow to log the model artifact as sklearn model after it is fitted on training data.

model.fit(X, y)
mlflow.sklearn.log_model(model, model_uri)

When I tried to load the model from mlflow for scoring though, I got the error "This ColumnTransformer instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator."

run_model = mlflow.sklearn.load_model(model_uri)
run_model.predict(X_pred)

I also ran check_is_fitted on the second step of the Pipeline which is the xgboost model itself after loaded from mlflow and it is NOT fitted either.

Is Mlflow not compatible with sklearn Pipeline with multiple steps?

To directly answer your question: yes, MLFlow is compatible with sklearn Pipelines, our own AutoML glassbox generator uses a (nearly identical) Pipeline for xgboost training.

I have attached a sample notebook that I was able to run end-to-end to train your exact Pipeline on. the adult-income dataset, log it to MLFlow, load it back from MLFlow, and use it to predict values. I also included some calls to check_is_fitted to see that

• it's not fitted until you run model.fit
• then everything in the Pipeline is fitted
• the. logged model is fitted

Hope this helps as a guide, let me know if you have any other questions or issues.

To directly answer your question: yes, MLFlow is compatible with sklearn Pipelines, our own AutoML glassbox generator uses a (nearly identical) Pipeline for xgboost training.

I have attached a sample notebook that I was able to run end-to-end to train your exact Pipeline on. the adult-income dataset, log it to MLFlow, load it back from MLFlow, and use it to predict values. I also included some calls to check_is_fitted to see that

• it's not fitted until you run model.fit
• then everything in the Pipeline is fitted
• the. logged model is fitted

Hope this helps as a guide, let me know if you have any other questions or issues.

Hi @Nasreddin , MLflow is compatible with sklearn Pipeline with multiple steps. The error you're encountering, "This ColumnTransformer instance is not fitted yet. Call’ fit’ with appropriate arguments before using this estimator." is likely because ColumnTransformer of the Pipeline is not fitted correctly before being logged to MLflow. Looking at your code, it seems you are fitting and logging the model accurately. The issue might be with the loading of the model. When you load the model from MLflow, ensure that the model_uri correctly points to the logged model. Also, provide the model was successfully logged and saved in the correct path. Please verify these points and try again. If the problem persists, it might be a good idea to check the versions of sklearn, xgboost, and MLflow you are using and ensure they are compatible.

Sources:
1. [Docs: models-in-uc-example]( https://docs.databricks.com/mlflow/models-in-uc-example.html )
2. [Docs: workspace-model-registry-example]( https://docs.databricks.com/mlflow/workspace-model-registry-example.html )
3. [Docs: scikit-learn]( https://docs.databricks.com/machine-learning/preprocess-data/scikit-learn.html )

Isolation Forest prediction failing DLT pipeline, the same model works fine when prediction is done outside DLT pipeline. in Machine Learning 05-04-2023 Logging spark pipeline model using mlflow spark , leads to PythonSecurityException in Machine Learning 02-17-2023 Azure-core or AzureML version packages incompatibility in Machine Learning 12-15-2022 © Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.

Privacy Notice

| Terms of Use

| Your Privacy Choices

| Your California Privacy Rights