SageMaker experiment tracking now has two paths. The original SageMaker Experiments Python SDK and comparison browser belong to Studio Classic. For a new setup, AWS recommends the current Studio experience with MLflow; AWS recommends MLflow Apps over the older managed MLflow Tracking Servers. This guide creates an MLflow App and logs a Python training run, then explains what existing Studio Classic users should know about native Experiments.

In MLflow, an experiment groups related runs. A run records one attempt, including parameters, metrics, tags, and optional model or file artifacts. These records help compare configurations and trace results back to the code and data that produced them. They do not replace versioning your training code and datasets.

Initial setup

Check access, roles, and storage

Before creating an MLflow App, prepare:

  • A SageMaker AI account and a development environment, such as SageMaker Studio JupyterLab or a local Python environment configured with AWS credentials.
  • An existing general purpose S3 bucket for artifacts. The bucket must be in the same AWS Region as the MLflow App.
  • An IAM service role for the App that can access its artifact bucket. If the bucket uses a customer-managed KMS key, allow the App role to use that key as required.
  • An IAM role for the person or job that logs runs. It needs permission to call the MLflow App API. For an MLflow App, the data-plane action is sagemaker:CallMlflowAppApi.

These roles do different jobs: the App role writes artifacts to S3, while the notebook or training-job role authenticates MLflow API calls. Have an administrator configure the required SageMaker MLflow activities and resource-scoped permissions if you cannot manage IAM yourself. Avoid using a broad full-access policy as a shortcut. See AWS's MLflow App IAM guidance and the App API authorization action. For help scoping access to S3, see our guide to Amazon S3 access management.

Install the Python dependencies

In a SageMaker Studio notebook, install the AWS MLflow plugin and the example's scikit-learn and pandas dependencies:

%pip install sagemaker-mlflow scikit-learn pandas

The AWS plugin connects MLflow to SageMaker using AWS credentials and request signing. The AWS plugin package installs the lightweight MLflow client needed to log parameters and metrics. Pandas is used by the run-history query below. If you add model-flavor features such as sklearn model logging, install the full MLflow dependencies and the relevant flavor package, then use the versions supported by the current App release. Restart the notebook kernel if the environment asks you to.

Create an MLflow App and log a run

Create the tracking resource

Create an MLflow App in SageMaker Studio, or use a current AWS CLI when you need more control over the setup. Older CLI versions may not include the create-mlflow-app and describe-mlflow-app commands; check the AWS CLI command reference and update the CLI if needed. The role ARN below is the App's service role; the S3 bucket must already exist in us-west-2 and the role must be allowed to access it. AWS does not support an artifact bucket in a different Region from the App; see its App creation requirements.

aws sagemaker create-mlflow-app \
  --name "model-experiments" \
  --artifact-store-uri "s3://example-mlflow-artifacts/mlflow/" \
  --role-arn "arn:aws:iam::123456789012:role/SageMakerMlflowAppRole" \
  --region "us-west-2"

Replace the example bucket, account ID, and role with resources in your account. Save the Arn returned by the command; its resource type is mlflow-app. App creation starts the resource automatically. AWS's App setup guide recommends MLflow Apps over managed Tracking Servers. AWS describes Apps as serverless and notes their faster startup and cross-account sharing; check region and feature availability before choosing a Region.

Wait until the App is ready before connecting. Check its status with:

aws sagemaker describe-mlflow-app \
  --arn "arn:aws:sagemaker:us-west-2:123456789012:mlflow-app/app-example" \
  --region "us-west-2"

Continue when Status is Created. If it is CreateFailed, check the IAM setup, S3 bucket Region, and App role access. See the DescribeMlflowApp status reference.

Log a complete training run

Use the App ARN as the tracking URI. This example trains a small logistic-regression model on scikit-learn's built-in Iris dataset, then logs its settings and validation accuracy:

import mlflow
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

mlflow.set_tracking_uri(
    "arn:aws:sagemaker:us-west-2:123456789012:mlflow-app/app-example"
)
mlflow.set_experiment("iris-classification")

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
parameters = {"max_iter": 200, "random_state": 42}

with mlflow.start_run(run_name="logistic-regression-baseline"):
    model = LogisticRegression(**parameters)
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)

    mlflow.log_params(parameters)
    mlflow.log_param("dataset", "scikit-learn Iris")
    mlflow.log_metric("validation_accuracy", accuracy_score(y_test, predictions))

Replace the sample App ARN with the ARN returned for your App. MLflow creates the named experiment if it does not already exist. Each call to log_param, log_params, or log_metric records only the values supplied; SageMaker does not infer arbitrary custom metrics from a local training script.

Record enough context to reproduce a result

Along with model settings and evaluation metrics, record the dataset version or S3 URI, code revision, random seed, feature-processing version, and relevant library versions. Use stable names for experiments and runs, and avoid placing secrets or personal data in tags or parameters. For a model or report artifact, use the MLflow logging function for that artifact type and install its required MLflow flavor dependencies.

Use MLflow from a SageMaker training job

Configure the job's execution role

The same MLflow calls can run in a SageMaker training script or other supported Python workload. Install sagemaker-mlflow in the training environment, set the App ARN as the tracking URI, and grant the training job's execution role access to the App API. The training job does not automatically use the role attached to your notebook. If the job runs in a VPC with restricted egress, configure the SageMaker MLflow and SageMaker API interface endpoints and endpoint policies it needs; AWS lists the endpoint service names in its MLflow VPC endpoint guide.

Log parameters before training and metrics at the points where they are calculated. A metric from a SageMaker training job is not automatically logged to MLflow just because the job uses SageMaker. In SDK v2, Estimator.fit(experiment_config=...) associates a training job with native Experiment and Trial records; it does not configure MLflow. For App-based MLflow tracking, call the MLflow SDK in your training script or use an explicit MlflowConfig only where the particular SageMaker job API supports it. See the SDK v2 estimator reference and the relevant job API guide.

Choose metrics and artifacts deliberately

Track validation metrics that help you compare runs, with the evaluation split and method recorded alongside them. Keep large datasets and model files in S3 rather than putting them in run parameters. MLflow's artifact store is an S3 location; access to that location is controlled by the App's service role and bucket or KMS policies.

Compare runs in MLflow

Open the UI and query run history

Open the MLflow App from the SageMaker Studio MLflow interface, or generate a presigned UI URL with the SageMaker API. In the UI, select iris-classification to review its runs, parameters, metrics, and any logged artifacts. The URL used for browser access is temporary; share the App ARN and access permissions with teammates rather than treating a presigned URL as a permanent link.

You can also open the App UI from a terminal using a presigned URL:

aws sagemaker create-presigned-mlflow-app-url \
  --arn "arn:aws:sagemaker:us-west-2:123456789012:mlflow-app/app-example" \
  --region "us-west-2" \
  --query "AuthorizedUrl" \
  --output text

Open the returned URL in a browser. It is temporary. See the CreatePresignedMlflowAppUrl API reference.

You can also query the runs from Python and sort by a logged metric:

experiment = mlflow.get_experiment_by_name("iris-classification")
runs = mlflow.search_runs(
    experiment_ids=[experiment.experiment_id],
    order_by=["metrics.validation_accuracy DESC"],
)
print(runs[["run_id", "metrics.validation_accuracy", "params.max_iter"]])

Keep the tracking setup useful and manageable

Use consistent names and record data lineage

Name experiments around a project or model task, then identify each run by the configuration being compared. Log an immutable dataset version and source-code revision where possible. A run can be compared accurately only when its inputs and evaluation method are also clear.

Understand storage and compute charges

Experiment metadata and artifacts have different storage paths. SageMaker MLflow keeps run metadata in its managed backend and stores artifacts in the S3 bucket you configure. Training jobs, notebooks, S3 storage, and data transfer can each incur charges. AWS lists compute-hour and metadata-storage charges for managed MLflow Tracking Servers on its current SageMaker pricing page; check the current pricing details for your selected MLflow resource and Region rather than relying on older savings percentages.

MLflow Apps are serverless, so they do not require you to size and keep a tracking-server instance running. Deleting an App is permanent; AWS says the S3 artifacts remain in the bucket, so App metadata and artifact storage must be considered separately. Back up or migrate experiment records before deleting an App, and apply an S3 retention policy that matches your data requirements.

Separate user, job, and App permissions

Grant each notebook or job role access only to the MLflow App and data it needs. Keep the App's S3 permissions limited to its artifact bucket and prefix. If you enable model registration or use a customer-managed KMS key, configure those additional permissions on the relevant service role. AWS's MLflow App IAM guide distinguishes the caller permissions from the App role's access to S3 and the Model Registry.

Use the original Experiments API in an existing Studio Classic environment

The original SageMaker Experiments model groups related trials under an experiment; a trial can contain trial components for steps such as processing and training. The SDK's newer Run abstraction is the supported entry point for logging a single run, as shown in the SDK v3 API reference. AWS currently documents native Experiments tracking and its browser in Studio Classic only. Studio Classic is no longer open to new onboarding, and AWS recommends moving new work to the current Studio experience with MLflow.

The standalone sagemaker-experiments package is a legacy library; its PyPI project note directs users to the SageMaker Python SDK and warns that the repository is not kept current with product improvements.

If you already have access to Studio Classic and need to keep using native Experiments, choose the import for the SDK version installed in that environment. SageMaker Python SDK v2 uses:

from sagemaker.experiments.run import Run

SDK v3 reorganized the module path:

from sagemaker.core.experiments.run import Run

With either import, call logging methods inside the run context:

with Run(
    experiment_name="iris-classification",
    run_name="logistic-regression-baseline",
) as run:
    run.log_parameter("max_iter", 200)
    run.log_metric("validation_accuracy", 0.93)

This native API is separate from MLflow. Existing Experiments Classic data remains viewable in Studio Classic; changing a tracking URI does not convert it. AWS's migration guidance says to update the training code to use the MLflow SDK and rerun the experiments you want recorded in MLflow. Read AWS's current Experiments Classic and migration guidance before planning a transition.

Next steps

Make each run traceable

For new SageMaker tracking, create an MLflow App, grant scoped access to its artifact store and API, and log parameters and evaluation metrics from the code that produces them. Include code and data versions so you can explain why one run performed differently from another. Keep native SageMaker Experiments for existing Studio Classic workflows, and plan migration by logging future runs to MLflow rather than expecting old records to convert automatically.

For a related data-labeling workflow, see our guide to preparing image datasets with SageMaker Ground Truth.