AWS Glue Data Quality lets you define rules in Data Quality Definition Language (DQDL) and evaluate data in two places: against tables in the AWS Glue Data Catalog or inside an AWS Glue ETL job. The right choice depends on what should happen when a rule fails. A catalog evaluation reports on data that is already cataloged; an ETL check can participate in the pipeline before data reaches its target.

Use the practices below to turn business expectations into rules, review failures, and decide whether a pipeline should continue, stop, or route records for remediation. If you are new to Glue tables and jobs, start with our AWS Glue core concepts tutorial.

Assessing Data Quality

Start by choosing where the check belongs. AWS documents these as separate entry points with different sources and outputs; neither choice changes the meaning of your business rules.

AWS Glue Data Quality

For governance checks on a table that is already in the Data Catalog, create a ruleset and run it as a data quality task. You can review recommendations, edit the suggested rules, run checks on demand or on a schedule, and inspect rule-level outcomes. Recommendations are based on the available data, so treat them as a starting point: observed values do not establish what your business considers valid.

For a check that must run as data is transformed, add the Evaluate Data Quality transform to an AWS Glue ETL job. This is the in-pipeline option when you need to stop a load or use row-level results to separate records. ETL supports Glue data sources, including custom and third-party connectors; Catalog evaluation has its own supported-source list. Check the current feature comparison and source limitations before choosing.

A data quality score is the share of rules that passed, not the share of records that are good. A score can hide the operational importance of a failed rule, so inspect individual results and decide which failures block a pipeline.

AWS Glue DataBrew

DataBrew is a separate visual data preparation service. Use a project to explore data and build a reusable recipe, then run a profile job to summarize a dataset or a recipe job to transform it and write prepared output. Glue Data Quality evaluates explicit rules; it does not apply cleaning steps. If both are useful, use DataBrew to prepare a dataset and Glue Data Quality to check the resulting data against requirements.

Profiling Data and Finding Anomalies

Profile data to learn its shape, then ask data owners which observations should become rules. A profile can show nulls, distributions, or changing counts; it cannot decide whether those values are acceptable for a business process.

Data Profiling

For a cataloged table, Glue can recommend rules from the data it sees. Review each suggestion against the schema, source contract, and downstream use before adding it to a ruleset. A sample that currently contains one status value does not prove that other valid statuses should be rejected.

Use the Data Catalog getting-started guide to create recommendations and edit the resulting DQDL ruleset. Glue Data Quality does not evaluate nested or list-type sources as-is; check the current limitations and flatten or otherwise prepare the input if needed.

Detecting Anomalies

Glue Data Quality can use historical statistics to flag unusual changes, such as a shift in row counts or a monitored column statistic. Anomaly detection needs at least three data points, does not change the data quality score, and treats a detected value as normal in later runs unless you exclude it. An anomaly is a signal to investigate, not a diagnosis or a substitute for a business rule. Review flagged observations and give feedback so the model does not learn from an event you still consider abnormal. See AWS’s guide to anomaly detection in Glue Data Quality.

Setting Quality Rules

DQDL is case-sensitive. A ruleset contains a comma-separated list inside Rules = [ ... ]. For example, the following rules make sense only if every order must have a non-null unique identifier, the listed statuses are the full accepted set, and an empty batch is invalid:

Rules = [
  IsComplete "order_id",
  IsUnique "order_id",
  ColumnValues "status" in ["NEW", "PAID", "CANCELLED"],
  RowCount > 0
]

IsComplete requires every value to be complete, and IsUnique requires values to be unique. Keep the row-count rule only if zero rows are actually invalid for that input. Choose thresholds from the data contract and business tolerance, not a generic percentage. Also decide how the source represents missing values: nulls, empty strings, and whitespace can behave differently. The DQDL reference documents rule syntax and supported types.

Cleaning and Transforming Data

A passing or failing result is an assessment. It does not correct, delete, or quarantine source records by itself. Define remediation separately so a quality check cannot silently change business data.

Data Cleansing

Decide what to do with each failure based on its meaning. A missing optional description might be acceptable; a missing order identifier might require rejecting the row. Preserve the original input and enough run context to investigate changes. Avoid filling nulls, deduplicating, or standardizing values unless the owner has defined the intended outcome.

Transforming Data

Use a DataBrew recipe for visual, reusable preparation steps, or transform records in a Glue ETL job when the logic belongs in the pipeline. A DataBrew recipe job writes transformed output to a configured destination and leaves its source unchanged. A Glue job can place evaluation before its target write so the rules participate in the load decision.

In Glue Studio, the default action on ruleset failure is to let the job continue. Configure a failure action if the load must stop when the ruleset fails, and choose when it should stop relative to target loading. Check the job’s result-publishing settings as well: stopping immediately can prevent results from being written to the configured target.

Routing Failed Rows

When you need a quarantine path, use the ETL transform’s row-level outcome columns to branch records whose DataQualityRulesFail array is populated, then write them to an access-controlled location for review. Send passing rows to the normal target. This routing is work you build into the job; Glue Data Quality does not automatically clean or quarantine them.

Not every rule can identify a bad row. Dataset-level rules such as RowCount describe the input as a whole, so they can fail while individual rows pass. Use row-level checks for row routing and handle aggregate failures as a separate job-level decision. See AWS’s ETL data quality tutorial for output columns and supported behavior.

Monitoring and Alerts

Monitor the results Glue actually publishes. A failed-rule count is not a completeness percentage or a count of rejected records; choose the signal that matches the decision you need to make.

Monitoring Data Quality

Review rule outcomes and run history to see which expectation failed, its evaluated metric, and the run that produced it. For Catalog evaluations, you can optionally write rule, profile, anomaly-observation, and row-level results to Apache Iceberg tables in the Data Catalog. This output is disabled by default; enable the result types you need and set a row limit when writing row-level results. The evaluation role needs permission to create or update the result tables and write to their S3 location. See AWS’s guide to writing Glue Data Quality results to Catalog tables. A row-level result table is useful for investigation, but it is still a results table rather than an automatic quarantine workflow.

In an ETL job, choose the transform output that includes row-level pass, fail, or skip details when you need to identify records. Some aggregate rules do not identify failing records, and the overall ruleset can fail even when a particular row passes. Include a stable business key if operators need to match a result back to a source record.

Setting Up Alerts

AWS Glue Data Quality emits an EventBridge event when an evaluation run completes. Filter for the outcomes that require action; call GetDataQualityResult when the event does not contain enough detail. AWS recommends EventBridge for data quality alerts. You can also publish the per-run CloudWatch metrics glue.data.quality.rules.passed and glue.data.quality.rules.failed and use them in alarms. Enable metric publishing for the run before relying on those metrics. See AWS’s alerting guide for the event and metric setup.

Integrating with Other AWS Services

Using Amazon S3

For S3 data, the Data Catalog holds table metadata that Glue uses to locate and interpret the data; the data itself remains in S3. Give the evaluation or job role access to the source objects and any output prefix it uses. Keep quality results and quarantined records in deliberate locations with retention and access rules suited to their sensitivity.

Amazon Redshift for Analytics

If validated Glue output feeds Redshift, put the quality decision before the write when invalid batches must not reach the target. For the separate Glue-to-Redshift connection, staging, and IAM setup, see our guide to automating Redshift ETL with AWS Glue.

Querying with Amazon Athena

When you enable Catalog result-table output, the results are stored as Iceberg tables in S3 and registered in the Data Catalog. Query those tables with a compatible engine such as Athena to review failed rules or rows over time. Keep the run identifier and source table columns in your queries so results from different evaluations do not get mixed together.

Automating Data Quality Workflows

Using AWS Step Functions

Use Step Functions when a workflow must sequence a Glue job, inspect its outcome, and choose a later action such as notifying an owner or starting a downstream load. For a simpler periodic Catalog check, schedule the ruleset evaluation directly; Glue creates the schedule through Amazon EventBridge. Keep the schedule aligned with how often new data arrives and how quickly consumers need a decision.

Documenting Data and Metadata

AWS Glue Data Catalog

The Data Catalog stores table metadata and can associate data quality rulesets with cataloged tables. Keep the rule owner, rationale, evaluation cadence, and expected response to failure alongside the dataset documentation so maintainers know what each rule protects.

Document Lineage Explicitly

Record where the data came from, which transformations ran, and where failed rows were sent. Do not assume that a Catalog table definition or a data quality score alone explains the full business lineage or the reason a record was rejected.

Permissions for Data Quality Runs

Scope the Run Role

Give the role that runs a Catalog evaluation the Glue Data Quality actions it needs, read access to the Catalog table and source objects, and logging access. Add permission to publish CloudWatch metrics or write results only when those options are enabled. If you write Iceberg result tables, add the required Data Catalog create/update and S3 write permissions; add KMS or Lake Formation access when your data uses those controls. Glue ETL jobs also need the source and target permissions for their pipeline. Follow AWS’s Data Quality IAM guidance and scope access to the resources this run uses.

Scheduled Catalog evaluations need the caller’s iam:PassRole permission for the evaluation role and a role trust policy that allows AWS Glue and EventBridge Scheduler to assume it. Limit that permission to the intended evaluation role; AWS documents the scheduler setup in its IAM guidance.

Managing Evaluation Performance and Cost

Scope the Evaluation

Use a partition predicate for incremental Catalog checks when the source and rule semantics allow it; Glue ETL can use job bookmarks for incremental processing. Limit the data read to the batch or partitions the rule is meant to assess, and test that the chosen scope still catches the failures your contract cares about.

Estimate Cost and Results Storage

Catalog recommendations and evaluation tasks consume Glue DPUs; the current Glue pricing page specifies a minimum of two DPUs and a one-minute minimum billing duration for these tasks. Data quality checks in ETL add job runtime or DPU consumption. Anomaly detection for Catalog evaluations can add compute. S3 storage, requests and data transfer, Data Catalog usage, and published CloudWatch custom metrics can add charges, depending on what you enable. DataBrew has separate session and job pricing. Estimate against your Region, run frequency, data volume, and retention needs using the AWS Glue pricing page and CloudWatch pricing.

For row-level Catalog results, choose whether you need failed rows only or all rows and set an appropriate maximum row count. This controls how much result data is written; it does not replace cost estimation for the evaluation itself or the S3 storage that retains results.

Conclusion

Start with the contract each dataset must meet, then choose the Glue evaluation path that can support the action you require.

Key Takeaways

  • DQDL rules check data; they do not automatically clean it.
  • A failed ruleset does not stop an ETL job unless you configure a failure action.
  • Route individual failed rows in ETL; treat aggregate failures as job-level decisions.
  • Catalog evaluations can write optional row-level results to Iceberg tables, but that output is not a quarantine workflow.
  • Budget for Glue compute and any S3, CloudWatch, or EventBridge resources you enable.