Glue

Module glue

Important Capabilities

Capability	Status	Notes
Detect Deleted Entities	✅	Enabled by default when stateful ingestion is turned on.
Domains	✅	Supported via the domain config field
Platform Instance	✅	Enabled by default

Note: if you also have files in S3 that you'd like to ingest, we recommend you use Glue's built-in data catalog. See here for a quick guide on how to set up a crawler on Glue and ingest the outputs with DataHub.

This plugin extracts the following:

• Tables in the Glue catalog
• Column types associated with each table
• Table metadata, such as owner, description and parameters
• Jobs and their component transformations, data sources, and data sinks

IAM permissions

For ingesting datasets, the following IAM permissions are required:

{
    "Effect": "Allow",
    "Action": [
        "glue:GetDatabases",
        "glue:GetTables"
    ],
    "Resource": [
        "arn:aws:glue:$region-id:$account-id:catalog",
        "arn:aws:glue:$region-id:$account-id:database/*",
        "arn:aws:glue:$region-id:$account-id:table/*"
    ]
}

For ingesting jobs (extract_transforms: True), the following additional permissions are required:

{
    "Effect": "Allow",
    "Action": [
        "glue:GetDataflowGraph",
        "glue:GetJobs",
    ],
    "Resource": "*"
}

plus s3:GetObject for the job script locations.

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[glue]'

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: glue
  config:
    # Coordinates
    aws_region: "my-aws-region"

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options

Field [Required]	Type	Description	Default	Notes
aws_region [✅]	string	AWS region code.	None
aws_access_key_id	string	AWS access key ID. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.	None
aws_endpoint_url	string	Autodetected. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html	None
aws_profile	string	Named AWS profile to use. Only used if access key / secret are unset. If not set the default will be used	None
aws_proxy	map(str,string)		None
aws_secret_access_key	string	AWS secret access key. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.	None
aws_session_token	string	AWS session token. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.	None
catalog_id	string	The aws account id where the target glue catalog lives. If None, datahub will ingest glue in aws caller's account.	None
emit_s3_lineage	boolean	Whether to emit S3-to-Glue lineage.	None
extract_owners	boolean	When enabled, extracts ownership from Glue directly and overwrites existing owners. When disabled, ownership is left empty for datasets.	True
extract_transforms	boolean	Whether to extract Glue transform jobs.	True
glue_s3_lineage_direction	string	If upstream, S3 is upstream to Glue. If downstream S3 is downstream to Glue.	upstream
ignore_unsupported_connectors	boolean	Whether to ignore unsupported connectors. If disabled, an error will be raised.	True
platform	string	The platform to use for the dataset URNs. Must be one of ['glue', 'athena'].	glue
platform_instance	string	The instance of the platform that all assets produced by this recipe belong to	None
use_s3_bucket_tags	boolean	If an S3 Buckets Tags should be created for the Tables ingested by Glue. Please Note that this will not apply tags to any folders ingested, only the files.	None
use_s3_object_tags	boolean	If an S3 Objects Tags should be created for the Tables ingested by Glue.	None
env	string	The environment that all assets produced by this connector belong to	PROD
aws_role	UnionType (See notes for variants)	AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role	None	One of string,union(anyOf),string,AwsAssumeRoleConfig
aws_role.RoleArn [❓ (required if aws_role is set)]	string	ARN of the role to assume.	None
aws_role.ExternalId	string	External ID to use when assuming the role.	None
database_pattern	AllowDenyPattern	regex patterns for databases to filter in ingestion.	{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
database_pattern.allow	array(string)		None
database_pattern.deny	array(string)		None
database_pattern.ignoreCase	boolean	Whether to ignore case sensitivity during pattern matching.	True
domain	map(str,AllowDenyPattern)	A class to store allow deny regexes	None
domain.key.allow	array(string)		None
domain.key.deny	array(string)		None
domain.key.ignoreCase	boolean	Whether to ignore case sensitivity during pattern matching.	True
table_pattern	AllowDenyPattern	regex patterns for tables to filter in ingestion.	{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
table_pattern.allow	array(string)		None
table_pattern.deny	array(string)		None
table_pattern.ignoreCase	boolean	Whether to ignore case sensitivity during pattern matching.	True
profiling	GlueProfilingConfig	Configs to ingest data profiles from glue table	None
profiling.column_count	string	The parameter name for column count in glue table.	None
profiling.max	string	The parameter name for the max value of a column.	None
profiling.mean	string	The parameter name for the mean value of a column.	None
profiling.median	string	The parameter name for the median value of a column.	None
profiling.min	string	The parameter name for the min value of a column.	None
profiling.null_count	string	The parameter name for the count of null values in a column.	None
profiling.null_proportion	string	The parameter name for the proportion of null values in a column.	None
profiling.row_count	string	The parameter name for row count in glue table.	None
profiling.stdev	string	The parameter name for the standard deviation of a column.	None
profiling.unique_count	string	The parameter name for the count of unique value in a column.	None
profiling.unique_proportion	string	The parameter name for the proportion of unique values in a column.	None
profiling.partition_patterns	AllowDenyPattern	Regex patterns for filtering partitions for profile. The pattern should be a string like: "{'key':'value'}".	{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
profiling.partition_patterns.allow	array(string)		None
profiling.partition_patterns.deny	array(string)		None
profiling.partition_patterns.ignoreCase	boolean	Whether to ignore case sensitivity during pattern matching.	True
stateful_ingestion	StatefulStaleMetadataRemovalConfig	Base specialized config for Stateful Ingestion with stale metadata removal capability.	None
stateful_ingestion.enabled	boolean	The type of the ingestion state provider registered with datahub.	None
stateful_ingestion.ignore_new_state	boolean	If set to True, ignores the current checkpoint state.	None
stateful_ingestion.ignore_old_state	boolean	If set to True, ignores the previous checkpoint state.	None
stateful_ingestion.remove_stale_metadata	boolean	Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.	True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "GlueSourceConfig",
  "description": "Base configuration class for stateful ingestion for source configs to inherit from.",
  "type": "object",
  "properties": {
    "aws_access_key_id": {
      "title": "Aws Access Key Id",
      "description": "AWS access key ID. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.",
      "type": "string"
    },
    "aws_secret_access_key": {
      "title": "Aws Secret Access Key",
      "description": "AWS secret access key. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.",
      "type": "string"
    },
    "aws_session_token": {
      "title": "Aws Session Token",
      "description": "AWS session token. Can be auto-detected, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html for details.",
      "type": "string"
    },
    "aws_role": {
      "title": "Aws Role",
      "description": "AWS roles to assume. If using the string format, the role ARN can be specified directly. If using the object format, the role can be specified in the RoleArn field and additional available arguments are documented at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html?highlight=assume_role#STS.Client.assume_role",
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "array",
          "items": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "$ref": "#/definitions/AwsAssumeRoleConfig"
              }
            ]
          }
        }
      ]
    },
    "aws_profile": {
      "title": "Aws Profile",
      "description": "Named AWS profile to use. Only used if access key / secret are unset. If not set the default will be used",
      "type": "string"
    },
    "aws_region": {
      "title": "Aws Region",
      "description": "AWS region code.",
      "type": "string"
    },
    "aws_endpoint_url": {
      "title": "Aws Endpoint Url",
      "description": "Autodetected. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html",
      "type": "string"
    },
    "aws_proxy": {
      "title": "Aws Proxy",
      "description": "Autodetected. See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html",
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    },
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "database_pattern": {
      "title": "Database Pattern",
      "description": "regex patterns for databases to filter in ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "table_pattern": {
      "title": "Table Pattern",
      "description": "regex patterns for tables to filter in ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "platform_instance": {
      "title": "Platform Instance",
      "description": "The instance of the platform that all assets produced by this recipe belong to",
      "type": "string"
    },
    "stateful_ingestion": {
      "$ref": "#/definitions/StatefulStaleMetadataRemovalConfig"
    },
    "platform": {
      "title": "Platform",
      "description": "The platform to use for the dataset URNs. Must be one of ['glue', 'athena'].",
      "default": "glue",
      "type": "string"
    },
    "extract_owners": {
      "title": "Extract Owners",
      "description": "When enabled, extracts ownership from Glue directly and overwrites existing owners. When disabled, ownership is left empty for datasets.",
      "default": true,
      "type": "boolean"
    },
    "extract_transforms": {
      "title": "Extract Transforms",
      "description": "Whether to extract Glue transform jobs.",
      "default": true,
      "type": "boolean"
    },
    "ignore_unsupported_connectors": {
      "title": "Ignore Unsupported Connectors",
      "description": "Whether to ignore unsupported connectors. If disabled, an error will be raised.",
      "default": true,
      "type": "boolean"
    },
    "emit_s3_lineage": {
      "title": "Emit S3 Lineage",
      "description": " Whether to emit S3-to-Glue lineage.",
      "default": false,
      "type": "boolean"
    },
    "glue_s3_lineage_direction": {
      "title": "Glue S3 Lineage Direction",
      "description": "If `upstream`, S3 is upstream to Glue. If `downstream` S3 is downstream to Glue.",
      "default": "upstream",
      "type": "string"
    },
    "domain": {
      "title": "Domain",
      "description": "regex patterns for tables to filter to assign domain_key. ",
      "default": {},
      "type": "object",
      "additionalProperties": {
        "$ref": "#/definitions/AllowDenyPattern"
      }
    },
    "catalog_id": {
      "title": "Catalog Id",
      "description": "The aws account id where the target glue catalog lives. If None, datahub will ingest glue in aws caller's account.",
      "type": "string"
    },
    "use_s3_bucket_tags": {
      "title": "Use S3 Bucket Tags",
      "description": "If an S3 Buckets Tags should be created for the Tables ingested by Glue. Please Note that this will not apply tags to any folders ingested, only the files.",
      "default": false,
      "type": "boolean"
    },
    "use_s3_object_tags": {
      "title": "Use S3 Object Tags",
      "description": "If an S3 Objects Tags should be created for the Tables ingested by Glue.",
      "default": false,
      "type": "boolean"
    },
    "profiling": {
      "title": "Profiling",
      "description": "Configs to ingest data profiles from glue table",
      "allOf": [
        {
          "$ref": "#/definitions/GlueProfilingConfig"
        }
      ]
    }
  },
  "required": [
    "aws_region"
  ],
  "additionalProperties": false,
  "definitions": {
    "AwsAssumeRoleConfig": {
      "title": "AwsAssumeRoleConfig",
      "type": "object",
      "properties": {
        "RoleArn": {
          "title": "Rolearn",
          "description": "ARN of the role to assume.",
          "type": "string"
        },
        "ExternalId": {
          "title": "Externalid",
          "description": "External ID to use when assuming the role.",
          "type": "string"
        }
      },
      "required": [
        "RoleArn"
      ]
    },
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    },
    "DynamicTypedStateProviderConfig": {
      "title": "DynamicTypedStateProviderConfig",
      "type": "object",
      "properties": {
        "type": {
          "title": "Type",
          "description": "The type of the state provider to use. For DataHub use `datahub`",
          "type": "string"
        },
        "config": {
          "title": "Config",
          "description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19)."
        }
      },
      "required": [
        "type"
      ],
      "additionalProperties": false
    },
    "StatefulStaleMetadataRemovalConfig": {
      "title": "StatefulStaleMetadataRemovalConfig",
      "description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
      "type": "object",
      "properties": {
        "enabled": {
          "title": "Enabled",
          "description": "The type of the ingestion state provider registered with datahub.",
          "default": false,
          "type": "boolean"
        },
        "ignore_old_state": {
          "title": "Ignore Old State",
          "description": "If set to True, ignores the previous checkpoint state.",
          "default": false,
          "type": "boolean"
        },
        "ignore_new_state": {
          "title": "Ignore New State",
          "description": "If set to True, ignores the current checkpoint state.",
          "default": false,
          "type": "boolean"
        },
        "remove_stale_metadata": {
          "title": "Remove Stale Metadata",
          "description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    },
    "GlueProfilingConfig": {
      "title": "GlueProfilingConfig",
      "type": "object",
      "properties": {
        "row_count": {
          "title": "Row Count",
          "description": "The parameter name for row count in glue table.",
          "type": "string"
        },
        "column_count": {
          "title": "Column Count",
          "description": "The parameter name for column count in glue table.",
          "type": "string"
        },
        "unique_count": {
          "title": "Unique Count",
          "description": "The parameter name for the count of unique value in a column.",
          "type": "string"
        },
        "unique_proportion": {
          "title": "Unique Proportion",
          "description": "The parameter name for the proportion of unique values in a column.",
          "type": "string"
        },
        "null_count": {
          "title": "Null Count",
          "description": "The parameter name for the count of null values in a column.",
          "type": "string"
        },
        "null_proportion": {
          "title": "Null Proportion",
          "description": "The parameter name for the proportion of null values in a column.",
          "type": "string"
        },
        "min": {
          "title": "Min",
          "description": "The parameter name for the min value of a column.",
          "type": "string"
        },
        "max": {
          "title": "Max",
          "description": "The parameter name for the max value of a column.",
          "type": "string"
        },
        "mean": {
          "title": "Mean",
          "description": "The parameter name for the mean value of a column.",
          "type": "string"
        },
        "median": {
          "title": "Median",
          "description": "The parameter name for the median value of a column.",
          "type": "string"
        },
        "stdev": {
          "title": "Stdev",
          "description": "The parameter name for the standard deviation of a column.",
          "type": "string"
        },
        "partition_patterns": {
          "title": "Partition Patterns",
          "description": "Regex patterns for filtering partitions for profile. The pattern should be a string like: \"{'key':'value'}\".",
          "default": {
            "allow": [
              ".*"
            ],
            "deny": [],
            "ignoreCase": true
          },
          "allOf": [
            {
              "$ref": "#/definitions/AllowDenyPattern"
            }
          ]
        }
      },
      "additionalProperties": false
    }
  }
}

Compatibility

To capture lineage across Glue jobs and databases, a requirements must be met – otherwise the AWS API is unable to report any lineage. The job must be created in Glue Studio with the "Generate classic script" option turned on (this option can be accessed in the "Script" tab). Any custom scripts that do not have the proper annotations will not have reported lineage.

Code Coordinates

• Class Name: datahub.ingestion.source.aws.glue.GlueSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Glue, feel free to ping us on our Slack