MongoDB

Module mongodb

Important Capabilities

Capability	Status	Notes
Table-Level Lineage	✅	Enabled by default

This plugin extracts the following:

• Databases and associated metadata
• Collections in each database and schemas for each collection (via schema inference)

By default, schema inference samples 1,000 documents from each collection. Setting schemaSamplingSize: null will scan the entire collection. Moreover, setting useRandomSampling: False will sample the first documents found without random selection, which may be faster for large collections.

Note that schemaSamplingSize has no effect if enableSchemaInference: False is set.

Really large schemas will be further truncated to a maximum of 300 schema fields. This is configurable using the maxSchemaSize parameter.

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[mongodb]'

Starter Recipe

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

source:
  type: "mongodb"
  config:
    # Coordinates
    connect_uri: "mongodb://localhost"

    # Credentials
    username: admin
    password: password
    authMechanism: "DEFAULT"

    # Options
    enableSchemaInference: True
    useRandomSampling: True
    maxSchemaSize: 300

sink:
  # sink configs

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options

Field [Required]	Type	Description	Default	Notes
authMechanism	string	MongoDB authentication mechanism.	None
connect_uri	string	MongoDB connection URI.	mongodb://localhost
enableSchemaInference	boolean	Whether to infer schemas.	True
maxDocumentSize	integer		16793600
maxSchemaSize	integer	Maximum number of fields to include in the schema.	300
options	object	Additional options to pass to pymongo.MongoClient().	None
password	string	MongoDB password.	None
schemaSamplingSize	integer	Number of documents to use when inferring schema size. If set to 0, all documents will be scanned.	1000
useRandomSampling	boolean	If documents for schema inference should be randomly selected. If False, documents will be selected from start.	True
username	string	MongoDB username.	None
env	string	The environment that all assets produced by this connector belong to	PROD
collection_pattern	AllowDenyPattern	regex patterns for collections to filter in ingestion.	{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
collection_pattern.allow	array(string)		None
collection_pattern.deny	array(string)		None
collection_pattern.ignoreCase	boolean	Whether to ignore case sensitivity during pattern matching.	True
database_pattern	AllowDenyPattern	regex patterns for databases to filter in ingestion.	{'allow': ['.*'], 'deny': [], 'ignoreCase': True}
database_pattern.allow	array(string)		None
database_pattern.deny	array(string)		None
database_pattern.ignoreCase	boolean	Whether to ignore case sensitivity during pattern matching.	True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "MongoDBConfig",
  "description": "Any source that produces dataset urns in a single environment should inherit this class",
  "type": "object",
  "properties": {
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "connect_uri": {
      "title": "Connect Uri",
      "description": "MongoDB connection URI.",
      "default": "mongodb://localhost",
      "type": "string"
    },
    "username": {
      "title": "Username",
      "description": "MongoDB username.",
      "type": "string"
    },
    "password": {
      "title": "Password",
      "description": "MongoDB password.",
      "type": "string"
    },
    "authMechanism": {
      "title": "Authmechanism",
      "description": "MongoDB authentication mechanism.",
      "type": "string"
    },
    "options": {
      "title": "Options",
      "description": "Additional options to pass to `pymongo.MongoClient()`.",
      "default": {},
      "type": "object"
    },
    "enableSchemaInference": {
      "title": "Enableschemainference",
      "description": "Whether to infer schemas. ",
      "default": true,
      "type": "boolean"
    },
    "schemaSamplingSize": {
      "title": "Schemasamplingsize",
      "description": "Number of documents to use when inferring schema size. If set to `0`, all documents will be scanned.",
      "default": 1000,
      "exclusiveMinimum": 0,
      "type": "integer"
    },
    "useRandomSampling": {
      "title": "Userandomsampling",
      "description": "If documents for schema inference should be randomly selected. If `False`, documents will be selected from start.",
      "default": true,
      "type": "boolean"
    },
    "maxSchemaSize": {
      "title": "Maxschemasize",
      "description": "Maximum number of fields to include in the schema.",
      "default": 300,
      "exclusiveMinimum": 0,
      "type": "integer"
    },
    "maxDocumentSize": {
      "title": "Maxdocumentsize",
      "default": 16793600,
      "exclusiveMinimum": 0,
      "type": "integer"
    },
    "database_pattern": {
      "title": "Database Pattern",
      "description": "regex patterns for databases to filter in ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "collection_pattern": {
      "title": "Collection Pattern",
      "description": "regex patterns for collections to filter in ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    }
  },
  "additionalProperties": false,
  "definitions": {
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.mongodb.MongoDBSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for MongoDB, feel free to ping us on our Slack