gable data-asset register

Registers a data asset with Gable

Usage

gable data-asset register [OPTIONS]

Options

Option	Type	Required	Default	Description
`--source-type`	`postgres`	`mysql`	`mssql`	`json_schema`	`avro`	`protobuf`	`python`	`pyspark`	`typescript`	`java`	`s3`	`dataframe`	`kotlin`	`swift`	`php`	`golang`	The type of data asset. For databases (mysql, mssql) a data asset is a table within the database. For protobuf/avro a data asset is message/schema within a file.
`--dry-run`	boolean		false	Perform a dry run without actually registering the data asset.
`--host`, `-h`	text			The host name of the production database, for example ‘service-one.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com’. Despite not needing to connect to the production database, the host is still needed to generate the unique resource name for the real database tables (data assets).
`--port`, `-p`	integer			The port of the production database. Despite not needing to connect to the production database, the port is still needed to generate the unique resource name for the real database tables (data assets).
`--db`	text			The name of the production database. Despite not needing to connect to the production database, the database name is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the database name, so this value may not match the name of the database in the proxy database instance. If this is the case, you can set the —proxy-db value to the name of the database in the proxy instance, but we’ll use the value of —db to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’.
`--schema`, `-s`	text			The schema of the production database containing the table(s) to register. Despite not needing to connect to the production database, the schema is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the schema name, so this value may not match the name of the schema in the proxy database instance. If this is the case, you can set the —proxy-schema value to the name of the schema in the proxy instance, but we’ll use the value of —schema to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’.
`--table`, `--tables`, `-t`	text			A comma delimited list of the table(s) to register. If no table(s) are specified, all tables within the provided schema will be registered. Table names in the proxy database instance must match the table names in the production database instance, even if the database or schema names are different.
`--proxy-host`, `-ph`	text			The host string of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
`--proxy-port`, `-pp`	integer			The port of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
`--proxy-db`, `-pdb`	text			Only needed if the name of the database in the proxy instance is different than the name of the production database. If not specified, the value of —db will be used to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’.
`--proxy-schema`, `-ps`	text			Only needed if the name of the schema in the proxy instance is different than the name of the schema in the production database. If not specified, the value of —schema will be used to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’.
`--proxy-user`, `-pu`	text			The user that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
`--proxy-password`, `-ppw`	text			If specified, the password that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
`--files`	tuple			Space delimited path(s) to the assets to register, with support for glob patterns.
`--project-root`	text			This should be the directory location of the Python project that will be analyzed.
`--emitter-function`	text			Name of the emitter function
`--emitter-payload-parameter`	text			Name of the parameter representing the event payload
`--event-name-key`	text			Input must be a ”.” delimited list of field property access directives or array indexes to the event name key field. The field property access directive is a valid dictionary key, or the wildcard character . The array index is square brackets[] with either a digit in it (targeting a specific element of the array), or (targeting all elements in the array). Example: “fieldName.[0].eventName” describes the access pay to the event name in: {‘fieldName’: [{‘eventName’: ‘event_one’}]}.
`--emitter-file-path`	text			Relative path from the root of the project to the file that contains the emitter function
`--exclude`	text			Comma separated list of paths to be excluded from the analysis, with support for glob patterns. Gable automatically excludes ’/node_modules, /pycache, */.’. Example: ‘*/tests,docs/‘
`--project-root`	text			This should be the directory location of the project containing the Pyspark job that will analyzed.
`--spark-job-entrypoint`	text			Entrypoint to execute spark job, starting with python file and including any arguments. Example: “main.py —arg1 value1 —arg2 value2”
`--connection-string`	text			Connection string to Hive cluster used to pull schemas of input tables
`--metastore-connection-string`	text			Connection string to the Hive metastore used to pull S3 to table mappings
`--csv-schema-file`	text			Path to the CSV schema file
`--csv-path-to-table-file`	text			Path to a CSV schema file containing a mapping of Delta table paths to table names
`--config-file`	file			Path to YAML configuration file containing the necessary configurations for the Pyspark job.
`--config-entrypoint-path`	text			The path to the property of the YAML config file containing the spark job entrypoint, which is the main Python script for the job. For example: ‘spec.mainApplicationFile’
`--config-args-path`	text			The path to the property of the YAML config file containing the spark job arguments. For example: ‘spec.arguments’
`--project-root`	text			This should be the directory location of the Typescript project that will be analyzed.
`--library`	`brandviews`	`segment`	`amplitude`	`udf`			This should indicate the library emitting the events you want detected as data assets.
`--rules-file`	text			File containing match rules for egress points. Can be used in conjunction with —library, but takes precedence over —emitter-* args.
`--node-modules-include`	text			Comma delimited list of filenames or patterns of node modules to include in the analysis.
`--emitter-file-path`	text			DEPRECATED: Use —emitter-location instead.
`--emitter-location`	text			NPM package name, or relative path from the root of the project to the file that contains the emitter function
`--emitter-function`	text			Name of the emitter function. This can be a standalone function like ‘trackEvent’ or a class method like ‘AnalyticsClient.track’
`--emitter-payload-parameter`	text			Name of the parameter representing the event payload
`--emitter-name-parameter`	text			Name of the emitter function parameter that contains the event name. Either this option, or the —event-name-key option must be provided when using —emitter-function.
`--event-name-key`	text			Name of the event property that contains the event name. Either this option, or the —emitter-name-parameter option must be provided when using —emitter-function.
`--exclude`	text			Comma delimited list of filenames or extended globbing patterns of node modules to include in the analysis. Defaults toexclude common test patterns like .test.js, .spec.js, etc.
`--inventory-dir`	text			Local directory or S3 URI where inventory .csv.gz files are stored. Used when —use-inventory is enabled.
`--use-inventory`	boolean		false	Enable S3 Inventory-based discovery instead of listing files live from S3.
`--bucket`	text			This should indicate the S3 bucket containing the files to be analyzed.
`--include-prefix`	text			This optional parameter allows you to specify what to include in your S3 bucket. If not specified, all files in the bucket will be analyzed.
`--exclude-prefix`	text			This optional parameter allows you to specify what to exclude in your S3 bucket. If —include-prefix is specified, this parameter must be a subset of include to be considered.
`--lookback-days`	integer		2	Number of days to look back from the latest day in the list of paths, defaults to 2. For example if the latest path is 2024/01/02, and lookback_days is 3, then the paths return will have 2024/01/02, 2024/01/01, and 2023/12/31
`--history`	boolean		false	This optional parameter allows you to do a historical analysis between 2 dates.
`--skip-profiling`	boolean		false	This optional parameter allows you to turn off data profiling.
`--row-sample-count`	integer		1000	Number of rows of data per file to sample for schema detection and data profiling. Default is 1000. Accuracy increases with larger sample size, but processing time and AWS costs also increases.
`--recent-file-count`	integer (≥1)		3	Specifies the number of most recent files whose schema will be used for inference per data asset. Default is 3. For example, if the latest file is 2024/01/10 and `--recent-file-count` is 2, then only files 2024/01/10 and 2024/01/09 will be used for schema inference, even if —lookback-days is greater than 2. Increase this value to improve schema accuracy over more schema history, at the cost of increased runtime. Must be at least 1.
`--endpoint`	text			Customer API endpoint for Gable, in the format https://api.company.gable.ai/. Can also be set with the GABLE_API_ENDPOINT environment variable.
`--api-key`	text			API Key for Gable. Can also be set with the GABLE_API_KEY environment variable.

Example

gable data-asset register --source-type mysql \
    --host prod.pg.db.host --port 5432 --db transit --schema public --table routes \
    --proxy-host localhost --proxy-port 5432 --proxy-user root --proxy-password password

Getting Started

auth

contract

data-asset

lineage

Other

gable data-asset register

Usage

Options

Example

​Usage

​Options

​Example

Usage

Options

Example