Skip to main content
Registers a data asset with Gable

​
Usage

gable data-asset register [OPTIONS]

​
Options

OptionTypeRequiredDefaultDescription
--source-typepostgresmysqlmssqljson_schemaavroprotobufpythonpysparktypescriptjavas3dataframekotlinswiftphpgolangThe type of data asset. For databases (mysql, mssql) a data asset is a table within the database. For protobuf/avro a data asset is message/schema within a file.
--dry-runbooleanfalsePerform a dry run without actually registering the data asset.
--host, -htextThe host name of the production database, for example ‘service-one.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com’. Despite not needing to connect to the production database, the host is still needed to generate the unique resource name for the real database tables (data assets).
--port, -pintegerThe port of the production database. Despite not needing to connect to the production database, the port is still needed to generate the unique resource name for the real database tables (data assets).
--dbtextThe name of the production database. Despite not needing to connect to the production database, the database name is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the database name, so this value may not match the name of the database in the proxy database instance. If this is the case, you can set the —proxy-db value to the name of the database in the proxy instance, but we’ll use the value of —db to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’.
--schema, -stextThe schema of the production database containing the table(s) to register. Despite not needing to connect to the production database, the schema is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the schema name, so this value may not match the name of the schema in the proxy database instance. If this is the case, you can set the —proxy-schema value to the name of the schema in the proxy instance, but we’ll use the value of —schema to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’.
--table, --tables, -ttextA comma delimited list of the table(s) to register. If no table(s) are specified, all tables within the provided schema will be registered. Table names in the proxy database instance must match the table names in the production database instance, even if the database or schema names are different.
--proxy-host, -phtextThe host string of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
--proxy-port, -ppintegerThe port of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
--proxy-db, -pdbtextOnly needed if the name of the database in the proxy instance is different than the name of the production database. If not specified, the value of —db will be used to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’.
--proxy-schema, -pstextOnly needed if the name of the schema in the proxy instance is different than the name of the schema in the production database. If not specified, the value of —schema will be used to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’.
--proxy-user, -putextThe user that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
--proxy-password, -ppwtextIf specified, the password that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when registering tables in the CI/CD workflow.
--filestupleSpace delimited path(s) to the assets to register, with support for glob patterns.
--project-roottextThis should be the directory location of the Python project that will be analyzed.
--emitter-functiontextName of the emitter function
--emitter-payload-parametertextName of the parameter representing the event payload
--event-name-keytextInput must be a ”.” delimited list of field property access directives or array indexes to the event name key field. The field property access directive is a valid dictionary key, or the wildcard character *. The array index is square brackets[] with either a digit in it (targeting a specific element of the array), or * (targeting all elements in the array). Example: “fieldName.[0].eventName” describes the access pay to the event name in: {‘fieldName’: [{‘eventName’: ‘event_one’}]}.
--emitter-file-pathtextRelative path from the root of the project to the file that contains the emitter function
--excludetextComma separated list of paths to be excluded from the analysis, with support for glob patterns. Gable automatically excludes ’**/node_modules, **/pycache, /.*’. Example: ‘/tests,docs/*‘
--project-roottextThis should be the directory location of the project containing the Pyspark job that will analyzed.
--spark-job-entrypointtextEntrypoint to execute spark job, starting with python file and including any arguments. Example: “main.py —arg1 value1 —arg2 value2”
--connection-stringtextConnection string to Hive cluster used to pull schemas of input tables
--metastore-connection-stringtextConnection string to the Hive metastore used to pull S3 to table mappings
--csv-schema-filetextPath to the CSV schema file
--csv-path-to-table-filetextPath to a CSV schema file containing a mapping of Delta table paths to table names
--config-filefilePath to YAML configuration file containing the necessary configurations for the Pyspark job.
--config-entrypoint-pathtextThe path to the property of the YAML config file containing the spark job entrypoint, which is the main Python script for the job. For example: ‘spec.mainApplicationFile’
--config-args-pathtextThe path to the property of the YAML config file containing the spark job arguments. For example: ‘spec.arguments’
--project-roottextThis should be the directory location of the Typescript project that will be analyzed.
--librarybrandviewssegmentamplitudeudfThis should indicate the library emitting the events you want detected as data assets.
--rules-filetextFile containing match rules for egress points. Can be used in conjunction with —library, but takes precedence over —emitter-* args.
--node-modules-includetextComma delimited list of filenames or patterns of node modules to include in the analysis.
--emitter-file-pathtextDEPRECATED: Use —emitter-location instead.
--emitter-locationtextNPM package name, or relative path from the root of the project to the file that contains the emitter function
--emitter-functiontextName of the emitter function. This can be a standalone function like ‘trackEvent’ or a class method like ‘AnalyticsClient.track’
--emitter-payload-parametertextName of the parameter representing the event payload
--emitter-name-parametertextName of the emitter function parameter that contains the event name. Either this option, or the —event-name-key option must be provided when using —emitter-function.
--event-name-keytextName of the event property that contains the event name. Either this option, or the —emitter-name-parameter option must be provided when using —emitter-function.
--excludetextComma delimited list of filenames or extended globbing patterns of node modules to include in the analysis. Defaults toexclude common test patterns like *.test.js, *.spec.js, etc.
--inventory-dirtextLocal directory or S3 URI where inventory .csv.gz files are stored. Used when —use-inventory is enabled.
--use-inventorybooleanfalseEnable S3 Inventory-based discovery instead of listing files live from S3.
--buckettextThis should indicate the S3 bucket containing the files to be analyzed.
--include-prefixtextThis optional parameter allows you to specify what to include in your S3 bucket. If not specified, all files in the bucket will be analyzed.
--exclude-prefixtextThis optional parameter allows you to specify what to exclude in your S3 bucket. If —include-prefix is specified, this parameter must be a subset of include to be considered.
--lookback-daysinteger2Number of days to look back from the latest day in the list of paths, defaults to 2. For example if the latest path is 2024/01/02, and lookback_days is 3, then the paths return will have 2024/01/02, 2024/01/01, and 2023/12/31
--historybooleanfalseThis optional parameter allows you to do a historical analysis between 2 dates.
--skip-profilingbooleanfalseThis optional parameter allows you to turn off data profiling.
--row-sample-countinteger1000Number of rows of data per file to sample for schema detection and data profiling. Default is 1000. Accuracy increases with larger sample size, but processing time and AWS costs also increases.
--recent-file-countinteger (≥1)3Specifies the number of most recent files whose schema will be used for inference per data asset. Default is 3. For example, if the latest file is 2024/01/10 and --recent-file-count is 2, then only files 2024/01/10 and 2024/01/09 will be used for schema inference, even if —lookback-days is greater than 2. Increase this value to improve schema accuracy over more schema history, at the cost of increased runtime. Must be at least 1.
--endpointtextCustomer API endpoint for Gable, in the format https://api.company.gable.ai/. Can also be set with the GABLE_API_ENDPOINT environment variable.
--api-keytextAPI Key for Gable. Can also be set with the GABLE_API_KEY environment variable.

​
Example

gable data-asset register --source-type mysql \
    --host prod.pg.db.host --port 5432 --db transit --schema public --table routes \
    --proxy-host localhost --proxy-port 5432 --proxy-user root --proxy-password password