--source-type | postgres | mysql | mssql | json_schema | avro | protobuf | python | pyspark | typescript | java | s3 | dataframe | kotlin | swift | php | golang | ✓ | | The type of data asset. For databases (postgres, mysql, mssql) the check will be performed for all tables within the database. For protobuf/avro the check will be performed for all file(s) |
--include-unchanged-assets | boolean | | false | Include assets that are the same as Gable’s registered version of the asset. This is useful for checking the current state of an asset but should not be used for automated checks on branches. | | | | | | | | | | | | | | | |
-o, --output | text | json | markdown | | text | Format of the output. Options are: text (default), json, or markdown which is intended to be used as a PR comment | | | | | | | | | | | | | |
--host, -h | text | | | The host name of the production database, for example ‘service-one.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com’. Despite not needing to connect to the production database, the host is still needed to generate the unique resource name for the real database tables (data assets). | | | | | | | | | | | | | | | |
--port, -p | integer | | | The port of the production database. Despite not needing to connect to the production database, the port is still needed to generate the unique resource name for the real database tables (data assets). | | | | | | | | | | | | | | | |
--db | text | | | The name of the production database. Despite not needing to connect to the production database, the database name is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the database name, so this value may not match the name of the database in the proxy database instance. If this is the case, you can set the —proxy-db value to the name of the database in the proxy instance, but we’ll use the value of —db to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’. | | | | | | | | | | | | | | | |
--schema, -s | text | | | The schema of the production database containing the table(s) to check. Despite not needing to connect to the production database, the schema is still needed to generate the unique resource name for the real database tables (data assets). Database naming convention frequently includes the environment (production/development/test/staging) in the schema name, so this value may not match the name of the schema in the proxy database instance. If this is the case, you can set the —proxy-schema value to the name of the schema in the proxy instance, but we’ll use the value of —schema to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’. | | | | | | | | | | | | | | | |
--table, --tables, -t | text | | | A comma delimited list of the table(s) to check. If no table(s) are specified, all tables within the provided schema will be checked. Table names in the proxy database instance must match the table names in the production database instance, even if the database or schema names are different. | | | | | | | | | | | | | | | |
--proxy-host, -ph | text | | | The host string of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow. | | | | | | | | | | | | | | | |
--proxy-port, -pp | integer | | | The port of the database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow. | | | | | | | | | | | | | | | |
--proxy-db, -pdb | text | | | Only needed if the name of the database in the proxy instance is different than the name of the production database. If not specified, the value of —db will be used to generate the unique resource name for the data asset. For example, if your production database is ‘prod_service_one’, but your test database is ‘test_service_one’, you would set —db to ‘prod_service_one’ and —proxy-db to ‘test_service_one’. | | | | | | | | | | | | | | | |
--proxy-schema, -ps | text | | | Only needed if the name of the schema in the proxy instance is different than the name of the schema in the production database. If not specified, the value of —schema will be used to generate the unique resource name for the data asset. For example, if your production schema is ‘production’, but your test database is ‘test’, you would set —schema to ‘production’ and —proxy-schema to ‘test’. | | | | | | | | | | | | | | | |
--proxy-user, -pu | text | | | The user that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow. | | | | | | | | | | | | | | | |
--proxy-password, -ppw | text | | | If specified, the password that will be used to connect to the proxy database instance that serves as the proxy for the production database. This is the database that Gable will connect to when checking tables in the CI/CD workflow. | | | | | | | | | | | | | | | |
--files | tuple | | | Space delimited path(s) to the assets to check, with support for glob patterns. | | | | | | | | | | | | | | | |
--project-root | text | | | This should be the directory location of the Python project that will be analyzed. | | | | | | | | | | | | | | | |
--emitter-function | text | | | Name of the emitter function | | | | | | | | | | | | | | | |
--emitter-payload-parameter | text | | | Name of the parameter representing the event payload | | | | | | | | | | | | | | | |
--event-name-key | text | | | Input must be a ”.” delimited list of field property access directives or array indexes to the event name key field. The field property access directive is a valid dictionary key, or the wildcard character *. The array index is square brackets[] with either a digit in it (targeting a specific element of the array), or * (targeting all elements in the array). Example: “fieldName.[0].eventName” describes the access pay to the event name in: {‘fieldName’: [{‘eventName’: ‘event_one’}]}. | | | | | | | | | | | | | | | |
--emitter-file-path | text | | | Relative path from the root of the project to the file that contains the emitter function | | | | | | | | | | | | | | | |
--exclude | text | | | Comma separated list of paths to be excluded from the analysis, with support for glob patterns. Gable automatically excludes ’**/node_modules, **/pycache, /.*’. Example: ‘/tests,docs/*‘ | | | | | | | | | | | | | | | |
--project-root | text | | | This should be the directory location of the Typescript project that will be analyzed. | | | | | | | | | | | | | | | |
--library | brandviews | segment | amplitude | udf | | | This should indicate the library emitting the events you want detected as data assets. | | | | | | | | | | | | |
--rules-file | text | | | File containing match rules for egress points. Can be used in conjunction with —library, but takes precedence over —emitter-* args. | | | | | | | | | | | | | | | |
--node-modules-include | text | | | Comma delimited list of filenames or patterns of node modules to include in the analysis. | | | | | | | | | | | | | | | |
--emitter-file-path | text | | | DEPRECATED: Use —emitter-location instead. | | | | | | | | | | | | | | | |
--emitter-location | text | | | NPM package name, or relative path from the root of the project to the file that contains the emitter function | | | | | | | | | | | | | | | |
--emitter-function | text | | | Name of the emitter function. This can be a standalone function like ‘trackEvent’ or a class method like ‘AnalyticsClient.track’ | | | | | | | | | | | | | | | |
--emitter-payload-parameter | text | | | Name of the parameter representing the event payload | | | | | | | | | | | | | | | |
--emitter-name-parameter | text | | | Name of the emitter function parameter that contains the event name. Either this option, or the —event-name-key option must be provided when using —emitter-function. | | | | | | | | | | | | | | | |
--event-name-key | text | | | Name of the event property that contains the event name. Either this option, or the —emitter-name-parameter option must be provided when using —emitter-function. | | | | | | | | | | | | | | | |
--exclude | text | | | Comma delimited list of filenames or extended globbing patterns of node modules to include in the analysis. Defaults toexclude common test patterns like *.test.js, *.spec.js, etc. | | | | | | | | | | | | | | | |
--project-root | text | | | This should be the directory location of the project containing the Pyspark job that will analyzed. | | | | | | | | | | | | | | | |
--spark-job-entrypoint | text | | | Entrypoint to execute spark job, starting with python file and including any arguments. Example: “main.py —arg1 value1 —arg2 value2” | | | | | | | | | | | | | | | |
--connection-string | text | | | Connection string to Hive cluster used to pull schemas of input tables | | | | | | | | | | | | | | | |
--metastore-connection-string | text | | | Connection string to the Hive metastore used to pull S3 to table mappings | | | | | | | | | | | | | | | |
--csv-schema-file | text | | | Path to the CSV schema file | | | | | | | | | | | | | | | |
--csv-path-to-table-file | text | | | Path to a CSV schema file containing a mapping of Delta table paths to table names | | | | | | | | | | | | | | | |
--config-file | file | | | Path to YAML configuration file containing the necessary configurations for the Pyspark job. | | | | | | | | | | | | | | | |
--config-entrypoint-path | text | | | The path to the property of the YAML config file containing the spark job entrypoint, which is the main Python script for the job. For example: ‘spec.mainApplicationFile’ | | | | | | | | | | | | | | | |
--config-args-path | text | | | The path to the property of the YAML config file containing the spark job arguments. For example: ‘spec.arguments’ | | | | | | | | | | | | | | | |
--bucket | text | | | This should indicate the S3 bucket containing the files to be analyzed. | | | | | | | | | | | | | | | |
--include-prefix | text | | | This optional parameter allows you to specify what to include in your S3 bucket. If not specified, all files in the bucket will be analyzed. | | | | | | | | | | | | | | | |
--exclude-prefix | text | | | This optional parameter allows you to specify what to exclude in your S3 bucket. If —include-prefix is specified, this parameter must be a subset of include to be considered. | | | | | | | | | | | | | | | |
--lookback-days | integer | | 2 | Number of days to look back from the latest day in the list of paths, defaults to 2. For example if the latest path is 2024/01/02, and lookback_days is 3, then the paths return will have 2024/01/02, 2024/01/01, and 2023/12/31 | | | | | | | | | | | | | | | |
--history | boolean | | false | This optional parameter allows you to do a historical analysis between 2 dates. | | | | | | | | | | | | | | | |
--skip-profiling | boolean | | false | This optional parameter allows you to turn off data profiling. | | | | | | | | | | | | | | | |
--row-sample-count | integer | | 1000 | Number of rows of data per file to sample for schema detection and data profiling. Default is 1000. Accuracy increases with larger sample size, but processing time and AWS costs also increases. | | | | | | | | | | | | | | | |
--recent-file-count | integer (≥1) | | 3 | Specifies the number of most recent files whose schema will be used for inference per data asset. Default is 3. For example, if the latest file is 2024/01/10 and --recent-file-count is 2, then only files 2024/01/10 and 2024/01/09 will be used for schema inference, even if —lookback-days is greater than 2. Increase this value to improve schema accuracy over more schema history, at the cost of increased runtime. Must be at least 1. | | | | | | | | | | | | | | | |
--endpoint | text | | | Customer API endpoint for Gable, in the format https://api.company.gable.ai/. Can also be set with the GABLE_API_ENDPOINT environment variable. | | | | | | | | | | | | | | | |
--api-key | text | | | API Key for Gable. Can also be set with the GABLE_API_KEY environment variable. | | | | | | | | | | | | | | | |
