Working with utilities

General

Utilities are single task/process tools to do specific actions. They are generally for admins and advanced users. Depending on the utility it can manipulate files, pull reports, process information, update project assets (such as schemas), and run sophisticated processes like 'fuzzy matching'. The Inputs and Outputs for these utilities can be specifically uploaded to the utility area or can be a project asset (depending on the utility).

Utility List

Available (Admins and Utility Users)

Data migration focussed

• Data Files: Split CSV into smaller chunks - Takes a CSV in and converts it into a number of smaller CSVs based on row count(maximum of 100 files), and places them in the project.
• DM Report: Target Schema Mapping Report - Creates an Excel file detailing the mappings for a Target schema in a project.
• Export Schema - Generate a csv extract of the schema metadata in a format suitable for ingestion.
  
• Generate Synthetic Field Descriptions - Generate synthetic descriptions for schema fields using AI.
• SFTP: Export a Conversion Run from a Project - Export all files from a conversion run to an SFTP server. Files are uploaded into a folder named with the timestamp of the Conversion Run. To use this SFTP utility the access credentials for the SFTP server (keys) must have been added to this deployment by your Zengines support team.

Data Lineage focussed

• Import Lineage Descriptions - Given a target project and a CSV file with module or variable descriptions, apply those descriptions.
• Lineage Search Report - For a configured set of target module-variables, describe how those variables are populated.
• Lineage Variable Report - Write a statement/usage report of variables and their descendants.
• SRD Module Variables - Given the db dump of the CST report configurations, produce a lineage search config file for a given SRD and account type.
• Analyze Snowflake SQL - Finds and parses snowflake SQL and produces lineage documentation.

General

• Data Files: combine multiple CSVs - Combine multiple delimited data files (CSV or pipe-delimited .txt) in the same project that have the same column headers into a single file, preserving the input delimiter and extension.

• Data Files: Convert XLSX to CSVs - Read in an XLSX file from a project and save each sheet back in the project as a CSV file.
  

• Fuzzy Matching - Report on possible duplication of data between a left and right data records.

• JSON to CSV - Converts a JSON file into a CSV file. Handles nested JSON by flattening.

• Reconciliation - Compare data in two data files and report on exceptional differences.
• Sample Data File - Create a smaller randomly sampled version of a Data file. The sampled file will be created as a project file in the same project with up to 1000 rows.
• SFTP: Import Files to Project - Import .csv, .json, .txt, .xls, and .xlsx files from an SFTP server to a project. To use this SFTP utility the access credentials for the SFTP server (keys) must have been added to this deployment by your Zengines support team.
• SFTP: Export Files from a Project - Export a single file or a folder (including files and subfolders) from a project to an SFTP server added to the platform.
• SFTP: List Files and Directories - Connect to an SFTP server and produce a CSV report listing all files and directories at the specified path. To use this SFTP utility the access credentials for the SFTP server (keys) must have been added to this deployment by your Zengines support team.
• SQL: Modify a CSV data file - Read in a CSV data file from a project. Using an ANSI SQL (DuckDB flavor) statement, modify, filter, and then push the modified form back to the project as a new file.