Use change data capture to stream flat files from a folder in Google Drive to the any data sink and transform them on the way.
At startup, the connector extracts data from all (matching) files from the given folder. After this initial sync, it watches the folder for new or updated files and syncs only relevant changes.
Please create a service account in Google Cloud and share the Google Drive folder with it. Please make sure that you have enabled the Drive API.
This source connector supports the following configuration options:
The content of the JSON-based credentials file provided by Google Cloud for the service account.
The name of the Google Drive folder from which we shall extract files. Please click the Fetch folder name button to fetch the list of available folders.
Regular expression applied to files from the Google Drive folder. Only files with a name matching the regular expression will be extracted. Default value: .* (matches all file names).
The format of the extracted files. At the moment, this connector only supports CSV files, JSON files, and Google Sheets.
Only available for the file formats CSV and Google Sheet. Whether to use the first row of the file for extracting attribute names or not. If this option is set to false, DataCater will generate attribute names based on the index of the attribute, and name them column_1, column_2, etc.
Only available for the file format CSV. The character that delimits different columns (default: ,).
Only available for the file format CSV. Character used for quotes (default: ").
Only available for the file format CSV. Character used for escaping quotes (default: ").
Only available for the file format CSV. String used for separating multiple lines (default: \n).
Only available for the file format CSV. Character used for comments. It must appear at the beginning of a line (default: #).
Only available for the file formats CSV and Google Sheet. Number of lines to skip for parsing from the beginning of the file (default: 0).
Only available for the file format JSON. By default, DataCater expects either a single JSON objedt or an array of JSON objects, each resembling a record, at the root level of the respective file. For all other cases, you may provide a JSON pointer pointing to the location of the data of interest within a possibly deepled-nested JSON structure.
Only available for the file format JSON. Whether JSON files contain a single object or an array of objects (default: false).
Name of the attribute that uniquely identifies records, similar to a primary key in a database system.
The interval in seconds between the synchronization of the Google Drive folder and DataCater (default: 120). When synchronizing, DataCater consumes only those files from the Google Drive folder, which have not yet been processed by the pipeline, allowing to implement change data capture to some degree.
DataCater imports all columns of CSV files and Google Sheets as attributes of type string.
DataCater automatically extends the set of attributes with the attribute __datacater_file_name and fills it with the name of the file.