Google Drive

This source connector allows to stream flat files from a folder in Google Drive to the DataCater platform.


Change Data Capture

At startup, the connector extracts data from all (matching) files from the given folder. After this initial sync, it watches the folder for new or updated files and syncs only relevant changes.


Requirements

Please create a service account in Google Cloud and share the Google Drive folder with it. Please make sure that you have enabled the Drive API.


Configuration

This source connector supports the following configuration options:

Google Cloud credentials (JSON)

The content of the JSON-based credentials file provided by Google Cloud for the service account.

Folder name

The name of the Google Drive folder from which we shall extract files. Please click the Fetch folder name button to fetch the list of available folders.

File name filter

Regular expression applied to files from the Google Drive folder. Only files with a name matching the regular expression will be extracted. Default value: .* (matches all file names).

File format

The format of the extracted files. At the moment, this connector only supports CSV files.

Generate attribute names from CSV header row

Only available for the file format CSV. Whether to use the first row of the CSV file for extracting attribute names or not. If this option is set to false, DataCater will generate attribute names based on the index of the attribute, and name them column_1, column_2, etc.

CSV delimiter value

Only available for the file format CSV. The character that delimits different columns (default: ,).

CSV quote character

Only available for the file format CSV. Character used for quotes (default: ").

CSV quote escape character

Only available for the file format CSV. Character used for escaping quotes (default: ").

CSV line separator

Only available for the file format CSV. String used for separating multiple lines (default: \n).

CSV comment character

Only available for the file format CSV. Character used for comments. It must appear at the beginning of a line (default: #).

CSV number of lines to skip from beginning

Only available for the file format CSV. Number of lines to skip for parsing from the beginning of the file (default: 0).

Primary key column

Name of the attribute that uniquely identifies records, similar to a primary key in a database system.

Sync interval (s)

The interval in seconds between the synchronization of the Google Drive folder and DataCater (default: 120). When synchronizing, DataCater consumes only those files from the Google Drive folder, which have not yet been processed by the pipeline, allowing to implement change data capture to some degree.


Data Types

DataCater imports all columns of a CSV file as attributes of type string.

DataCater automatically extends the set of attributes with the attribute __datacater_file_name and fills it with the name of the file.