DataCater can stream insertions and updates processed by a data pipeline to a REST/HTTP endpoint. Deletions are ignored.
This sink connector supports the following configuration options:
The URI of the REST endpoint, including the HTTP scheme, i.e., http:// or https://.
POST, PUT, PATCH or DELETE.
The header for the HTTP request (default: Content-Type:application/json). Multiple header entries can be separated by a pipe (|).
When the REST endpoint is unavailable, the sink connector automatically retries sending requests to the REST endpoint up to a given number of attempts (default: 10).
The sink connectors waits a given amount of time between making retry attempts (default: 10,000 ms).
If the REST endpoint is using time-based access tokens for authentication, you can here specify an URI, which is called for generating a new token when the REST endpoint returns HTTP status code 401.
HTTP method to use for requesting a new token.
HTTP request headers to use for requesting a new token.
HTTP request body to use for requesting a new token. Only available if POST is used as Token exchange HTTP method.
If the token endpoint does not directly return the token string but a JSON object, you may provide a JSON pointer to locate the token string in the JSON structure.
DataCater does not require schema mapping for this sink connector but passes the pipeline records as JSON objects to the REST endpoint.
Note that string values, which hold valid JSON objects or arrays, are automatically transformed into a JSON representation. This way, the sink connector allows to ingest arbitrarily complex JSON structures into REST endpoints.
In the configuration fields HTTP URI and HTTP request header, you can make use of the Freemarker template engine to build dynamic requests, which is especially useful when using temporary access tokens.
We provide the following variables for the templating: