Ingesting Data into Shaper

You can ingest data into Shaper’s database through the HTTP API. We create tables and add columns automatically based on the data ingested.

1. To ingest data you first need to create an API Key with “Ingest Data” permission in the “Admin” settings of the Shaper UI.

   Tip

   When you run Shaper locally for development and you haven’t set up a user yet, the system doesn’t require any authentication.

   Not only the UI is accessible without login, but also when using the Ingest API via HTTP you can skip the authentication.

2. Then you can write JSON data to the HTTP API directly: Endpoint:

   POST http://localhost:5454/api/data/:tablename

   Authentication is done through Bearer token in the Authorization header.

   You can pass a single JSON object or an array of objects.

   Example:

   Single

   curl -X POST http://localhost:5454/api/data/my_table \
   -H "Authorization: Bearer <your-api-key>" \
   -H "Content-Type: application/json" \
   -d '{"col1": "value1", "col2": 124}'

   Multiple

   curl -X POST http://localhost:5454/api/data/my_table \
   -H "Authorization: Bearer <your-api-key>" \
   -H "Content-Type: application/json" \
   -d '[{"col1": "value1", "col2": 123}, {"col1": "value2", "col2": 456}]'

3. If you click on “New” in the sidebar now, you can run the following query:

   DESC my_table;
   SELECT * FROM my_table;

   column_name	column_type	null	key	default	extra
   _id	VARCHAR	YES
   _ts	TIMESTAMP	YES
   col1	VARCHAR	YES
   col2	DOUBLE	YES

   You can see that Shaper auto-creates columns with appropriate data types, and we always add _id and _ts columns. You can override the default values by passing data in the JSON object for them.

   Shaper detects boolean and numbers in JSON. We also detect date and timestamp strings in various formats. If any data is a complex data type such as an array or object, we store them as JSON column in DuckDB.

Automate Data Loading

Shaper supports tasks to automatically run SQL scripts in the background similar to CRON jobs.

Tasks are especially helpful to routinely load and cleanup data.

Learn more in the Tasks Documentation.