Copy data from any source to any destination without any code
ingestr is a command-line app that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
- ✨ copy data from your database into any destination
- ➕ incremental loading:
append,mergeordelete+insert - 🐍 single-command installation
ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
You can install ingestr using the install script:
curl -LsSf https://getbruin.com/install/ingestr | shAlternatively, you can install it with pip:
pip install ingestringestr ingest \
--source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
--source-table 'public.some_data' \
--dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
--dest-table 'ingestr.some_data'That's it.
This command:
- gets the table
public.some_datafrom the Postgres instance. - uploads this data to your BigQuery warehouse under the schema
ingestrand tablesome_data.
You can see the full documentation here.
Join our Slack community here.
Pull requests are welcome. However, please open an issue first to discuss what you would like to change. We maybe able to offer you help and feedback regarding any changes you would like to make.
Note
After cloning ingestr make sure to run make setup to install githooks.
| Source | Destination | |
|---|---|---|
| Databases | ||
| BigQuery | ✅ | ✅ |
| ClickHouse | ✅ | ✅ |
| CrateDB | ✅ | ✅ |
| Databricks | ✅ | ✅ |
| IBM Db2 | ✅ | - |
| DuckDB | ✅ | ✅ |
| DynamoDB | ✅ | - |
| Elasticsearch | ✅ | ✅ |
| GCP Spanner | ✅ | - |
| Local CSV file | ✅ | ✅ |
| Microsoft SQL Server | ✅ | ✅ |
| MongoDB | ✅ | ✅ |
| MotherDuck | ✅ | ✅ |
| MySQL | ✅ | ❌ |
| Oracle | ✅ | ❌ |
| Postgres | ✅ | ✅ |
| Redshift | ✅ | ✅ |
| SAP Hana | ✅ | ❌ |
| Snowflake | ✅ | ✅ |
| SQLite | ✅ | ✅ |
| Trino | ✅ | ✅ |
| Platforms | ||
| Adjust | ✅ | - |
| Airtable | ✅ | - |
| Amazon Kinesis | ✅ | - |
| Apache Kafka | ✅ | - |
| AppsFlyer | ✅ | - |
| App Store | ✅ | - |
| Asana | ✅ | - |
| Attio | ✅ | - |
| Chess.com | ✅ | - |
| Facebook Ads | ✅ | - |
| GitHub | ✅ | - |
| Gorgias | ✅ | - |
| Google Sheets | ✅ | - |
| Google Ads | ✅ | - |
| Google Analytics | ✅ | - |
| Granola | ✅ | - |
| Klaviyo | ✅ | - |
| LinkedIn Ads | ✅ | - |
| Notion | ✅ | - |
| Personio | ✅ | - |
| Phantombuster | ✅ | - |
| Pipedrive | ✅ | - |
| S3 | ✅ | - |
| Salesforce | ✅ | - |
| Shopify | ✅ | - |
| Slack | ✅ | - |
| Smartsheets | ✅ | - |
| Solidgate | ✅ | - |
| Stripe | ✅ | - |
| TikTok Ads | ✅ | - |
| Zendesk | ✅ | - |
Feel free to create an issue if you'd like to see support for another source or destination.
This project is licensed under the MIT License - see the LICENSE file for details.
Some components are licensed under Apache 2.0 - see the NOTICE file for details.