Ingest data using Airbyte

Y42 leverages Airbyte (opens in a new tab) to integrate data from external sources. For each data ingestion job, Y42 creates an instance of the Airbyte open-source connector (opens in a new tab). To pull data in Y42 you will need two define two objects:

• Source: For each Airbyte connector you will need to create a source object. Sources provide metadata information about external data sources, such as table or column-level information.
• Secret: Secrets hold information about authentication. When you set up a source, you don't need to worry about creating additional accounts, you can use your existing account with the data provider (e.g., username and password for database sources or an API token)

Every airbyte connectors allows various levels of configurations and features: incremental imports, CDC replication, and others. For a deeper dive into these features, please refer to the individual documentation pages for each source.

Add an Airbyte source

FAQ

Do I need to whitelist any IP addresses?

Yes, you do. The IP address you'll need to whitelist is 35.198.72.34. All data extraction requests come from Y42, and this IP address is where those requests originate. To establish a successful connection with a database, this IP must be whitelisted.

Do I need to set up additional accounts to make it work?

No. We use the Airbyte open-source connectors, and Y42 takes care of all the computation. There's no need to worry about setting up additional accounts, like an Airbyte cloud account, to get things up and running.

Can I verify the row count of a table after data ingestion?

Yes, you can inspect the row count and other ingestion statistics easily. Just go to the bottom section of the app and hover over (or click on) an ingestion job that successfully finished.

What kind of sync mode will be performed when a scheduled build updates a source table?

If a source supports incremental imports, a scheduled build will, by default, trigger an incremental import. To perform a full refresh, you need to override the default configurations and include the --full-refresh flag at the end of the y42 build command.