Ingest data

Create a working branch

As you may have gathered from the asset lifecycle, Y42 follows the GitOps paradigm. We'll dive further into this in section 6, but let's implement a best practice immediately. We should create a new branch whenever we want to make changes to our pipelines — such as adding a data source. This gives us a safe environment to work in without disrupting live pipelines.

To create a new branch, click on main [default] at the top center of the screen and type in a name for the new branch (e.g., getting-started-guide). Now hit enter; that's all there is to it.

Create a new source asset

To ingest the Space Fuel Shop dataset, we'll use Airbyte's Postgres connector:

1. In the left panel, select the List view under Asset Editor
2. Press on + in the file navigator
3. Select Source
4. Select Ingest
5. Select Postgres (Airbyte) as the Ingest type
6. Name the source: raw_space_fuel_shop_data

Now, we'll configure the source asset to point to the right Postgres database and pull in all of the data.

Add a connection secret

Source assets (generally) need a secret to authorize their connection to the data source.

1. In the Authorization panel, click on the search box Search a secret. When no secrets exist yet, you'll be prompted to add a new secret. If secrets do exist, you can select one of them or click on Add new.

2. Name the secret postgres_space_fuel_shop_data_{YOUR_NAME}, replacing YOUR_NAME with your name.

3. Insert the following type-specific configuration variables:

   Variable	Value
   Host	35.198.165.208
   Port	5432 (default)
   Database Name	y42-demo-data
   Username	public_reader
   Password	dontpanic

4. Set Replication Method to Standard

5. Set SSH Tunnel Method to No Tunnel

6. Leave JDBC URL Parameters (Advanced) empty

7. Set SSL Modes to allow

When you have entered all authorization settings, click Next in the Settings panel, and wait for the schema to load in the Schema selection panel.

Select the the tables and columns you would like to import

1. Open the Schema panel.
2. Wait for the schema to load. For larger datasets this may take a few minutes.
3. Select all three tables: public/payments, public/orders, and public/customers
4. You can open the dropdown of a single table to see the individual columns and types. For now, just import all columns.

Commit the new source assets

Our first source is now fully configured. Before we can build the tables and ingest the data, we'll need to commit and push our changes.

1. Click on the Commit & Push button at the top.
2. Enter a name for your Commit: ingest: add Space Fuel Shop data source.
3. Press Commit.
4. Wait for Y42's pre-configured checks to run.

Y42 will automatically push the changes to the Git remote.

Build: materialize the table

1. Open the drawer at the bottom of your screen
2. Select the Build tab
3. Insert the build command to build and import all three tables: y42 build -s source:raw_space_fuel_shop_data.*
4. Click on Visualize DAG to see the assets this command will build
5. Click on Build now. You'll see a new row added to the build window. As the blue quarter bar indicates, the build is in queueing mode.
6. Click on the build row to see more details, where you'll see the pending jobs. You can also navigate to Logs to view build logs. If you click on an individual build job, you'll see that job's details.
7. After a couple of minutes, the build should be Ready.

Publication: add metadata

We'll add metadata to the assets to get the most out of our data. Y42 has extensive metadata features, which is really helpful when working with many data assets and experts. For now, we'll focus on just a few metadata features: column types, column tests, and column descriptions.

1. Click on raw_orders in the left file selector

2. Click Sync columns

3. Seven columns should be added. If they do not have the correct Type, update the type by double-clicking on the Type and entering the appropriate one:

   Column	Type
   ID	string
   CUSTOMER_ID	string
   UPDATED_AT	timestamp_ntz
   FULFILLMENT_METHOD	string
   SERVICE_STATION	string
   LINE_ITEMS	string
   STATUS	int
   _Y42_EXTRACTED_AT	timestamp_ntz

4. Add the following column tests:

   Column	Test	Values
   ID	Unique
   CUSTOMER_ID	Not null
   FULFILLMENT_METHOD	Accepted Values	SERVICED, SELF_SERVICE

5. Add the following descriptions:

   Column	Type
   UPDATED_AT	Time at which the order was published to the database in CET.
   STATUS	0 for canceled, 1 for declined, 2 for accepted

6. Commit and push the changes with an appropriate commit title (e.g., meta: add raw_orders metadata)

You can add more extensive tests and metadata, of course! As a general practice, it is a good idea to have at least a description per column and the appropriate data types selected.

View the data

We can now take a sneak peek at the data to see how it's shaping up:

1. Open the bottom drawer
2. Navigate to Data, where you will see a preview of the row count
3. Wait for the data to load and verify that it shows up