5. Schedule a build

Schedule a build

With a robust pipeline and data quality tests in place, it's time to take advantage of Y42's scheduling. By setting up scheduled builds, we ensure our data is updated regularly. In this section, we'll set up a scheduler and alert.

Create and run a scheduled build

Set up a schedule

  1. Open the bottom drawer
  2. Navigate to Build scheduler
  3. Click on + Add
  4. Provide a name for the scheduled build: orders_every_monday
  5. Enter the following selector statement: y42 build -s source:raw_space_fuel_shop.public_orders+
  6. Click on Visualize DAG and verify that you see the DAG preview
Scheduler DAG preview

Scheduler DAG preview

  1. Set up the Schedule frequency unit: Weekly
  2. Set Options to Run every Monday
  3. Click on Save'
  4. Commit and push these changes with an appropriate commit title (e.g., scheduler: add schedule to build orders every Monday)

Trigger a manual run

  1. You should now see the schedule as a row in the Build scheduler tab
  2. Click on Trigger Run to trigger an ad-hoc out-of-schedule run
  3. You'll see a blue bar appearing in the Last Runs column
  4. Once the run finishes, the blue bar will change color to reflect the run status: green for a successful run, yellow for a partially successful run, and red for an invalid one

View the Schedule as code

  1. In the Asset Editor, select the Code view

  2. Click on the orchestrations directory

  3. Click on orders_15m_schedule.yml

  4. You should see Y42’s YML structure for orchestrations:

    version: 2
    - name: orders_15m_schedule
    enabled: true
    command: y42 build -s y42 build -s source:raw_space_fuel_shop.public_orders+
    cron: 0 0/1 1/1 * *
    apiVersion: v1

Create an alert

It's great to have automatic pipeline runs set up, but we also want to know when the pipeline fails. Therefore, we'll set up an alert that notifies us if something goes wrong.

  1. Open the bottom drawer
  2. Navigate to Alerts
  3. Click on Add an alert
  4. Provide a name for the alert: invalid_orders_run
  5. Under Trigger, select orders_15m_schedule
  6. Select Partially invalid and Invalid as statuses
  7. Under destinations, provide your user email
  8. Commit and push the changes with an appropriate commit title (e.g., alert: add failure alert for orders pipeline)

Up next

That's all there is to setting up orchestrations! With everything configured and tested, there's just one thing left to do: time to merge our changes to the main branch.