Get Snowflake Projects Live Faster. Reliably.

Register for webinar
Back to Product Updates

Python Ingest - logging, and incremental loads

Python Ingest assets now support logging and incremental loads.

You can record messages using the logging module for your Python ingest assets. Access these logs by navigating to the Logs tab in the asset's Build History.

Visualize Python asset logs in the Builg history tab.

Visualize Python asset logs in the Builg history tab.

Incremental loads

Y42 has introduced support for incremental data loading through the context.state variable in Python-based data ingestion assets. context.state is a dictionary that gets updated with each asset refresh and can store any key-value pair for your data process.

from y42.v1.decorators import data_loader

import requests
import pandas as pd
import json

@data_loader
def todos(context) -> pd.DataFrame:

    prev_state = context.state.get()

    if prev_state: 
        last_update = prev_state['last_update']
        # perform incremental activities, such as filtering the dataset by last_update variable
    else: 
        # perform full refresh activities 

    context.state.set({"last_update": datetime.utcnow()})

    return df 

Learn more →