Yes Energy® continuously collects energy market data. Knowing when Yes Energy publishes and updates this data helps optimize data pipelines and analytic processes – a feature now possible with the Snowflake Streams functionality in the DataSignals Cloud (DSC) product.
Snowflake’s Data Cloud is powered by an advanced data platform provided as a self-managed service, enabling data storage, processing, and analytic solutions that are faster, easier to use, and far more flexible than traditional offerings.
Streams allow for smarter data consumption by enabling your DataSignals Cloud® (DSC) processes to query only data that has changed since the last time these processes checked (Figure 1). This saves time and Snowflake compute resources. You can set up a consumer object, such as a table, to consume the stream content (see our stream documentation for an example). While you can use the ROW_ID column to track changes to a specific row over time, you can also add additional functionality like the example INGESTIONDATE column (Figure 1). This column captures when the data entered the consumer object, which functions as a proxy for the publication time of the data (depending on how fast and frequently you consume the data out of the stream) and captures the relative timing of when changes affect that row. For more information on the stream-specific metadata columns in this example table, see Snowflake’s Stream documentation.
Figure 1. Example table that stores stream content with an additional INGESTIONDATE timestamp column to capture when the stream content enters the table. With Snowflake streams, your processes can track and/or consume only records that have changed since the last time these processes checked.
In addition to data pipeline optimization, streams also provide a way to track and capture data revisions of non-forecast time series not currently vintaged by Yes Energy (e.g., prices). This is often called "point in time" data (Figure 2).
Figure 2. Example of capturing data revisions for the same time interval across two records. Streams capture non-forecast data revisions. The bottom record is the (hypothetical) revision of the top record. Streams therefore help you maintain the full price history.
Streams also grant more visibility into Yes Energy’s data collection processes. For example, you can back into the publication frequency of a data type using streams, which allows you to further optimize your pipelines by only checking for new data at the publication time.
Save time and compute resources with Snowflake – a complimentary feature for Yes Energy’s DataSignals Cloud customers.
Yes Energy provides documentation and sample code for implementing streams. Snowflake’s Streams documentation is linked in our documentation. Streams are available for major objects on AWS us-east-1 and Azure US East 2 regions.
For more information, check out our Snowflake Partner Page or contact partners@yesenergy.com.
To see how Snowflake enables data storage, processing, and analytic solutions that are faster, easier to use, and far more flexible than traditional offerings, schedule a demo.
At Yes Energy, we understand the complexity and unique challenges of nodal power markets. It’s why we’re committed to helping you Win the Day Ahead™ by delivering superior data how and when you want it.
At Yes Energy, we excel at running data at scale – critical when you’re working with terabytes of data. Because you need answers within hours, not days – and sometimes faster.