Share this
by Sam Lockshin and Cassie Finer
At Yes Energy®, we see our customers increasingly rely on Snowflake as their central data hub for power market data. Snowflake's long-standing data marketplace has become progressively more valuable as more vendors host their data on the Snowflake Platform. As Snowflake expands its data ingestion, processing, and analytics capabilities, we’re seeing our customers enjoy incremental benefits.
If you use Snowflake for Yes Energy data analytics in power markets, here’s how Snowflake’s latest features can make the product even more beneficial.
Streamline Your Data Analytics in Power Markets with Dynamic Tables
Dynamic tables let you simplify data pipelines by defining the desired end state of your data using SQL. Snowflake then handles the data transformation and pipeline management for you. This reduces the need to have multiple Snowflake objects, tasks, and procedures for processing the data.
Dynamic tables combine the capabilities and performance of a table with the flexible logic of views and materialized views. With the benefit of pipeline management and data observability, we see this as a robust data transformation tool.
In the image below, you can see how dynamic tables provide more flexibility than traditional views. You can query and create streams on dynamic tables like any other table – which is not possible on every view – with the added benefit of putting data transformation logic into the dynamic table definition just like with traditional views.
Source: Dynamic table created by Yes Energy for change data capture (CDC) on energy price data. You can create streams on the dynamic tables so you can more easily automate when Yes Energy updates and writes energy prices to Snowflake.
For more information on dynamic tables, check out Snowflake’s dynamic table documentation.
Snowpark Container Services
Snowpark Container Services allows you to write applications in any language and run them through Snowflake's infrastructure, and Snowflake manages the whole process. C++ coders can now conveniently write and package their code, with Snowflake managing the end-to-end process from app creation to deployment to runtime. That way, you get the same data governance and management as your other data and objects on Snowflake.
Snowflake is also working on enabling the sharing and running of applications with consumers, who can install and run apps. Snowpark Container Services are now generally available in Amazon Web Services (AWS) regions and will be available in other cloud regions in the future.
Democratizing apps to nontechnical users is another benefit of Snowpark Container Services. This allows you to create interactive visualizations and accessible user interface apps for end users.
Source: Simplified schematic of Snowpark Container Services. Snowflake manages your app end-to-end. Snowflake
How Generative AI Can Enhance Big Data Energy Markets
With the rise of generative artificial intelligence (GenAI), various companies are developing their own large language models (LLMs), with each performing differently. Snowflake provides multiple LLMs that can learn your database architecture and data structure. Our customers have used this for:
- Data anomaly detection
- Trade opportunities
- Internal documentation knowledge.
Retrieval augmented generation (RAG) is an architectural approach that can improve the efficacy of LLM applications by leveraging custom and private data. When we onboard the customer support team at Yes Energy, we provide documentation to help them understand the tables and support customers using our data via Snowflake. Going through all the documentation can be a time-consuming process. The model does this instead by accessing the documentation and tables where the various energy market data types are stored in our database. This tool then provides a simple web app for immediate use without needing extensive developer resources.
In the example below, the model was based on Snowflake's RAG QuickStart and was built and trained on our proprietary DataSignals® Cloud help documentation. This model is currently internal to Yes Energy.
In the future, we can make this RAG model customer-facing using Snowflake Native Apps. It can provide direct answers along with document links to questions like, "Where do I find price data in Yes Energy’s Snowflake database?" for customers.
Source: Yes Energy’s RAG model for DataSignals Cloud documentation, which was trained on table and schema descriptions, object characteristics, and general product information.
The Benefits of Apache Iceberg on Data Science Energy Markets
Another benefit is Apache Iceberg, which allows you to perform data science on one platform rather than outside of it. For those using Snowflake and Databricks, this capability allows you to query external data from within Snowflake without piping data out.
Essentially, it acts like another stage. That’s important because data analysts and data scientists frequently use Databricks, and this feature lets you query large data tables across multiple systems simultaneously without moving data.
Snowflake continues to roll out features for Iceberg tables. Most recently, Snowflake introduced the Polaris Catalog, an open-source catalog for Apache Iceberg cross-platform interoperability, enabling you to more easily query and manage data that lives on separate platforms using only Snowflake.
Elevate Your Energy Market Data Analytics and Data Processing
Snowflake is more than a place to query and pull down Yes Energy’s power market data. You can enrich and augment your energy market data analytics using robust table, GenAI, and app container services features that Snowflake continues to release.
Learn more about Snowflake Marketplace and access Yes Energy’s free sample data. Or reach out to learn how Yes Energy and Snowflake’s partnership can help elevate your energy market data analytics and data processing tasks.
About the author: Sam Lockshin is the product manager of the data products at Yes Energy, which programmatically deliver Yes Energy’s high-quality power market data catalog to customers using a variety of modern API and cloud technologies. He has a passion for extracting value from data using modern data analytics and machine learning techniques. You can catch him at hot yoga, at the club dancing, or playing piano. He also enjoys good a horror flick.
About the author: Cassie Finer leads Yes Energy's internal data and analytics team, leveraging Snowflake to empower data-informed decision-making across the business. With over a decade of experience at Yes Energy starting as an intern, Cassie is passionate about solving complex data challenges. When not immersed in Snowflake, you can find Cassie skiing in the trees or training in Olympic weightlifting in her garage.
