Share this
by Eli DeQuiroz
If independent system operators (ISOs) make changes to their systems, it affects the entire power market ecosystem. When the New England Independent System Operator (ISO-NE) moved its entire web services API to cloud services, Yes Energy® was ready. Here’s how a proactive approach protects your data.
In August 2023, ISO-NE informed the market that it would change its website structure and move their web services API to a cloud server. Our market monitoring team is dedicated to sifting through stakeholder communication to identify changes that will affect you. This allows critical lead time and direction for our teams to navigate the transition proactively.
Although ISO-NE didn’t expect the change to impact collections, our data operations team knew from experience that the transition could lead to system issues. If even a fraction of an ISO’s data breaks, it can force companies to scramble to fix it. Smaller data breaks can take hours to resolve, while larger breaks can take days.
How ISO-NE’s Data Transition Played Out
When we learned about the ISO-NE move to cloud services, our goal was to ensure a seamless transition for our clients. Then, we took these steps.
The Challenge
When ISOs implement a change, they first implement the changes in a test environment.
We embarked on ISO-NE’s transition by accessing their new environment and ensuring we could collect necessary data. We also verified through extensive testing that the changes didn’t lead to issues in our system.
Our next step was to set up a monitoring system gauging expected behavior, including creating alerts to notify us of ISO-NE’s data behavior changes, such as:
- Less rows collected than expected.
- Less data files collected than expected.
- Bad or empty files coming in unexpectedly.
- Files collected at different time intervals than expected.
Our system ran automatically, constantly monitoring our collections ISO-NE data to make sure they remained healthy.
Additional communication with ISO-NE informed us that the changes would happen on a Saturday. Although ISOs typically give longer timeframes for making these changes, our teams were given a rough estimate of a two-hour window.
The Solution
Our data operations team remained on standby during the transition, and the monitoring system also continued to run through the weekend. Both were ready and available if our automated monitors detected any anomalies or failures.
ISO-NE’s transition to cloud services went smoothly. Historically, that hasn’t always been the case during similar transitions. This is why we put additional plans in place to fix issues that might arise.
What ISO-NE’s Transition Might Have Looked Like
When ISOs migrate information, several issues can occur. Here are some examples of what could go wrong:
- IP Address Changes: If an API moves to a cloud platform, its IP address could change. Processes relying on a specific IP address could fail to connect to the new data source.
- New Firewall Issues: ISO-NE’s move to the new cloud environment could have introduced new firewalls and other security measures, inadvertently blocking incoming data requests and denying us access to the data.
- Service Outages: ISO-NE’s move to the new environment could have caused a data outage during migration. Therefore, our code had to be ready to re-collect data during that interval.
- Geographical Latency: There are several different data centers to store data during a migration like this. Depending on which region ISO-NE stored its data, it could have introduced latency in data collection. To ensure low-latency, real-time data, we needed monitors to verify existing data collections were as quick and consistent as before the migration.
How Similar Transitions Have Affected Other ISOs
After 15 years in the energy industry, we know what could go wrong (and how to fix it). When companies don’t have the bandwidth to monitor ISO meetings, they miss critical information. And without the resources to research, clarify, and troubleshoot upcoming changes, it can take days to weeks to update and implement changes.
While some changes are minor, others are extensive. For example, when (PJM) moved from three financial transmission right (FTR) peak types to four, the change deeply impacted nodal power traders and front, middle, and back office systems and processes. It was critical that the whole infrastructure could handle these changes, and not every company was ready.
At Yes Energy, our market monitoring team provided critical lead time and direction for our teams, then helped ensure clear communication with our clients. Although the transition involved the bulk of a year, our internal teams ensured a flawless segue for our clients.
Across the ISOs, we understand data collection policies and are a “good steward” of adhering to those rules. This leads to higher data ingestion integrity, which also helps promote a healthy market for others. Additionally, ISOs tend to know our IP address and will often reach out directly if there’s an issue.
Without this type of infrastructure in your corner, it’s common for something to break. And when that happens, you lose time and money. You also erode your clients’ trust.
Avoid scrambling to keep pace with ISO changes, and see how Yes Energy provides you with better data so that you can Win the Day Ahead™.
About the author: Eli DeQuiroz has more than 10 years of experience building and maintaining data-intensive energy market software solutions. The majority of that experience focused on data collection and data engineering of real-time energy market data. At Yes Energy, he oversees the technical operations of all Yes Energy solutions, ensuring the reliability of the applications and data.
