How to Evaluate Power Demand Forecasts

You must navigate billions of data points when making decisions about power market participation. But evaluating the quality of your market data can be complicated, since the process requires different approaches and statistics, which may change depending on your end goal. 

Over the years, power market participants evaluating short-term demand forecasts have asked for help discerning appropriate metrics and their applications. Some of the common questions we’ve received include: 

• Which statistics should I use?
• How should I interpret those statistics?
• How can I feel confident in the value of a demand forecast?

Here’s how to ensure that your electric load demand forecasts are accurate, consistent, and valuable, whether you’re a trader, power producer, load-serving entity, or asset manager. 

Metrics for Accurate Demand Forecasting 

Good metrics and statistics are critical to making informed decisions amid complex, rapidly changing nodal power markets. 

However, statistics can mislead you, and it’s important to use the right tool for the right job. There’s a risk to relying on familiar techniques in statistics. When you’re not using the most effective technique for evaluating forecasts, it can lead to inaccurate conclusions.

One of the reasons we place so much emphasis on understanding what our clients are trying to accomplish is that it allows us to provide guidance that’s aligned with your goals.  

The key question for power traders, utilities, and asset managers to ask regarding their chosen metrics and applications are: Are these forecasts useful, valuable, and actionable?

Mean Absolute Percent Error (MAPE)

Although mean absolute percentage error (MAPE) is commonly used to evaluate energy forecasts, it’s not ideal in many cases because it places too much emphasis on low-demand periods, like overnight or weekends, which skews accuracy assessments.  

How This Plays Out

For example, let’s say that our TESLA model forecast is off by 100 MW when the load is 10,000 MW at the peak hour for the day. This is a 1% absolute percentage error (APE). Now, that same 100 MW error in the middle of the night, when the load is 5,000 MW, has an APE of 2.0%.

When you average the APEs to get the MAPE, that nighttime 100 MW error affects the MAPE by twice as much as the same size error at the daytime peak. This is clearly putting too much emphasis on low-demand hours when prices fall and decisions are less important.

But those two errors contribute the same amount of influence to the root mean square errors (RMSE) figure, which is why RMSE is a better statistic than MAPE.

Additional Considerations

Lower-value MAPEs indicate better performance. Typical bid-close MAPEs for power demand forecasts range from 1 to 10%, depending on consumption patterns. In larger markets, MAPEs typically fall between 2 to 4% over extended periods.

While MAPE tends to place more weight on lower load values — less relevant for traders due to lower prices — you can use it for comparing periods with significantly different consumption levels. For instance, some markets experience higher demand in summer than in winter.

Root mean square errors (see below) put a proportionate weight on errors during periods of high consumption while MAPE expresses errors as a percentage of load, making it easier to compare forecasts across different consumption levels. 

Root Mean Square Error (RMSE)

For power demand forecasting, root mean square error (RMSE) is considered a more robust metric than MAPE. RMSE is a better measure because it treats all errors equally, providing a more balanced evaluation of forecast performance. 

One key advantage of RMSE is that it’s expressed in the same units as the data in question. For example, for power demand, an RMSE of 100 means the typical forecast error is 100 MW. This can especially help traders, power providers, and asset managers, since they can easily translate a 100 MW error into financial terms, understanding the direct impact on dollars earned or saved based on the value of 1 MW of power. Lower RMSE values indicate better forecast accuracy, with zero being ideal.

Contrasting MAPE and RMSE in PJM Forecasts

In the images below, we’ve used Yes Energy TESLA Demand Forecasting to show the differences between MAPE and RMSE in the Pennsylvania-New Jersey-Maryland Interconnection (PJM). 

We've filtered all the dates from June, July, and August 2024 to find five dates where the PJM bid-close forecast MAPE was lower than our Yes Energy® forecast, but our RMSE was lower than PJM’s.

In the image below, we can see that July 17, 2024, was the 14th-highest peak load day in PJM during these three summer months.

Source: Yes Energy’s TESLA Demand Forecasts

As you can see from these metrics, Yes Energy performed much better during the peak hours, which is part of why our RMSE is lower than PJM's, even when PJM’s MAPE is lower.

For users comparing our forecast to the regional transmission organization (RTO), this highlights the importance of considering both RMSE and MAPE, with RMSE being more valuable for those focused on high-demand hours.

Win Rate

A higher percentage win rate indicates how often a forecast has the smallest error compared to others over multiple time intervals. While MAPE and RMSE provide insight into the typical error magnitude across many intervals, they may not fully capture performance consistency. For example, a forecast might have middle-of-the-road error magnitudes but could still perform better than others more frequently in individual intervals.

For traders or utilities using multiple demand forecasts, a forecast with a strong win rate may be more valuable, even if its MAPE or RMSE is not the best, because it occasionally has larger errors. Power market participants can assign more weight to forecasts with better win rates during certain conditions to optimize decision-making by considering which forecast is more likely to perform well in specific situations. 

This brings us to the next metric to consider: bias. 

Bias 

Bias measures whether a forecast tends to consistently go too high or too low, and by how much it missed on average. In an academic setting, we strive for unbiased models. In reality, most models and forecasts have some bias, and if that bias is consistent, traders can use that to their advantage.

For example, traders often compare vendor power demand forecasts with the public load forecast published by the market operator. If a forecast has a consistent bias, power traders can anticipate the direction of the error. This allows them to make more informed decisions, even if the forecast isn’t perfectly balanced between over-and-under-estimation. 

Some power market participants may even prefer a forecast with bias in a particular direction, depending on their strategy.

Forecast Bias in PJM 

In the example below, we’ve used Yes Energy’s TESLA Demand Forecasting to illustrate trends related to forecast bias (i.e., whether the forecast is high or low compared to the actual demand) from four dates in July and August of 2024. For all days, Yes Energy’s Demand Forecasting stats (win rate, MAPE, RMSE) are better than PJM's. 

When we filter by whether PJM missed high or low, however, there's a clear difference. TESLA performs well in both cases, and we perform much better when PJM projects too low. 

When we restrict even further to cases when we both forecast too low or too high, the differences are even bigger.

In the screenshot below, you can see four dates in July and August where PJM's forecasts were low at several daily peaks, and we outperformed them.

Source: Yes Energy’s TESLA Demand Forecasting 

The screenshot below highlights performance differences based on the directional error in PJM's public forecast.

Source: Yes Energy’s TESLA Demand Forecasting

Daily Peak Timing

This metric is particularly relevant for battery system operators working to predict peak demand. Forecast performance is typically evaluated using three key measures:

• Timing win rate: The forecast that most accurately predicts the time of peak demand wins.
• RMSE of time (in hours): Lower values are better, indicating smaller errors in predicting the peak hour.
• Bias: This is critical for battery deployment. If the peak is forecast too late, the battery may miss the highest demand hour. Deploying too early or too late risks using battery capacity after the true peak, reducing the probability of fully capitalizing reducing consumption over the peak demand period.

Accurate timing is essential to ensure batteries deploy at the optimal time for reducing consumption during the highest price hours and avoiding penalties for high consumption on market coincident peak days.

Evaluating Daily Peak Timing in PJM 

In the example below, we calculated our statistics to evaluate peak timing for all daily peaks, including the top five, ten, and 20 peaks. Since most battery operators decide when to start discharging close to the peak hour, which typically occurs after 1 p.m. in the summer, we used the intraday forecasts published by noon. 

As you can see in the image below, we predicted the precise peak hour timing for all five of the highest daily peak dates. For four of those dates, PJM missed by one hour.

The top five peaks are especially important for battery operators due to PJM's coincident peak penalty program, which is based on the highest five peaks in both summer (June to September) and winter (December to February). Accurately forecasting the timing of the daily peak hour for all five summer coincident peaks is valuable for anyone looking to manage their peak load consumption.

Source: Yes Energy’s TESLA Demand Forecasting in PJM

Additional Demand Forecasting Considerations 

Accurately assessing your goals will help ensure you’re making the most informed decisions. Here are some additional questions to consider when you’re evaluating demand forecasts: 

1. Which groups of hours are you most interested in? On-peak, off-peak, all hours, etc.
2. Which time of year are you most interested in? Performance can differ by season, weather pattern, etc.

Gain a Competitive Edge with Your Demand Forecasting 

Accurate power demand forecasting is an important part of maintaining grid stability and ensuring cost-effective energy distribution. 

Yes Energy's TESLA Demand Forecasting solutions allow you to manage risk and gain a competitive edge in the power market industry with access to highly accurate and reliable wholesale, grid, and utility-level forecasts. Our proprietary forecasting engines analyze historical demand and weather patterns while incorporating the latest real-time data to adjust for changing weather conditions, extreme events, and holidays that may affect energy demand. 

Our team of expert analysts and engineers continuously review and fine-tune our models to ensure that we’re providing you with the most accurate and reliable forecasts in the power market industry.

Request a demo to learn how Yes Energy TESLA’s forecasting solutions can help you Win the Day Ahead™. 

About the Author: Ben Perry is the senior product manager of forecasting at Yes Energy. Ben brings 10 years of experience as a power demand forecast analyst at TESLA to the Yes Energy product team. He's now focused on applying that experience to steering the Yes Energy forecasting product roadmap to best serve the industry through the energy transition.