Share this
How to Evaluate Power Demand Forecasts
by Ben Perry
You must navigate billions of data points when making decisions about power market participation. But evaluating the quality of your market data can be complicated, since the process requires different approaches and statistics, which may change depending on your end goal.
Over the years, power market participants evaluating short-term demand forecasts have asked for help discerning appropriate metrics and their applications. Some of the common questions we’ve received include:
- Which statistics should I use?
- How should I interpret those statistics?
- How can I feel confident in the value of a demand forecast?
Here’s how to ensure that your electric load demand forecasts are accurate, consistent, and valuable, whether you’re a trader, power producer, load-serving entity, or asset manager.
Metrics for Accurate Demand Forecasting
Good metrics and statistics are critical to making informed decisions amid complex, rapidly changing nodal power markets.
However, statistics can mislead you, and it’s important to use the right tool for the right job. There’s a risk to relying on familiar techniques in statistics. When you’re not using the most effective technique for evaluating forecasts, it can lead to inaccurate conclusions.
One of the reasons we place so much emphasis on understanding what our clients are trying to accomplish is that it allows us to provide guidance that’s aligned with your goals.
The key question for power traders, utilities, and asset managers to ask regarding their chosen metrics and applications are: Are these forecasts useful, valuable, and actionable?
Mean Absolute Percent Error (MAPE)
Although mean absolute percentage error (MAPE) is commonly used to evaluate energy forecasts, it’s not ideal in many cases because it places too much emphasis on low-demand periods, like overnight or weekends, which skews accuracy assessments.
How This Plays Out
For example, let’s say that our TESLA model forecast is off by 100 MW when the load is 10,000 MW at the peak hour for the day. This is a 1% absolute percentage error (APE). Now, that same 100 MW error in the middle of the night, when the load is 5,000 MW, has an APE of 2.0%.
When you average the APEs to get the MAPE, that nighttime 100 MW error affects the MAPE by twice as much as the same size error at the daytime peak. This is clearly putting too much emphasis on low-demand hours when prices fall and decisions are less important.
But those two errors contribute the same amount of influence to the root mean square errors (RMSE) figure, which is why RMSE is a better statistic than MAPE.
Additional Considerations
Lower-value MAPEs indicate better performance. Typical bid-close MAPEs for power demand forecasts range from 1 to 10%, depending on consumption patterns. In larger markets, MAPEs typically fall between 2 to 4% over extended periods.
While MAPE tends to place more weight on lower load values — less relevant for traders due to lower prices — you can use it for comparing periods with significantly different consumption levels. For instance, some markets experience higher demand in summer than in winter.
Root mean square errors (see below) put a proportionate weight on errors during periods of high consumption while MAPE expresses errors as a percentage of load, making it easier to compare forecasts across different consumption levels.
Root Mean Square Error (RMSE)
For power demand forecasting, root mean square error (RMSE) is considered a more robust metric than MAPE. RMSE is a better measure because it treats all errors equally, providing a more balanced evaluation of forecast performance.
One key advantage of RMSE is that it’s expressed in the same units as the data in question. For example, for power demand, an RMSE of 100 means the typical forecast error is 100 MW. This can especially help traders, power providers, and asset managers, since they can easily translate a 100 MW error into financial terms, understanding the direct impact on dollars earned or saved based on the value of 1 MW of power. Lower RMSE values indicate better forecast accuracy, with zero being ideal.
Contrasting MAPE and RMSE in PJM Forecasts
In the images below, we’ve used Yes Energy TESLA Demand Forecasting to show the differences between MAPE and RMSE in the Pennsylvania-New Jersey-Maryland Interconnection (PJM).
We've filtered all the dates from June, July, and August 2024 to find five dates where the PJM bid-close forecast MAPE was lower than our Yes Energy® forecast, but our RMSE was lower than PJM’s.
Date |
TESLA MAPE |
PJM MAPE |
TESLA RMSE |
PJM RMSE |
6/14/2024 |
2.024 |
1.982 |
2206.286 |
2364.041 |
7/17/2024 |
1.407 |
1.238 |
1876.843 |
1905.841 |
8/10/2024 |
2.382 |
2.341 |
2442.487 |
2525.504 |
8/13/2024 |
1.496 |
1.477 |
1656.023 |
1996.7 |
8/23/2024 |
0.883 |
0.881 |
934.223 |
1427.125 |
In the image below, we can see that July 17, 2024, was the 14th-highest peak load day in PJM during these three summer months.
Source: Yes Energy’s TESLA Demand Forecasts
As you can see from these metrics, Yes Energy performed much better during the peak hours, which is part of why our RMSE is lower than PJM's, even when PJM’s MAPE is lower.
For users comparing our forecast to the regional transmission organization (RTO), this highlights the importance of considering both RMSE and MAPE, with RMSE being more valuable for those focused on high-demand hours.
Win Rate
A higher percentage win rate indicates how often a forecast has the smallest error compared to others over multiple time intervals. While MAPE and RMSE provide insight into the typical error magnitude across many intervals, they may not fully capture performance consistency. For example, a forecast might have middle-of-the-road error magnitudes but could still perform better than others more frequently in individual intervals.
For traders or utilities using multiple demand forecasts, a forecast with a strong win rate may be more valuable, even if its MAPE or RMSE is not the best, because it occasionally has larger errors. Power market participants can assign more weight to forecasts with better win rates during certain conditions to optimize decision-making by considering which forecast is more likely to perform well in specific situations.
This brings us to the next metric to consider: bias.
Bias
Bias measures whether a forecast tends to consistently go too high or too low, and by how much it missed on average. In an academic setting, we strive for unbiased models. In reality, most models and forecasts have some bias, and if that bias is consistent, traders can use that to their advantage.
For example, traders often compare vendor power demand forecasts with the public load forecast published by the market operator. If a forecast has a consistent bias, power traders can anticipate the direction of the error. This allows them to make more informed decisions, even if the forecast isn’t perfectly balanced between over-and-under-estimation.
Some power market participants may even prefer a forecast with bias in a particular direction, depending on their strategy.
Forecast Bias in PJM
In the example below, we’ve used Yes Energy’s TESLA Demand Forecasting to illustrate trends related to forecast bias (i.e., whether the forecast is high or low compared to the actual demand) from four dates in July and August of 2024. For all days, Yes Energy’s Demand Forecasting stats (win rate, MAPE, RMSE) are better than PJM's.
When we filter by whether PJM missed high or low, however, there's a clear difference. TESLA performs well in both cases, and we perform much better when PJM projects too low.
When we restrict even further to cases when we both forecast too low or too high, the differences are even bigger.
In the screenshot below, you can see four dates in July and August where PJM's forecasts were low at several daily peaks, and we outperformed them.
Source: Yes Energy’s TESLA Demand Forecasting
The screenshot below highlights performance differences based on the directional error in PJM's public forecast.
Dates |
TESLA Daily Peak Win Rate |
TESLA Daily Peak MAPE |
PJM Daily Peak MAPE |
TESLA Daily Peak RMSE |
PJM Daily Peak RMSE |
PJM Peak Forecast Missed Low |
64.29% |
1.17% |
2.00% |
2609.57 |
3221.61 |
PJM Peak Forecast Missed High |
48.00% |
1.78% |
1.85% |
2609.57 |
3040.50 |
PJM and TESLA Peak Forecast Missed Low |
70.00% |
1.88% |
2.35% |
2977.68 |
3625.64 |
PJM and TESLA Peak Forecast Missed High |
37.50% |
2.11% |
2.05% |
3261.27 |
3301.65 |
All Days |
55.43% |
1.70% |
1.92% |
2791.83 |
3124.48 |
Source: Yes Energy’s TESLA Demand Forecasting
Daily Peak Timing
This metric is particularly relevant for battery system operators working to predict peak demand. Forecast performance is typically evaluated using three key measures:
- Timing win rate: The forecast that most accurately predicts the time of peak demand wins.
- RMSE of time (in hours): Lower values are better, indicating smaller errors in predicting the peak hour.
- Bias: This is critical for battery deployment. If the peak is forecast too late, the battery may miss the highest demand hour. Deploying too early or too late risks using battery capacity after the true peak, reducing the probability of fully capitalizing reducing consumption over the peak demand period.
Accurate timing is essential to ensure batteries deploy at the optimal time for reducing consumption during the highest price hours and avoiding penalties for high consumption on market coincident peak days.
Evaluating Daily Peak Timing in PJM
In the example below, we calculated our statistics to evaluate peak timing for all daily peaks, including the top five, ten, and 20 peaks. Since most battery operators decide when to start discharging close to the peak hour, which typically occurs after 1 p.m. in the summer, we used the intraday forecasts published by noon.
As you can see in the image below, we predicted the precise peak hour timing for all five of the highest daily peak dates. For four of those dates, PJM missed by one hour.
STATS |
RTO |
TESLA |
Wins |
0 |
4 |
Win Rate |
0.00% |
80.00% |
Forecast Peak Early |
4 |
0 |
Forecast Peak Late |
0 |
0 |
Forecast Peak Exactly |
1 |
5 |
The top five peaks are especially important for battery operators due to PJM's coincident peak penalty program, which is based on the highest five peaks in both summer (June to September) and winter (December to February). Accurately forecasting the timing of the daily peak hour for all five summer coincident peaks is valuable for anyone looking to manage their peak load consumption.
Source: Yes Energy’s TESLA Demand Forecasting in PJM
Additional Demand Forecasting Considerations
Accurately assessing your goals will help ensure you’re making the most informed decisions. Here are some additional questions to consider when you’re evaluating demand forecasts:
- Which groups of hours are you most interested in? On-peak, off-peak, all hours, etc.
- Which time of year are you most interested in? Performance can differ by season, weather pattern, etc.
Gain a Competitive Edge with Your Demand Forecasting
Accurate power demand forecasting is an important part of maintaining grid stability and ensuring cost-effective energy distribution.
Yes Energy's TESLA Demand Forecasting solutions allow you to manage risk and gain a competitive edge in the power market industry with access to highly accurate and reliable wholesale, grid, and utility-level forecasts. Our proprietary forecasting engines analyze historical demand and weather patterns while incorporating the latest real-time data to adjust for changing weather conditions, extreme events, and holidays that may affect energy demand.
Our team of expert analysts and engineers continuously review and fine-tune our models to ensure that we’re providing you with the most accurate and reliable forecasts in the power market industry.
Request a demo to learn how Yes Energy TESLA’s forecasting solutions can help you Win the Day Ahead™.
About the Author: Ben Perry is the senior product manager of forecasting at Yes Energy. Ben brings 10 years of experience as a power demand forecast analyst at TESLA to the Yes Energy product team. He's now focused on applying that experience to steering the Yes Energy forecasting product roadmap to best serve the industry through the energy transition.
Share this
- Industry News & Trends (96)
- Power Traders (65)
- Data, Digital Transformation & Data Journey (44)
- Asset Managers (38)
- Market Events (30)
- Asset Developers (26)
- Market Driver Alerts - Live Power (25)
- Utilities (25)
- ERCOT (23)
- ISO Changes & Expansion (22)
- Renewable Energy (20)
- PowerSignals (19)
- Infrastructure Insights Dataset (18)
- Energy Storage / Battery Technology (17)
- Live Power (17)
- DataSignals (16)
- Risk Management (15)
- Data Scientists (13)
- TESLA Forecasting (13)
- CAISO (12)
- PJM (9)
- Power Markets 101 (9)
- QuickSignals (9)
- MISO (8)
- SPP (8)
- Position Management (7)
- EnCompass (6)
- Financial Transmission Rights (6)
- Snowflake (6)
- Powered by Yes Energy (5)
- Submission Services (5)
- Asset Developers/Managers (4)
- Data Centers (4)
- Solutions Developers (4)
- Commercial Vendors (3)
- FTR Positions Dataset (3)
- Geo Data (3)
- Battery Operators (2)
- PeopleOps (2)
- AI and Machine Learning (1)
- Crypto Mining (1)
- FERC (1)
- ISO-NE (1)
- Independent Power Producers (1)
- Japanese Power Markets (1)
- Natural Gas (1)
- hydro storage (1)
- September 2024 (5)
- August 2024 (7)
- July 2024 (9)
- June 2024 (5)
- May 2024 (7)
- April 2024 (8)
- March 2024 (6)
- February 2024 (9)
- January 2024 (7)
- December 2023 (4)
- November 2023 (5)
- October 2023 (6)
- September 2023 (2)
- August 2023 (6)
- July 2023 (3)
- May 2023 (4)
- April 2023 (2)
- March 2023 (2)
- February 2023 (2)
- January 2023 (5)
- December 2022 (2)
- November 2022 (1)
- October 2022 (3)
- September 2022 (5)
- August 2022 (5)
- July 2022 (3)
- June 2022 (3)
- May 2022 (1)
- April 2022 (3)
- March 2022 (3)
- February 2022 (6)
- January 2022 (3)
- December 2021 (2)
- November 2021 (4)
- October 2021 (4)
- September 2021 (3)
- August 2021 (2)
- July 2021 (4)
- June 2021 (5)
- May 2021 (3)
- April 2021 (3)
- March 2021 (4)
- February 2021 (3)
- December 2020 (3)
- November 2020 (4)
- October 2020 (2)
- September 2020 (5)
- August 2020 (2)
- July 2020 (2)
- June 2020 (1)
- May 2020 (9)
- November 2019 (1)
- August 2019 (2)
- June 2019 (2)
- May 2019 (2)
- January 2019 (1)