Demand Side Analytics

demand-side-analytics-llc_medium

Meter-Based Methods from Coast to Coast

Demand Side Analytics (DSA) recently conducted two similar studies on the accuracy of using smart meter data for evaluation and settlement of energy efficiency, also known as meter-based methods. Different jurisdictions refer to these meter-based methods with distinctive terminology. Across our two studies, California (for Pacific Gas & Electric) refers to these methods as normalized metered energy consumption (NMEC) while Vermont (for the Vermont Department of Public Service) refers to them as Advanced Measurement & Verification (M&V).

What are Meter-Based Methods?

The primary challenge of estimating energy savings is the need to accurately detect changes in energy consumption due to the energy efficiency intervention, while systematically eliminating plausible alternative explanations for those changes. Did the introduction of energy efficiency measures cause a change in energy use? Or can the differences be explained by other factors (such as the effects of the COVID-19 pandemic)? To evaluate energy savings, it is necessary to estimate what energy consumption would have been in the absence of program intervention—the counterfactual or baseline.

Meter-based methods rely on whole-building, site-specific electric and/or gas consumption data, either at the hourly or daily level, to construct the baseline. This data is then used to estimate energy savings associated with the installation of individual or multiple energy efficiency measures (EEMs) at the site.

Why rely on Meter-Based Methods?

Many methods exist to estimate savings associated with EEMs, all with varying degrees of modeling complexity, data requirements, accuracy, and precision. The benefits of using meter-based methods include:

  • Eliminating the need for sampling because data is available for nearly all participants.
  • Reducing the burden on participants because technicians don’t need to visit the home or business to install metering equipment.
  • Producing faster feedback on energy-saving performance.
  • Enabling program administrators to look beyond the average customer and explore how savings vary across segments of interest.
  • Opening new opportunities for program design and delivery (i.e., pay-for-performance programs).
  • Producing granular savings estimates that are useful for a wide range of planning and valuation functions.

California

Pacific Gas and Electric Company (PG&E) currently uses the CalTRACK Version 2.0 method (CalTRACK) to estimate avoided energy use for its energy efficiency programs based on the Population-Level NMEC methodology. A notable feature of the population NMEC method has been the lack of comparison groups, which are used to adjust the energy savings baseline and normalize the savings estimate for factors beyond weather. The pre-post method without a comparison group relies almost exclusively on weather normalization and effectively assumes that the only difference between the pre- and post-intervention periods is weather and the installation of EEMs. The COVID-19 pandemic laid bare the limitations of the adopted method. The pandemic led to changes in our commutes, business operations, and home use patterns. Not surprisingly, it has also changed how, when, and how much electricity and gas we use. Moreover, the impact on energy use differs for residential customers and various types of businesses.

Given the changes in energy consumption that have occurred over the course of the COVID-19 pandemic, the need for alternative approaches to CalTRACK and similar, simple pre-post regression methods for estimating EE impacts is paramount. While adding comparison groups typically improves the accuracy of these energy saving estimates, there are three main logistical challenges:

  • Privacy of non-participant customer data. Current California laws and regulation exist to protect the privacy of advanced metering infrastructure (AMI) or smart meter data for individual customers.
  • Transparency Challenges. Many evaluation methods that rely on a comparison group require extensive calculation in order to construct the group. This complexity can hinder independent review and/or replication of the findings.
  • Complexity and frequency. PG&E and third-party EE program implementers target a wide range of customer segments and geographic areas, each of which require regular and specifically targeted non-participant data for evaluation. This consideration adds complexity to existing program administration processes.

To determine if there are viable alternative models that can accommodate the effects of the COVID-19 pandemic or other wide-scale non-routine events, DSA conducted an accuracy assessment of the existing Population NMEC methods as well as a variety of other methods with and without comparison groups.

What did we do?

Accurate and unbiased estimates of energy efficiency impacts are critical for utility program staff, third-party program implementers, and regulators. In evaluating the accuracy of the existing Population NMEC methods used in the PG&E territory, we tested a variety of other methods, with and without comparison groups, to simulate a competition and identify the methods that are unbiased and accurate (Figure 1).

The accuracy of these methods are assessed by applying placebo treatment on customers that did not participate in EE programs during the period analyzed. The impact of a program (or in this case, a pseudo-program) is calculated by estimating a counterfactual and comparing it to the observed consumption during the post-treatment period. Because no EEMs were installed in this simulation, any deviation between the counterfactual and actual loads is due to error. The process is repeated hundreds of times – a procedure known as bootstrapping – to construct the distribution of errors.

Figure 1: General Approach for Accuracy Assessment

What did we find?

  1. Population NMEC methods without comparison groups cannot account for the effects of the COVID-19 pandemic.
  2. The existing population NMEC methods without comparison groups show upward bias even prior to the effects of the pandemic.
  3. Comparison groups improve accuracy of the CalTRACK method.
  4. When constructing a matched control group, the choice of segmentation and matching characteristics matter more than the method of matching customers.
  5. Synthetic controls may perform well but are highly sensitive to the choice of segmentation used.
  6. Using aggregated granular profiles instead of individual matched controls in Difference-in-Differences methods yields comparable results to using individual customer matched controls.
  7. Accuracy and precision are dependent upon the number of sites aggregated together (Figure 2).
  8. No method is completely free of error.

Figure 2: Distribution of Error across Comparison Groups

Given these findings, rather than try to produce a single prescriptive method for NMEC analyses of energy efficiency programs, we instead recommend a framework by which proposed NMEC methods can be tested, certified, and used to estimate savings:

Vermont

The primary objective of the Hourly Impact of Energy Efficiency Evaluation Pilot was to better understand the time-value of energy efficiency measure savings and the implications for program design, delivery, and evaluation. Because energy efficiency in the Northeast qualifies for capacity value, accurate estimates of the contribution of energy efficiency to peak hours is critical. Using high-frequency 15-minute consumption data from Green Mountain Power’s AMI and program tracking data from Efficiency Vermont, the study team modeled energy consumption of participating homes and businesses separately in the pre-installation and the post-installation periods. These two periods were compared to understand how consumption changed following installation of an energy efficiency or beneficial electrification measure. A secondary objective of the study was to compare Advanced M&V methods, or regression-based modeling of utility meter data, with the approaches traditionally used in Vermont. This comparison helped to determine where Advanced M&V could offer cost savings, improve the accuracy and granularity of savings estimates, and identify lessons for program operations.

What did we do?

To generate savings for the 21 prescriptive measures and the 124 custom projects in Vermont, we implemented Advanced M&V procedures that build upon the International Performance Measurement and Verification Protocol (IPMVP) Option C Whole Facility approach to energy savings estimation. We do this through a regression model that follows Lawrence Berkeley National Laboratory’s (LBNL) Time-of-Week Temperature (TOWT) Model, where the dependent variable is hourly electric consumption from the meter and the independent variables contain information about the weather, day of week, and time of day.

This methodology estimates efficiency impacts in each hour of the year. Granular results provide insight into the distribution of energy savings across a year. For example, Figure 3 shows a heat map of the average energy savings from installing a variable speed heat pump. This measure’s model estimates a large load increase during the winter months (blue regions). Negative savings are a good thing in this case because it means Vermont homes are using the heat pump for heating and displacing delivered fuel consumption. There is also a pocket of denser load increase in the summer months during the middle of the day, presumably due to homes that may not have had air conditioning previously using the heat pump as an air conditioner.

Figure 3: Variable Speed Heat Pump Heat Map

What did we find?

  1. Modelling success for prescriptive measures is a function of effect size and number of participants.
  2. There are challenges when using Advanced M&V for “market opportunity” measures, where the baseline is a hypothetical new piece of equipment with code-minimum efficiency. This assumption creates issues because the pre-installation meter data reflects the replaced equipment at the end of its useful life.
  3. For custom projects, Advanced M&V methods work best for sites with predictable load patterns and large savings as a percent of total consumption (Figure 4).
  4. With the level of noise present, we caution against using site-specific results to determine incentive levels in Vermont and suggest Advanced M&V is more useful as a program evaluation tool.
  5. Advanced M&V is a powerful tool, but it is not the right tool for every job.

Figure 4: Example of a Well-Behaved Custom Project

Given these findings, to have a chance at accurately and precisely estimating savings from efficiency measures, the guidance below must be taken into consideration:

 

Does Residential Battery Storage Help the Grid?

Does Residential Battery Storage Help the Grid?

Posted on 

Do residential behind-the-meter batteries help the grid? The answer is: unfortunately, not as much as one would hope. The below plot shows my solar unit, two Tesla batteries, my whole home use, and my grid use on four select days.
 
 

The first plot is a very good outcome. It shows no grid usage. The home (blue) is exclusively being powered by battery storage (green) and solar (orange).  This pattern happens fairly often in the spring when household energy consumption is low and solar production is high. It also means no grid exports, even if the grid needs additional resources. The second case – contributing to the ramp – is disturbingly common. The battery starts charging as soon as the sun is up and is fully charged around mid-day, at which point, all of the solar comes online all at once. From a grid standpoint, it’s contributing to the ramps and is not helping absorb surplus solar. The last two scenarios are less common. The third plot shows some use during off-peak hours (I was charging a electric car), my intentional draw from the grid immediately before the 4-9 pm period, and use of the battery throughout that peak window (with a small amount of exports). It was also a very hot day. You see my AC unit cycling on and off. The last plot shows the full capability of the battery, close to 7 kW, which is rarely seen. The battery went into storm mode and drew power from the grid rather than charge only using the rooftop solar. When operated in default mode, the battery will almost never charge or discharge at its full capability. It means that behind-the-meter batteries are an under-utilized, untapped resource during periods from the grid needs resources the most and during period with excess generation on the grid.

If left to operate on their own, the batteries typically charge as soon as the sun comes up (the wrong time from a grid perspective), often don’t absorb surplus generation, and rarely, if ever, export to the grid when resources are needed most. By design, they operate with the customer in mind, which is an excellent objective. However, it is possible to lower customer bills, provide backup power, and also improve operations for the grid. As saying goes, “we can walk and chew gum at the same time.”

Why does this matter? Behind the meter battery storage is a growing, untapped resource, and the need for flexible, predictable resources is growing.  The below plot shows the growth in residential behind-the-meter battery storage in California. There are currently about 400 MW, but the magnitude of growing quickly. Roughly 8-10% of new solar installations are also install battery storage at the same time. And the share of solar sites electing battery storage is growing. What can be done to tap into this under-utilized resource? Clearly, it is not enough to have the batteries installed. It is necessary to operate them at the right times and to provide customers incentives to do so.

DSA is involved in several efforts to better use battery storage, including:
 
  • A virtual power plant study with over 1,000 residential batteries. The batteries are providing grid response based on day-ahead market prices  (after a strike price is hit) and in response to system operator alerts, warnings, and emergencies.
  • A battery storage pilot. Perhaps the most exciting part of the pilot is that we are using a randomized control trial to explicitly test how different incentive levels and incentive structures affect customer willingness to allow utilities to operate the battery for grid needs. In addition, we are testing daily operations with day-ahead market prices and time-of-use rates, and testing how to modify dispatch algorithms so behind the meter batteries can deliver a predictable, incremental resource. The pilot includes two tracks: one for customers with existing battery storage and for customers who are in the process of installing solar and/or battery storage (another sites).  DSA is in charge of all aspects of the turn-key pilot including design, recruitment, event operations, communication with the batteries (or more accurately, the battery API), setting up data tracking and collection databases, and evaluation. (Click the link for a presentation of battery pilot design: Battery Storage Pilot Design )
  • Programming a utility-scale battery to maximize load relief and demand charges for coops
  • Identifying high-value locations for distribution connected battery storage.
  • Assessing economic feasibility of utility-owned battery storage operated in response to market prices and T&D needs.

Is Electric Demand Rebounding? An Interactive Dashboard

Posted on 

Summer Demand Response Changes at PJM

Posted on 

Washington State Distributed Energy Resource Planning

Posted on 

There has been a rapidly growing level of interest in distribution planning and how to integrated distributed energy resources (DERs).  The growth of DERs is fundamentally changing the nature of transmission and distribution system forecasting, planning, and operations.   However, the current state of transmission and distribution planning and of DER integration into planning vary widely from utility to utility. For this project, our team conducted an inventory of current utility distribution planning practices and capabilities in Washington. The results were presented at Workshop on November 20th to a broad range of stakeholders.

WA DER Planning Workshop – Current utility capabilities

Price Elasticity of Demand Analysis for LED Lighting

Posted on 

Demand Side Analytics recently designed and analyzed an LED pricing trial for Efficiency Maine Trust. The study involved the two largest retailers in the state and provided some valuable program design information on managing free-ridership, setting incentive levels, and capturing off shelf product placement. Full report can be found at the link below:

LED Lighting Pricing Trial Results

Price Elasticity of Demand Analysis for LED Lighting

Posted on 

New Evidence on the Price Elasticity of Demand for Electric Vehicle Charging


If the U.S. is to address climate change, a key mechanism will be a transition to a decarbonized transportation sector. California has been leading the charge through policies aimed to increase electric vehicle (EV) adoption. A challenge of widespread adoption is integrating increased electricity demand from electric vehicles with the grid. In collaborative work with San Diego Gas & Electric (SDG&E), we estimated customer responsiveness to a dynamic, real-time electricity rate at commercial level 2 EV charging stations in multi-unit dwellings and workplaces. The rate is designed to mitigate the impact of EV demand on both distribution grid and system costs.


The key output of our analysis is estimates of the price elasticity of demand for electricity at charging stations that were subject to SDG&E’s Vehicle Grid Integration (VGI) rate. To our knowledge, these findings represent some of the first evidence of the responsiveness of EV charging demand to real-time prices. They also represent some of the only publicly available evidence on the price responsiveness of EV charging demand at commercial level 2 charging stations in multi-unit dwellings and workplaces. (Many of these stations are installed in bi-lateral agreements between site hosts and vendors, which retain the data.) The VGI rate has several features that enable us to recover credible causal estimates of the price elasticity of demand. Notably, the rate is made up of several components that vary at the hour- and distribution-circuit-level:


A nominal base rate;

A commodity component that is the hourly CAISO day-ahead wholesale market price;

A system event adder based on CAISO demand; and

A local event adder based on circuit-level demand.


Because not all sites were subject to local events, we can not only compare across time within a site (within variation) but also compare sites that were and were not experiencing events in the same hour (between variation). A second important feature is that there are also some site hosts that elect to pay the cost of charging on behalf of drivers, which we refer to as rate-to-host sites. Drivers at these sites have no incentive to curb consumption in response to price. We can therefore use these sites as a placebo test of our model. Our model attempts to account for the fact that events and high prices are not randomly assigned and therefore could be related to charging behavior in unobservable ways that result in biased estimates. A precisely estimated finding of minimal price responsiveness at rate-to-host sites where charging is free for drivers would bolster our confidence that our model has accounted for the potential endogeneity of price and events. Spoiler alert: we do estimate a precise zero effect at these sites!


The table below presents estimated price elasticities for each site type. The table includes coefficient estimates and standard errors from three separate Poisson regressions: rate-to-driver, workplace estimates are presented in column (1); rate-to-driver, multi-unit dwelling estimates are presented in column (2); and rate-to-host estimates are presented in column (3). These estimates pool data from program years 2022 and 2023. Our main findings are as follows:


At workplace sites, we estimate an elasticity of -0.337. This indicates that, on average, drivers decrease their charging by 3.4% for each 10% increase in prices. 

At multi-unit dwelling sites, the price responsiveness is similar, with an estimated elasticity of -0.37. These estimates are both statistically significant at the 1% level. 

At rate-to-host sites where charging is free at the port for drivers, we find there is insufficient evidence to conclude that drivers at these sites were price-responsive; we can also rule out price elasticities below -0.051 based on the coefficient estimate and standard error.

Estimated Elasticities (%) for PY 2022 and PY 2023 Combined

  (1) (2) (3)

  Rate-to-Driver Rate-to-Driver  

  Workplace MUD Rate-to-Host

   

ln(Price) -0.337*** -0.370*** 0.0265

  (0.0623) (0.0474) (0.0385)

   

Observations 21,489,394 12,577,911 12,325,088

Ports 1317 729 709

Sites 92 70 51

Pseudo-R-Squared 0.3343 0.1683 0.3832

Note: *** p<0.01, ** p<0.05, * p<0.1. This table reports coefficient estimates and standard errors from three separate Poisson regressions. All regressions are estimated using port-by-hour observations for October 1 2021 through September 30 2023. Standard errors are two-way clustered at the site and hour-of-sample level. Estimated effects are at the port-level and include fixed effects for port, date, day-of-week, weekend-by-hour-of-day, and temperature bin. Rate-to-host results in column (3) are reported for MUD and workplace combined because there is a single MUD rate-to-host site. Fixed effects in column (3) are interacted with MUD/workplace status. All specifications include controls for event anticipation and rebound hours; we do not report coefficients on controls.

These results are exciting for several reasons. Firstly, to our knowledge, these represent the first publicly available estimates of price responsiveness of EV charging demand at commercial level 2 charging stations. If more residents of multi-unit dwellings are to own EVs, as is the hope in a future of widespread adoption, many more such charging stations must be installed. Understanding the price elasticity of demand in this setting is critical to evaluating the effectiveness of rates and policies designed to shift electricity demand at these locations.

Secondly, the availability of rate-to-host sites to serve as a placebo test is a unique feature of this setting that we were able to leverage as a test of our preferred model. If we were to find statistically significant price elasticities at rate-to-host sites, where there is no reason to expect drivers to respond to price, we would be concerned our estimates for rate-to-driver sites were biased.

Finally, the degree of price responsiveness is striking. These drivers are more responsive than the average residential electricity consumer, implying that electric vehicle loads are easier to shift than typical household loads. A meta-analysis of short-run price elasticity of electricity demand for electricity yielded an average estimate of -0.22 (Zhu 2018). Modern applied research into consumer response to gasoline price fluctuations does find very similar estimates of price responsiveness. Recent short-run estimates of the price elasticity of gasoline demand have included -0.37 for U.S. drivers (Coglianese, et al. 2017), as well as between -0.27 and -0.35 (Levin, Lewis and Wolak 2017). While there may not be a deep connection between these charging elasticities and gasoline price elasticities (the estimates use different sources of variation and the available substitutes in each case are quite different), the similarity is remarkable.