[Bldg-sim] Energy model calibration - normalizing the utility bills to month start-end

Maria Karpman maria.karpman at karpmanconsulting.net
Thu Jun 25 08:48:56 PDT 2015


Jim,

 

FYI attached is a presentation from 2014 ASRHAE/IBPSA summarizing our findings on accuracy of the calibrated simulations based on projects that went through an incentive program in NJ. (You presented there also – loved your Sherlock Holmes hat and pipe J.) To clarify the language, ECM savings estimated via calibrated simulation are referred in the presentation as “projected savings”; ECM savings estimated via “whole building” approach are referred to as “realized savings”.  Slide 7 summarizes high level results – while there was a relatively good alignment between the total projected and realized savings for the entire sample, individual projects were all over the map. 

 

I agree with your last point below that the greatest source of error is too little measured data, but given the finite budget that each project has, I hoped that documents such as Guideline 14 would emphasize the link between the minimum scope of measurements and ECMs considered for a given project. For example Guidelines 14 says the following concerning simulation defaults, which in my opinion is impractical:  

 

5.3.3.3.7 Minimizing Default Values. Check and thoroughly 

understand all default input variables in the simulation

program, as many of the default values have little resemblance

to the actual building being simulated. The fewer the

number of default values used, the more representative the

simulation will be—but only if the changes are well reasoned.

This also includes inspection of the default performance

curves of the various systems and plant equipment, because

such curves can significantly impact the results of the simulation.

Any program default values that are altered, however,

should be well documented.

 

There is also a section on adjusting infiltration / ventilation rates to calibrate the model (quoted below), but it should only apply if project does not include ECMs involving air-sealing (which are common in multifamily and school projects) or ventilation controls such as DCV. If project includes infiltration or ventilation related measures, the related inputs cannot be used to calibrate the model even as “a last resort”.

 

5.3.3.3.6 Estimating Infiltration Rates. Infiltration

rates are difficult to measure and may be treated as an

unknown that is iteratively solved with the simulation program

once the other major parameters are determined. This 

approach is only recommended as a last resort. To solve for

the infiltration rate and/or the ventilation rate iteratively, conduct

a series of simulation runs such that only the infiltration

and/or ventilation rates are changed from one-tenth to as

much as ten times the expected rates. Next, compare the simulation

outputs produced to the measured building data as discussed

below. In addition, supporting evidence should be

used to justify the final choice of variables.

 

I think calibrated simulation should be treated as an extension of retrofit isolation approach, and site measurements performed in support of simulation must at minimum include (a) parameters that are modified to model ECM savings, and (b) simulation inputs that do not change but drive ECM savings. For lighting fixture replacement ECM, pre-retrofit wattage would be an example of (a), and lighting runtime hours would be an example of (b). On the other hand, if there are no ECMs that reduce base load (e.g. plug load, lighting, etc.), but there are ECMs involving HVAC controls, site measurements should focus on operation of existing controls, and establishing existing lighting wattage is not necessary because model calibrated to capture the overall base load (lighting + plug loads) usage may be good enough in this case. We had to develop guidelines for incentive programs that rely on calibrated simulation to specify minimum scope of measurements and acceptable “estimated” inputs for each common ECM type. I hope that a similar guidance would be included at some point in one of ASHRAE standards / guidelines. As it stands now, there is a sea of difference between the specificity of simulation requirements in 90.1 Appendix G compared to calibrated simulation of existing buildings. Unless I am missing some key reference?    

 

Maria

 

-- 

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

 <http://www.karpmanconsulting.net/> www.karpmanconsulting.net 

Phone 860.430.1909 

41C New London Turnpike

Glastonbury, CT 06033

 

From: Jim Dirkes [mailto:jim at buildingperformanceteam.com] 
Sent: Thursday, June 25, 2015 4:49 AM
To: Justin Spencer
Cc: Maria Karpman; Building Simulation
Subject: Re: [Bldg-sim] Energy model calibration - normalizing the utility bills to month start-end

 

Excellent insights in this thread!

Our "two cents":

*	We measure key variables such as larger motor kW and outdoor airflow.  The idea is to eliminate uncertainty for energy aspects that we know are "big".  Because it's easy to obtain, we use the actual weather data.  It's not always a big impact, but eliminates needless uncertainty.
*	We don't trust people's memory about almost anything; it's amazing how different the story can be when told by two different people, both of whom should be knowledgeable.
*	We also don't trust sensor calibration.  Not all sensors matter as much, however; discharge sensors on reheat systems, for example, matter more than room temp sensors.
*	More data is better.  Smart meters are very helpful, as is trend data if the client has taken time to set them up. Not many do :(
*	We have not yet tinkered with sensitivity analysis for uncertain variables such as boiler efficiency or infiltration, but we do run GenOpt for the best R-squared fit to each (exact) billing period and review / revise the output for reasonableness.
*	On the one hand, it bothers me that calibrated models are "grossly incorrect" in Maria's experience.  My bias is that those models may have too little measured data.  On the other hand, this is a brave new world and we probably all have a lot to learn.

 

On Tue, Jun 23, 2015 at 4:51 PM, Justin Spencer <jspencer17 at gmail.com> wrote:

I'll second that this has turned into a really good thread. My personal rule of thumb in doing calibration is, "don't mess with things that relate directly to an ECM." If you have a retrofit ECM going on, you should have detailed data on the pre-installation conditions associated with that ECM. If that isn't documented, your model is never going to provide you with accurate savings results. It pays to think about these things at the fundamental level. And also apply a reasonableness range to those values. What am I most unsure about? What's the possible range of this value? Personally, I twist my dials all in one direction and then use setpoint as my fine calibration factor at the end. We need to remember that we're using models as tools to help guide decision-making. When we make decisions in calibration, we should think about how it will impact the decision being made -- does it impose a bias (like the 35% assumed pre-retrofit boiler efficiency for a boiler retrofit project)? These models are just tools for extrapolating from known energy consumption data to unknown energy consumption data. 

 

An earlier post commented that hourly data might make calibration easier... It doesn't make it easier in the sense of less work, but it does make your model a lot better if you do it right. Calibrating models to real hourly consumption data tells you so much about what is likely wrong with your model and so much about how your real building performs. If you have hourly end use data, or close, then you are really in business. At the aggregate level, this kind of data can tell you that you didn't really know much to start with. For one project for Con Edison, we had access to hourly whole-premise consumption data for several thousand NYC buildings by building type. We were able to use this data to uncover that our models vastly underestimated consumption at nights and on weekends. We didn't know why, but we altered schedules accordingly. 

 

As for the monthly billing data question at the beginnning of this thread, if you're dealing with a building and climate where monthly variability in occupancy and weather are driving changes in energy consumption in smooth, continuous fashion, you can make some other approximations. We've used a simplified slope method, where you calculate the slope in usage between the two adjacent billing months, i.e. if period 1 is 1000 kWh/day, period 2 is 1100 kWh/day, and period 3 is 1200 kWh/day, and all are 30 days long, you wind up with a slope of 3.33 kWh/day. We then assign daily kWh for each month using this slope and then reslice the data to match our calendar months. This is important when you are calibrating an aggregate model, which is a common need for us when we're estimating program-level savings for an ECM. For example, we're using a building simulation model to extrapolate measured consumption at a large sample of sites to usage in a typical year. 

 

Alternatively, for individual buildings, we don't worry about it and just calibrate to the billing periods instead. We're not generally working in eQuest, but rather with the hourly output of whatever engine we're using. 

 

One thing I've been wondering about recently is how to avoid over-calibration. The econometrics/statty/mathy folks I occasionally consort with talk a lot about overfitting of regressions and the same problem applies here. Has anybody ever tried reserving part of their calibration data set to use as a test set? I've gotten some things that looked like really great calibrations in the past, but I'm wondering how folks have sought to prove whether they were getting things right or overfitting. When I teach junior staff about this sort of thing, I always include something about not getting too cute. I point them to the John Von Neumann quote:

·         With four parameters I can fit an  <https://en.wikiquote.org/wiki/Elephant> elephant, and with five I can make him wiggle his trunk.

 

On Tue, Jun 23, 2015 at 1:04 PM, Maria Karpman <maria.karpman at karpmanconsulting.net> wrote:

Jim and all,

 

We have incentive programs in NY (Multifamily Performance Program) and NJ (Pay for Performance Program for C&I buildings, P4P) that require developing calibrated models to estimate ECM savings. Both programs rely on spreadsheet-based tools to facilitate model calibration, and require that the proposed ECM package reduces overall energy consumption by at least 15%. The programs have been around for 5+ years, and have hundreds of participating projects. Part of the incentive is awarded based on the projected (i.e. modeled) savings, and the rest (as much as 50% for P4P) based on the actual achieved savings established using Whole Building approach by comparing pre/post utility bills. 

 

My biggest take away from the involvement with these programs was that a calibrated model that meets MBE,  CVRMSE, and uncertainty requirements of Guideline 14 may produce grossly incorrect ECM savings. It is not feasible to create a calibrated model that can be reliably used to project savings from any conceivable ECM in a commercial non-research setting because, aside from the modeling effort, it would require a lot of very detailed field work. On the other hand, developing a calibrated simulation to estimate savings from a particular set of measures considered for a given project is a much more manageable task. So the bulk of the recent updates to the technical requirements of our incentive programs were focused on itemizing parameters that should be tweaked to achieve calibration depending on the ECMs included in the project scope. For example, if project involves boiler replacement, efficiency of existing boilers that are being replaced must be measured. (We had a project which modeled existing boiler as 35% efficient because that produced calibrated simulation. Of course such model would very likely exaggerate savings from installing a new boiler.) 

 

Do any of you know references that outline calibration techniques depending on the ECMs being modeled, beyond the general advice included in IPM&VP?

 

Thanks,

 

Maria

 

-- 

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net <http://www.karpmanconsulting.net/>  

Phone 860.430.1909 

41C New London Turnpike

Glastonbury, CT 06033

 

From: Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] On Behalf Of Jim Dirkes
Sent: Tuesday, June 23, 2015 1:31 PM


To: bldg-sim at lists.onebuilding.org
Subject: Re: [Bldg-sim] Energy model calibration - normalizing the utility bills to month start-end

 

I'm encouraged to see so many people addressing this topic because it means you are modeling existing buildings; a lot of work is needed in this arena.  Keep it up!

 

We, as usual, have a spreadsheet solution.  In this case, the spreadsheet is happy to use billing periods of any length, such as is normal for day-of-reading variations, but also to combine "estimated" readings into a period that has an actual reading at each end.

It requires that you tell EnergyPlus to report hourly meter data for each fuel (e.g., electricity and natural gas).  A macro totals that data into the billing periods for your site, displays the predicted vs actual energy and calculates an R-squared value for each fuel and the total. An example is shown below.

 

Inline image 1

 

On Tue, Jun 23, 2015 at 12:02 PM, Maria Karpman <maria.karpman at karpmanconsulting.net> wrote:

Hello all,

 

We usually do the following to calibrate model to monthly utility bills:

1)      Create or purchase weather file corresponding to pre-retrofit period for which we have billing data. Lately we’ve been using WeatherAnalytics files, which we found to be more cost effective than creating our own (they charge $40 for an annual file).

2)      Run simulation using this weather file instead of TMY.

3)      Standard simulation reports (we typically use eQUEST) show usage by calendar month (e.g. January, February, etc.) which is usually not aligned with dates of utility bills, as noted in the question that started this thread. As Brian mentioned in one of the earlier posts, this may be circumvented by entering the actual meter read dates into eQUEST as shown in the screenshot below. This will align usages shown in eQUEST’s “E*” reports such as ES-E with the actual utility bills.  The approach does not allow entering more than one read date per month (e.g. we can’t capture April 3 – 28 bill). For projects where this limitation is an issue we generate hourly reports that show consumption by end use for each meter in the project, and aggregate it into periods that are aligned with utility bills. 



  

4)      We then copy simulation outputs (either from ES-E or hourly reports, depending on the method used) into a standard spreadsheet with utility data. The spreadsheet is set up to plot side by side monthly utility bills and simulated usage, and also calculates normalized mean bias error (NMBE) and variance CV(RMSE).   

5)      If we did not to where we want to be with NMBE and CV(RMSE) we adjust and re-run the model, and re-paste results into the same spreadsheet. 

 

In my experience regression analysis using weather as independent variable (i.e. running model with TMY file and normalizing for difference in weather) or relying on HDD to allocate usage to billing periods can be very misleading, mainly because on many projects weather is not the main driver of consumption. For example energy usage of a school during a given time period depends much more on vacation schedule than outdoor dry bulb temperatures.  

 

Thanks,

 

-- 

Maria Karpman LEED AP, BEMP, CEM

________________

Karpman Consulting

www.karpmanconsulting.net <http://www.karpmanconsulting.net/>  

Phone 860.430.1909 

41C New London Turnpike

Glastonbury, CT 06033

 

From: Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] On Behalf Of Jeff Haberl
Sent: Tuesday, June 23, 2015 10:16 AM
To: Joe Huang; bldg-sim at lists.onebuilding.org


Subject: Re: [Bldg-sim] Energy model calibration - normalizing the utility bills to month start-end

 

Hello Joe,

 

Yes, you can count the degree days and regress against that to show a correlation. However, one will get a better "fit" to the weather data if you regress to the degree day that is calculated for the balance point temperature of the building -- hence the inverse model toolkit or the variable based degree day method.

 

PRISM actually calculates the degree days to a variety of change points and actually provides a table for each location that you use as a look up. The IMT will actually perform a variable based degree day calculation that agrees well with PRISM. IMT will also provide you with the average daily temperature for the billing period.

 

When using DOE-2 for actual billing periods, one will have to extract the appropriate hourly variable, sum it to daily and then regroup to align with the billing periods. Here's a chunk of code that will create a dummy plant, display PV-A, PS-A, PS-E and BEPS, and extract the relevant hourly variables to normalize the BEPS to the utility bills:

 

INPUT PLANT ..

 

PLANT-REPORT VERIFICATION = (PV-A)

$ PV-A, EQUIPMENT SIZES

 

SUMMARY = (PS-A,PS-E,BEPS)

 

$ PS-A, PLANT ENERGY UTILIZATION SUMMARY

$ PS-E, MONTHLY ENERGY END USE SUMMARY

$ BEPS, BUILDING ENERGY PERFORMANCE SUMMARY

 

HVAC=PLANT-ASSIGNMENT ..

 

$ EQUIPMENT DESCRIPTION

$ ELECTRIC DOMESTIC WATER HEATER

 

BOIL-1 =PLANT-EQUIPMENT TYPE=ELEC-DHW-HEATER SIZE=-999 ..

 

$ ELECTRIC HOT-WATER BOILER

 

BOIL-2 =PLANT-EQUIPMENT TYPE=ELEC-HW-BOILER SIZE=-999 ..

 

$ HERMETICALLY SEALED CENT CHILLER

 

CHIL-1 =PLANT-EQUIPMENT TYPE=HERM-CENT-CHLR SIZE=-999 ..

 

$ Graphics block for Data Processing ***

 

RP-3 = SCHEDULE THRU DEC 31 (ALL) (1,24) (1) ..

 

$ 8 = Total PLANT heating load (Btu/h)

$ 9 = Total PLANT cooling load (Btu/h)

$ 10 = Total PLANT electric load (Btu/h)

 

BLOCK-3-1 = REPORT-BLOCK

VARIABLE-TYPE = PLANT

VARIABLE-LIST = (8,9,10) ..

BLOCK-3-2 = REPORT-BLOCK

VARIABLE-TYPE = GLOBAL

VARIABLE-LIST = (1) ..

HR-3 = HOURLY-REPORT

REPORT-SCHEDULE = RP-3

REPORT-BLOCK = (BLOCK-3-1,BLOCK-3-2) ..

 

END ..

 

COMPUTE PLANT ..

 

STOP ..

 

8=!  8=)  :=)  8=)  ;=)  8=)  8=(  8=)  8=()  8=)  8=|  8=)  :=')  8=) 8=?
Jeff S. Haberl, Ph.D.,P.E.inactive,FASHRAE,FIBPSA,......jhaberl at tamu.edu <mailto:........jhaberl at tamu.edu> 
Professor........................................................................Office Ph: 979-845-6507
Department of Architecture............................................Lab Ph:979-845-6065
Energy Systems Laboratory...........................................FAX: 979-862-2457
Texas A&M University...................................................77843-3581
College Station, Texas, USA, 77843.............................http://esl.tamu.edu
8=/  8=)  :=)  8=)  ;=)  8=)  8=()  8=)  :=)  8=)  8=!  8=)  8=? 8=) 8=0

  _____  

From: Bldg-sim [bldg-sim-bounces at lists.onebuilding.org] on behalf of Joe Huang [yjhuang at whiteboxtechnologies.com]
Sent: Monday, June 22, 2015 9:17 PM
To: bldg-sim at lists.onebuilding.org
Subject: Re: [Bldg-sim] Energy model calibration - normalizing the utility bills to month start-end

Maybe I'm missing something here, but why can't you just count up the degree days for the utility period?
I hope you're not working with average or "typical year" degree days, but the degree days from the same time period.

I also recall that the old Princeton Scorekeeping Method (PRISM) back in the 1980's allows the user to enter the degree days for that time period, so it's not a new problem.

Joe

Joe Huang
White Box Technologies, Inc.
346 Rheem Blvd., Suite 205A
Moraga CA 94556
yjhuang at whiteboxtechnologies.com
http://weather.whiteboxtechnologies.com for simulation-ready weather data
(o) (925)388-0265 <tel:%28925%29388-0265> 
(c) (510)928-2683 <tel:%28510%29928-2683> 
"building energy simulations at your fingertips"

On 6/22/2015 6:09 AM, Jones, Christopher wrote:

When calibrating an energy model to utility bills the utility bills often don’t align with the month start and end.  I have reviewed a couple methods to calendar normalize the utility bills but find them somewhat unsatisfactory.

 

For example the method I am looking at does the following:

The April gas bill runs from March 25 – April 24.  The algorithm takes the average number of m3 per day from that bill, applies it to the days in April.  Then it takes the average number of days from the May bill which runs from April 24 – May 25 and applies that average to the remaining days in April.  

 

The issue is that the March-April period has much higher HDD than the April-May period and the “normalized” gas usage is significantly lower than the simulation data for April.

 

I am wondering if there are any papers or other sources of information as to how others approach this problem.

 

 

cid:image003.png at 01D09C46.E75BA0D0

Christopher Jones,P.Eng. 
Senior Engineer

 

WSP Canada Inc.

2300 Yonge Street, Suite 2300

Toronto, ON M4P 1E4
T +1 416-644-4226 <tel:%2B1%20416-644-4226> 

F +1 416-487-9766 <tel:%2B1%20416-487-9766> 

C +1 416-697-0065 <tel:%2B1%20416-697-0065> 

 

www.wspgroup.com <http://www.wspgroup.com/>  

 

  _____  


You are receiving this communication because you are listed as a current WSP contact. Should you have any questions regarding WSP’s electronic communications policy, please consult our Anti-Spam Commitment www.wspgroup.com/casl <https://teesmail.tees.tamus.edu/owa/UrlBlockedError.aspx> . For any concern or if you believe you should not be receiving this message, please forward this message to us at caslcompliance at wspgroup.com so that we can promptly address your request. This message is intended only for the use of the individual or entity to which it is addressed, and may contain information which is privileged, confidential, proprietary or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying or in any way using this message. If you have received this communication in error, please notify the sender, and destroy and delete any copies you may have received. 

WSP provides professional land surveying services through the following entities: WSP Surveys (AB) Limited Partnership and WSP Surveys (BC) Limited Partnership 

 

_______________________________________________
Bldg-sim mailing list
http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
To unsubscribe from this mailing list send  a blank message to BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG

 

  _____  

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4800 / Virus Database: 4365/10055 - Release Date: 06/19/15


_______________________________________________
Bldg-sim mailing list
http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
To unsubscribe from this mailing list send  a blank message to BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG





 

-- 

James V Dirkes II, PE, BEMP, LEED AP
CEO/President
The Building Performance Team Inc.
1631 Acacia Dr, GR, Mi 49504

Direct: 616.450.8653
jim at buildingperformanceteam.com

Website  <http://buildingperformanceteamcom> l  LinkedIn <https://www.linkedin.com/pub/jim-dirkes/7/444/413> 

Studies show that four out of every three people have a hard time with math.

  _____  

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4800 / Virus Database: 4365/10055 - Release Date: 06/19/15


_______________________________________________
Bldg-sim mailing list
http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
To unsubscribe from this mailing list send  a blank message to BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG

 





 

-- 

James V Dirkes II, PE, BEMP, LEED AP
CEO/President
The Building Performance Team Inc.
1631 Acacia Dr, GR, Mi 49504

Direct: 616.450.8653
jim at buildingperformanceteam.com

Website  <http://buildingperformanceteamcom> l  LinkedIn <https://www.linkedin.com/pub/jim-dirkes/7/444/413> 

Studies show that four out of every three people have a hard time with math.

  _____  

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4800 / Virus Database: 4365/10055 - Release Date: 06/19/15

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20150625/01707468/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 139044 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20150625/01707468/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 58565 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20150625/01707468/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 6574 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20150625/01707468/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IBPSA_0910a.pdf
Type: application/pdf
Size: 692430 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20150625/01707468/attachment-0001.pdf>


More information about the Bldg-sim mailing list