[Bldg-sim] Lies, darn lies, and statistics

Tue Mar 14 15:27:17 PDT 2023

Hello Chris:

Thanks for weighing in on the IMT and G14.

First, in your original email you said:

As one of the co-developers of the IMT and a Committee member of G14-2023, I guess I'd better weigh-in. There's a lot to unpack in one email, but I'll try.

However, before I start I need to say that G14-2023 is due out this year after public review and comment. Hence, I'd suggest you get a copy and take a look to see if it is improved or not.

In addition, the discussion of the IMT and Guideline 14-2023 should be separated, since one was an ASHRAE Research Project, RP-1050, that had the goal of developing public regression code that others could take and use to put "ASHRAE under the hood" so to speak.

The reason for this was that there were many "methods" that were out there that were proprietary, or perhaps published, that did not give the use a computer code that they could use freely, and/or embed into their code to produce similar results -- in fact, that was the purpose and the wisdom of the PES that developed RP-1050, to produce public code that other could use.

I agree that your issues on hourly vs daily vs monthly is not addressed in Guideline 14, nor in the IMT. This was not on purpose, it just happened. The reason is pretty simple, over the last 20 years, things have evolved, and folks have hourly or sub-hourly data available, and there's no advice as to what's what, and or what to use.

This particular topic is actually being addressed by ASHRAE's new Guideline 45R (yes, another guideline) that seeks to address energy use, indoor environmental parameters (i.e., comfort, IAQ, daylighting, etc.) into one guideline. Stay tuned.

However, in the current G14-2023, there is little advice as to when to use hourly, daily or daily and what for. Only modeling guidelines for a generic time frame.

Finally, there are details about using the IMT (and ASHRAE RP-1093) to develop baseline and post-retrofit that are now contained in G14-2023. So, hopefully this will help.

I agree that the statistical parameters for G14-2023 are rather strict, and I would encourage you to contact the Committee members that developed these (I'll be happy to say who in a separate email). However, suffice it to say, these uncertainty stastics are complex.

 As for the comment about modeling K-12 schools with monthly use, this has been addresed previously for monthly data in Margaret Fels 1986 Energy and Buildings article using a "PRISM plus UNDERBAR" method. Email me and I'll send you the articles.

Your reference to "GENOPT" makes he shiver, as I don't trust anything that I can't take apart and see how the parts work. So, I can't comment on this.

Yes, there are a lot of variables that influence whole-building energy use, however, that is the purpose of the analyst to tease these variables out of the analysis to see what's what.

"Healing" that data is needed since there are most likely "blips" in a multi-year data stream that are "out of bounds", and most likely should not be included in the analysis. Details of how this "period" was identified, and/or how the data were "healed' are necessary to explain to the authorities (in the case of an ESCO contract).

Monthly models can be garbage if there are big differences in the building operation from one month to the next. One example is university buildings, that have semester, non-semester and vacation periods that most likely need daily or even hourly data.

I'm nervous about "monthly degree-day" models since most commercial buildings don't behave as modeled by a 65F degree day model.

CVRMSE and adj R2 can go a long way in describing the accuracy of a model.

I agree that we need "some sort of representative simulation model"... however, none such exists at this time. So we need to rely on regression.

In regard to M&V plans, we've made many of these for ESCO projects in Texas that are mindful of resources and accuracyt, we'd be happy to share.

Hope this helps.

Jeff

Jeff S. Haberl, Ph.D., P.E.inactive, FASHRAE,FIBPSA               We are like fluttering leaves on the branches of trees

Department of Architecture                                                       in the forests of the landscape that surrounds us.

Texas A&M University                                                                  If we could, for just a moment, flutter together,

College Station, TX 77845-3581                                                 We could lift the earth up to be a better place.  JSH 2022

Office: 979-845-6507, Lab: 979-845-6065

Fax 979-862-2457

jhaberl at tamu.edu,www.esl.tamu.edu<mailto:jhaberl at tamu.edu,www.esl.tamu.edu>

________________________________
From: Bldg-sim <bldg-sim-bounces at lists.onebuilding.org> on behalf of Chris Yates via Bldg-sim <bldg-sim at lists.onebuilding.org>
Sent: Tuesday, March 14, 2023 3:12 PM
To: David Eldridge <DEldridge at grummanbutkus.com>
Cc: BldgSim Mailing List <bldg-sim at onebuilding.org>
Subject: Re: [Bldg-sim] Lies, darn lies, and statistics

I agree that better measurement is the key. It's surprising how completely off the radar that is. There is an expectation of major interventions, new facades, plant replacement, etc, but the basics get neglected. I have a theory about using
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
I agree that better measurement is the key. It's surprising how completely off the radar that is. There is an expectation of major interventions, new facades, plant replacement, etc, but the basics get neglected.

I have a theory about using statistical models to assist with simulation models. In theory, we should not just aim for low rmse, but coefficients should be comparably close when performing the same regression on simulated results.

Some more words of wisdom :) https://en.wikipedia.org/wiki/There_are_unknown_unknowns?wprov=sfla1<https://urldefense.com/v3/__https://en.wikipedia.org/wiki/There_are_unknown_unknowns?wprov=sfla1__;!!KwNVnqRv!HrVN_15GIKX6UniAxigrcE2QlYxK3ZmZ5eeSfmH8RG2-HBoy2z92r5ff8OSSBB4ircXAC4pzIAkfHuEuTcpHdPocCio$>

On Tue, 14 Mar 2023, 19:35 David Eldridge, <DEldridge at grummanbutkus.com<mailto:DEldridge at grummanbutkus.com>> wrote:

You are allowed to make an M&V plan and then define what you did given the information available. In your case you are using G14 informatively, it wasn’t required that you follow it or report that you were using it.

The only thing I can add is maybe you can focus on validating some of the inputs – if the utility data is spotty then document what you do know from the input side – if you can show some of your most sensitive input variables are validated that would help give confidence  that the outputs are also likely to be useful.

David

David S. Eldridge, Jr., P.E., BEMP, BEAP, HBDP, GGF/A

Direct: (847) 316-9224 | Mobile: (773) 490-5038

Grumman|Butkus Associates | 820 Davis Street, Suite 300 | Evanston, IL 60201

Energy Efficiency Consultants and Sustainable Design Engineers

grummanbutkus.com<https://urldefense.com/v3/__http://grummanbutkus.com/__;!!KwNVnqRv!HrVN_15GIKX6UniAxigrcE2QlYxK3ZmZ5eeSfmH8RG2-HBoy2z92r5ff8OSSBB4ircXAC4pzIAkfHuEuTcpH5yZo2ow$> | Facebook<https://urldefense.com/v3/__https://www.facebook.com/pages/GrummanButkus-Associates/1385285015032526__;!!KwNVnqRv!HrVN_15GIKX6UniAxigrcE2QlYxK3ZmZ5eeSfmH8RG2-HBoy2z92r5ff8OSSBB4ircXAC4pzIAkfHuEuTcpH2WfwIxU$> | Twitter<https://urldefense.com/v3/__https://twitter.com/grummanbutkus__;!!KwNVnqRv!HrVN_15GIKX6UniAxigrcE2QlYxK3ZmZ5eeSfmH8RG2-HBoy2z92r5ff8OSSBB4ircXAC4pzIAkfHuEuTcpH8cRBjGM$> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/102242?trk=tyah&trkInfo=clickedVertical*3Acompany*2CclickedEntityId*3A102242*2Cidx*3A2-3-6*2CtarId*3A1445536322603*2Ctas*3Agrumman__;JSUlJSUlJSUl!!KwNVnqRv!HrVN_15GIKX6UniAxigrcE2QlYxK3ZmZ5eeSfmH8RG2-HBoy2z92r5ff8OSSBB4ircXAC4pzIAkfHuEuTcpHbqvGvIs$> | Instagram<https://urldefense.com/v3/__https://www.instagram.com/grummanbutkus/__;!!KwNVnqRv!HrVN_15GIKX6UniAxigrcE2QlYxK3ZmZ5eeSfmH8RG2-HBoy2z92r5ff8OSSBB4ircXAC4pzIAkfHuEuTcpHzLSB4c8$>

From: Bldg-sim <bldg-sim-bounces at lists.onebuilding.org<mailto:bldg-sim-bounces at lists.onebuilding.org>> On Behalf Of Chris Yates via Bldg-sim
Sent: Tuesday, March 14, 2023 1:10 PM
To: 'David Eldridge' <dancingdavide at hotmail.com<mailto:dancingdavide at hotmail.com>>; 'Jim Dirkes' <jvdirkes2 at protonmail.com<mailto:jvdirkes2 at protonmail.com>>
Cc: bldg-sim at onebuilding.org<mailto:bldg-sim at onebuilding.org>
Subject: Re: [Bldg-sim] Lies, darn lies, and statistics

I think I need to qualify this: informed by G14, but definitely not compliant with it! There is some allowance for repairing or “healing” data, but when the data has a lot of holes or modes/ category variables then forget it. This is my case, but the client still wants some kind of representative simulation.

Monthly models can be garbage. School holidays cut across months at different times, combined heat and power is popular which complicates gas usage especially when metering is limited… did the heat by-product of electricity generation go to the building, or was it rejected? They can work for heating in our temperate climate, but not for cooling.

Nevertheless, we need some kind of representative simulation model. We can’t make any ECM qualifying claims but we can do something useful. This is where these methods can give you a lot of insight before you start modelling.

I hadn’t tried the IMT previously. We tend to have limited our regression analysis to monthly “degree day” methods (your 2p model, I think). I plugged some project specific daily electricity data into the MVR example (multi variate regression) and it seemed to give decent CVRMSE (~2-3%) but low R2 (~0.7). However, it appeared to provide some insight on cooling usage (monthly 2p models are meaningless for this in the UK’s temperate climate).

I also made some 2p monthly models of gas. I thought these were good until I compared successive years. I guess this is were understanding a range of statistical indices is helpful.

Here’s the final rub, because the underlying data has so many inconsistencies that can only be made sense of with some regression models, it’s easier to “calibrate” the simulation model to the regression models than the original data. But I may use 2p monthly for gas, daily MVR for electric…

So I need to ask if I have wondered completely off-piste with this!

Chris

From: David Eldridge <dancingdavide at hotmail.com<mailto:dancingdavide at hotmail.com>>
Sent: Tuesday, March 14, 2023 12:53 PM
To: Jim Dirkes <jvdirkes2 at protonmail.com<mailto:jvdirkes2 at protonmail.com>>
Cc: chris.malcolm.yates at gmail.com<mailto:chris.malcolm.yates at gmail.com>; bldg-sim at onebuilding.org<mailto:bldg-sim at onebuilding.org>
Subject: Re: [Bldg-sim] Lies, darn lies, and statistics

Indirectly there probably isn’t a daily set of metrics in the Guideline since the simulation programs aren’t usually outputting daily results, but there’s no reason there couldn’t be one statistically.

You could make one if you had only daily utility data and had to aggregate the simulation results to daily totals, there isn’t a published target metric but you could still show that you calculated one and why you think it was a good or bad result.

DSE Mobile

On Mar 14, 2023, at 6:10 AM, Jim Dirkes via Bldg-sim <bldg-sim at lists.onebuilding.org<mailto:bldg-sim at lists.onebuilding.org>> wrote:

Dear Chris,

Kudos for appreciating a gap in your understanding. (I'm in your camp)

On the other hand, there are SO many variables in building operation that, short of a highly instrumented (and carefully calibrated) building for everything from lights to people to plug loads to HVAC - calibration is a fiction (and I'm confident that no such building exists). Daily calibration is a complete fiction, perhaps even a deception. On top of that, a "calibrated" model is just a moment in time; everything going forward is guaranteed to be different than during the calibration time period.

I think of "calibration" as more like a sensitivity analysis - determine which variables matter more and which matter less. GenOpt works nicely for that purpose https://github.com/lbl-srg/GenOpt<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flbl-srg%2FGenOpt&data=05%7C01%7Cjhaberl%40tamu.edu%7Cc3b177f812eb47870eea08db24d3d434%7C68f381e346da47b9ba576f322b8f0da1%7C1%7C0%7C638144264455996569%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iYv9ebvl%2FI1Ga04%2B6PTHKnfZfREgcvg5Ap26jdbkKqk%3D&reserved=0>

Jim Dirkes  1631 Acacia Drive NW Grand Rapids, MI 49504 -  616 450 8653

Coffee Conversation:

The "individual" is an impossible concept, conceived by the Enlightenment philosophers. It makes no sense to the Christian. In marriages, and families, in associations and friendships and religious orders, we are not individuals, but a communion of persons.

------- Original Message -------
On Tuesday, March 14th, 2023 at 6:52 AM, Chris Yates via Bldg-sim <bldg-sim at lists.onebuilding.org<mailto:bldg-sim at lists.onebuilding.org>> wrote:

Hi All

I do find ASHRAE Guideline 14 a little too hardcore for my basic understanding of statistics. I can plug any of the equations into Excel, but I’ve realised my statistics understanding is very limited! (I’m outed!)

We don’t actually have to work to G14 in the UK (probably good because my copy is a bit old). I finally realised I didn’t know enough after I’d been (lazily) using R2 in Excel on some monthly data. I thought that R2 > 0.9 was generally ok… yeah, it wasn’t.

So, are there any easy to understand resources available?

I’ve been messing around with the IMT as well. It’s been fun going back to DOS 😊. This got me into daily methods, which leads to my next question. Is there any reason why there isn’t a daily calibration option specified in G14?

Many thanks!

Chris Yates

_______________________________________________
Bldg-sim mailing list
http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org<https://urldefense.com/v3/__http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org__;!!KwNVnqRv!HrVN_15GIKX6UniAxigrcE2QlYxK3ZmZ5eeSfmH8RG2-HBoy2z92r5ff8OSSBB4ircXAC4pzIAkfHuEuTcpHTv1Vhko$>
To unsubscribe from this mailing list send  a blank message to BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG<mailto:BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20230314/7be42b51/attachment.htm>