[Bldg-sim] Lies, darn lies, and statistics

Tue Mar 14 19:32:27 PDT 2023

Hello Jeff,

Thank you kindly for the long reply and history lesson, it's interesting to
know BES originated from bomb shelters, there is one still operating not
far from where I live :)

As a gamer growing up, I've always wondered why BES is not voxel-based like
minecraft or why we are not using real-time ray-tracing from GPUs for solar
insolation computations.

Anyways, back to model training/calibration. Loss function (in ML training)
or cost function (in optimization) is the measuring stick telling the
optimization program how to assess the performance of the model. Usually
MSE/RMSE is used but it has its limitations, that's why in ML we sometimes
use log errors
<https://medium.com/analytics-vidhya/root-mean-square-log-error-rmse-vs-rmlse-935c6cc1802a>
to deal with outliers. Another example, in fraud detection, we customize
loss functions to penalize false negatives harsher than false positives.

Actually many years ago, someone mentioned custom cost function to deal
with peak loads for BES calibration, using ExcaliBEM:
https://unmethours.com/question/13182/genoptenergyplus-best-algorithm-for-peak-load-reduction-load-shift/

Now on cross-validation
<https://medium.com/@soumyachess1496/cross-validation-in-time-series-566ae4981ce4>
(CV), it is a wrapper around each model training process to make sure the
model is not over-fitting. It is usually the first thing I check when
seeing people applying ML models.
In theory these two practices should not interfere with the bounded
parameter space.

Thanks,
Shawn

On Tue, Mar 14, 2023 at 9:05 PM Haberl, Jeff <jhaberl at tamu.edu> wrote:

> Thanks Shawn:
>
> Good thoughts. Maybe outside my expertise, but I'll take a crack.
>
> First, BES are constrained models in that they only allow the user to
> change cerain parameters and not others. Furthermore, the BES is a
> preconfigured model of a building that tries to include all the relevant
> heat transfer properties to the ambient conditions and the SYSTEM and PLANT
> reactions to these loads using user-defined inputs.
>
> This situation only gets worse when on considers that even EP+ was
> influenced by the original LSPE architecture that was suggested for the
> Post Office program in the early design of the TACS program. Although EP+
> has evolved way beyond the constraints of LSPE BES programs, the underlying
> code still has ghosts of this architecture in its algorithms. We recently
> published one of three new articles for STBE that updates the previous
> "Origins" articles we published. The first one is "Origins of
> whole-building energy simulations for high-performance commercial
> buildings: Contributions of NATEOUS, SHEP, TACS, CP-26, and RESPTK
> programs", by Jounghwan Ahn and myself. Two more articles to follow soon in
> STBE.
>
> In the first article, thanks in part to Jason Glazier for dropping-off a
> pickup truck's worth of books when Robert Henniger retired, we outline how
> the early simulation programs can be traced to simulations of bomb shelters
> in the 1960s by Tamami Kusuda and Metin Lokmonhekim and the earliest
> FORTRAN programs that codified response factors by Tamami Kusuda (RESPTK).
> All of these books are now scanned and can be found on the onebuilding.org
> website.
>
> In these books you see that what was done in the early days was done
> because of constraints in the computing hardware in the 1960s and into the
> 1970s. Unfortunately, ghosts of this kind of thinking still exist today,
> with a few exceptions, when we try to define a "thermal zone" as a node
> that maintains a known temperature given varying internal inputs and
> external R-C networks that connect the node to the exterior. So, although
> we've come a long way, we still have a long way to go to "cut all the
> stings" to the old LSPE constraints.
>
> Second, there are prototypes of how to move forward into the future, but
> their "secrets" are highly guarded, which is not good for the general BES
> community. However, it is a fact of life and we have to live with it and
> put our minds to use to reverse engineer what the proprietary codes are
> doing so we can publish in the public domain so all can see what is done.
> EP+ is one effort that stands alone it what it has accomplished AND has
> accomplished in the public domain -- bravo.
>
> However, there are spectacular proprietary codes that are doing
> breathtaking simulations that are hidden "behind the wall", and so it goes.
>
> Examples include combined CFM and BES -- a daunting task. Another is full
> force raytracing and BES programs. Yet another is Urban Scale Building
> Energy Modeling (UBEM). All are paths into the future that are critical for
> BES modelers to learn and incorporate into the work. Finally, there are
> robo-simuilations that are now being attached to BIM, which is the future
> for BES. Yet, there is little discussion about public code, where to get it
> and whether or not it is worth a hoot.
>
> So, I think BLDG-SIM has a role in providing a public space where "issues"
> such as the above can be discussed, and vetted in the public domain, so
> those who have the development money can get a second opinion about how
> they are spending state or federal money.
>
> BLDG-SIM can also be seed-corn for student thesis topics since these are
> like hens teeth when thinking them up and then following through to
> publication
>
> Just some thoughts.
>
> Jeff
>
> Jeff S. Haberl, Ph.D., P.E.inactive, FASHRAE,FIBPSA               We are
> like fluttering leaves on the branches of trees
>
> Department of Architecture
>        in the forests of the landscape that surrounds us.
>
> Texas A&M University
>             If we could, for just a moment, flutter together,
>
> College Station, TX 77845-3581
>      We could lift the earth up to be a better place.  JSH 2022
>
> Office: 979-845-6507, Lab: 979-845-6065
>
> Fax 979-862-2457
>
> jhaberl at tamu.edu,www.esl.tamu.edu
>
> ------------------------------
> *From:* Bldg-sim <bldg-sim-bounces at lists.onebuilding.org> on behalf of
> Shawn Shi via Bldg-sim <bldg-sim at lists.onebuilding.org>
> *Sent:* Tuesday, March 14, 2023 7:19 PMGood
> *To:* Dru Crawley <dbcrawley at gmail.com>
> *Cc:* bldg-sim at onebuilding.org <bldg-sim at onebuilding.org>
> *Subject:* Re: [Bldg-sim] Lies, darn lies, and statistics
>
> Hello all, Just a couple thoughts from a random passerby. It’s rare to see
> this email group this active :p Now please bear with my 2 cents: One thing
> that always bother me when working on energy model calibration in the past
> was: why was cross
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
>
> ZjQcmQRYFpfptBannerEnd
> Hello all,
>
> Just a couple thoughts from a random passerby. It’s rare to see this email
> group this active :p
>
> Now please bear with my 2 cents:
>
> One thing that always bother me when working on energy model calibration
> in the past was: why was cross validation (CV) not used?
>
> One fundamental technique in Machine Learning to avoid overfitting is
> through cross validation. Fitting a complex first-principle model is
> similar to training a complex black-box model: high degree of freedom,
> risks of overfitting. CV can be adopted to avoid incorrect local minima
> during model training. There are multiple CV approaches for timeseries
> model training.
>
> And regarding predicting peak loads, etc. When training a model, the loss
> function (equation that describes model accuracy) dictates how you want the
> model to behave. Usually RMSE is used but you can customize it to make the
> model behave in a certain way better. For example, something like a
> compound loss function that combines prediction errors from both overall
> predictions as well as peak prediction. You can adjust the weight for each
> component to dictate which part is more important than another.
>
> Thanks,
> Shawn Shi
>
> On Tue, Mar 14, 2023 at 7:41 PM Dru Crawley via Bldg-sim <
> bldg-sim at lists.onebuilding.org> wrote:
>
> Yep, not me. But I use it a lot.
>
> On Tue, Mar 14, 2023, 4:19 PM Vikram Sami via Bldg-sim <
> bldg-sim at lists.onebuilding.org> wrote:
>
> George Box coined that one I think
>
>
>
> *From:* Bldg-sim <bldg-sim-bounces at lists.onebuilding.org> *On Behalf Of *Jim
> Dirkes via Bldg-sim
> *Sent:* Tuesday, March 14, 2023 12:13 PM
> *To:* chris.malcolm.yates at gmail.com
> *Cc:* bldg-sim at onebuilding.org
> *Subject:* Re: [Bldg-sim] Lies, darn lies, and statistics
>
>
>
> I think Dru Crawley coined the phrase, "All models are wrong. Some are
> useful."
>
>
>
> Chris, you highlighted an assortment of variables which are omnipresent,
> inconsistent and uncontrollable - so what, exactly, does your client
> expect? Is it a realistic expectation? For example, are they going to nail
> you to the wall when, inevitably, you are "wrong" next year?
>
>
>
> I have not calibrated many models, but have been made more appreciative of
> all the uncontrollables by the ones I calibrated :(. All the statisticians
> know that there is always more than one solution which will result in a
> high R2 value or low CVRSME, so which is correct?
>
>
>
> Rather than a calibrated model, lately I've been encouraging clients to
> consider one of the FDD platforms on top of their Building Automation
> System. Spending time and money to evaluate whether things are working
> properly makes more sense to me - it's "real life" vs a prediction. (Can't
> forget to mention thoughtful and thorough commissioning here; that's
> essential.)
>
>
>
> ps, I love your thoughtful approach. You're setting a great example! One
> aspect of that is to reach out to the wider modeling community to gather
> input and feedback.
>
>
>
> Jim Dirkes  1631 Acacia Drive NW Grand Rapids, MI 49504
> <https://urldefense.com/v3/__https://www.google.com/maps/search/1631*Acacia*Drive*NW*Grand*Rapids,*MI*49504?entry=gmail&source=g__;KysrKysrKw!!KwNVnqRv!Gyy3l3XXdU0uo6SKEX1uDKDVeqKvb3BM7ELdcAhCyGrNAZnPW1FVt64gzSX8gd4XJkQtPfaNE_z8YgDv9YFrSf_crIA$>
> -  616 450 8653
>
> *Coffee Conversation:*
>
> The "individual" is an impossible concept, conceived by the Enlightenment
> philosophers. It makes no sense to the Christian. In marriages, and
> families, in associations and friendships and religious orders, we are not
> individuals, but a communion of persons.
>
>
>
> ------- Original Message -------
> On Tuesday, March 14th, 2023 at 2:10 PM, chris.malcolm.yates at gmail.com <
> chris.malcolm.yates at gmail.com> wrote:
>
>
> I think I need to qualify this: informed by G14, but definitely not
> compliant with it! There is some allowance for repairing or “healing” data,
> but when the data has a lot of holes or modes/ category variables then
> forget it. This is my case, but the client still wants some kind of
> representative simulation.
>
>
>
> Monthly models can be garbage. School holidays cut across months at
> different times, combined heat and power is popular which complicates gas
> usage especially when metering is limited… did the heat by-product of
> electricity generation go to the building, or was it rejected? They can
> work for heating in our temperate climate, but not for cooling.
>
>
>
> Nevertheless, we need some kind of representative simulation model. We
> can’t make any ECM qualifying claims but we can do something useful. This
> is where these methods can give you a lot of insight before you start
> modelling.
>
>
>
> I hadn’t tried the IMT previously. We tend to have limited our regression
> analysis to monthly “degree day” methods (your 2p model, I think). I
> plugged some project specific daily electricity data into the MVR example
> (multi variate regression) and it seemed to give decent CVRMSE (~2-3%) but
> low R2 (~0.7). However, it appeared to provide some insight on cooling
> usage (monthly 2p models are meaningless for this in the UK’s temperate
> climate).
>
>
>
> I also made some 2p monthly models of gas. I thought these were good until
> I compared successive years. *I guess this is were understanding a range
> of statistical indices is helpful.*
>
>
>
> Here’s the final rub, because the underlying data has so many
> inconsistencies that can only be made sense of with some regression models,
> it’s easier to “calibrate” the simulation model to the regression models
> than the original data. But I may use 2p monthly for gas, daily MVR for
> electric…
>
>
>
> So I need to ask if I have wondered completely off-piste with this!
>
>
>
> Chris
>
>
>
> *From:* David Eldridge <dancingdavide at hotmail.com>
> *Sent:* Tuesday, March 14, 2023 12:53 PM
> *To:* Jim Dirkes <jvdirkes2 at protonmail.com>
> *Cc:* chris.malcolm.yates at gmail.com; bldg-sim at onebuilding.org
> *Subject:* Re: [Bldg-sim] Lies, darn lies, and statistics
>
>
>
> Indirectly there probably isn’t a daily set of metrics in the Guideline
> since the simulation programs aren’t usually outputting daily results, but
> there’s no reason there couldn’t be one statistically.
>
>
>
> You could make one if you had only daily utility data and had to aggregate
> the simulation results to daily totals, there isn’t a published target
> metric but you could still show that you calculated one and why you think
> it was a good or bad result.
>
>
>
> DSE Mobile
>
>
>
> On Mar 14, 2023, at 6:10 AM, Jim Dirkes via Bldg-sim <
> bldg-sim at lists.onebuilding.org> wrote:
>
> 
>
> Dear Chris,
>
> Kudos for appreciating a gap in your understanding. (I'm in your camp)
>
>
>
> On the other hand, there are SO many variables in building operation that,
> short of a highly instrumented (and carefully calibrated) building for
> everything from lights to people to plug loads to HVAC - calibration is a
> fiction (and I'm confident that no such building exists). Daily calibration
> is a complete fiction, perhaps even a deception. On top of that, a
> "calibrated" model is just a moment in time; everything going forward is
> guaranteed to be different than during the calibration time period.
>
>
>
> I think of "calibration" as more like a sensitivity analysis - determine
> which variables matter more and which matter less. GenOpt works nicely for
> that purpose https://github.com/lbl-srg/GenOpt
> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flbl-srg%2FGenOpt&data=05%7C01%7Cjhaberl%40tamu.edu%7C1b9e306823164bc0353308db24eb12ef%7C68f381e346da47b9ba576f322b8f0da1%7C1%7C0%7C638144364315435766%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=r5pzooBP7ILTcRngSXCtJT7IRkOKdgaBTz%2BlyxeZsaU%3D&reserved=0>
>
>
>
> Jim Dirkes  1631 Acacia Drive NW Grand Rapids, MI 49504
> <https://urldefense.com/v3/__https://www.google.com/maps/search/1631*Acacia*Drive*NW*Grand*Rapids,*MI*49504?entry=gmail&source=g__;KysrKysrKw!!KwNVnqRv!Gyy3l3XXdU0uo6SKEX1uDKDVeqKvb3BM7ELdcAhCyGrNAZnPW1FVt64gzSX8gd4XJkQtPfaNE_z8YgDv9YFrSf_crIA$>
> -  616 450 8653
>
> *Coffee Conversation:*
>
> The "individual" is an impossible concept, conceived by the Enlightenment
> philosophers. It makes no sense to the Christian. In marriages, and
> families, in associations and friendships and religious orders, we are not
> individuals, but a communion of persons.
>
>
>
> ------- Original Message -------
> On Tuesday, March 14th, 2023 at 6:52 AM, Chris Yates via Bldg-sim <
> bldg-sim at lists.onebuilding.org> wrote:
>
> Hi All
>
>
>
> I do find ASHRAE Guideline 14 a little too hardcore for my basic
> understanding of statistics. I can plug any of the equations into Excel,
> but I’ve realised my statistics understanding is very limited! (I’m outed!)
>
>
>
> We don’t actually have to work to G14 in the UK (probably good because my
> copy is a bit old). I finally realised I didn’t know enough after I’d been
> (lazily) using R2 in Excel on some monthly data. I thought that R2 > 0.9
> was generally ok… yeah, it wasn’t.
>
>
>
> So, are there any easy to understand resources available?
>
>
>
> I’ve been messing around with the IMT as well. It’s been fun going back to
> DOS 😊. This got me into daily methods, which leads to my next question.
> Is there any reason why there isn’t a daily calibration option specified in
> G14?
>
>
>
> Many thanks!
>
>
>
> Chris Yates
>
>
>
>
>
> _______________________________________________
> Bldg-sim mailing list
> http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
> <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.onebuilding.org%2Flistinfo.cgi%2Fbldg-sim-onebuilding.org&data=05%7C01%7Cjhaberl%40tamu.edu%7C1b9e306823164bc0353308db24eb12ef%7C68f381e346da47b9ba576f322b8f0da1%7C1%7C0%7C638144364315435766%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IX8Nt8I20rZVVw9YucgeSGLTAZrmk2oxBy8qT3QJHHY%3D&reserved=0>
> To unsubscribe from this mailing list send  a blank message to
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG
>
>
> _______________________________________________
> Bldg-sim mailing list
> http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
> <https://urldefense.com/v3/__http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org__;!!KwNVnqRv!Gyy3l3XXdU0uo6SKEX1uDKDVeqKvb3BM7ELdcAhCyGrNAZnPW1FVt64gzSX8gd4XJkQtPfaNE_z8YgDv9YFr5b8bHHo$>
> To unsubscribe from this mailing list send  a blank message to
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG
>
> _______________________________________________
> Bldg-sim mailing list
> http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
> <https://urldefense.com/v3/__http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org__;!!KwNVnqRv!Gyy3l3XXdU0uo6SKEX1uDKDVeqKvb3BM7ELdcAhCyGrNAZnPW1FVt64gzSX8gd4XJkQtPfaNE_z8YgDv9YFr5b8bHHo$>
> To unsubscribe from this mailing list send  a blank message to
> BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20230314/9df0cba0/attachment.htm>