[Bldg-sim] Lies, darn lies, and statistics
Jason Glazer
jglazer at gard.com
Tue Mar 14 12:52:34 PDT 2023
As a side note, Dru was probably quoting George Box:
https://en.wikipedia.org/wiki/All_models_are_wrong
Jason
On 3/14/2023 2:12 PM, Jim Dirkes via Bldg-sim wrote:
> I think Dru Crawley coined the phrase, "All models are
> wrong. Some are useful."
>
> Chris, you highlighted an assortment of variables which
> are omnipresent, inconsistent and uncontrollable - so
> what, exactly, does your client expect? Is it a realistic
> expectation? For example, are they going to nail you to
> the wall when, inevitably, you are "wrong" next year?
>
> I have not calibrated many models, but have been made more
> appreciative of all the uncontrollables by the ones I
> calibrated :(. All the statisticians know that there is
> always more than one solution which will result in a high
> R2 value or low CVRSME, so which is correct?
>
> Rather than a calibrated model, lately I've been
> encouraging clients to consider one of the FDD platforms
> on top of their Building Automation System. Spending time
> and money to evaluate whether things are working properly
> makes more sense to me - it's "real life" vs a prediction.
> (Can't forget to mention thoughtful and thorough
> commissioning here; that's essential.)
>
> ps, I love your thoughtful approach. You're setting a
> great example! One aspect of that is to reach out to the
> wider modeling community to gather input and feedback.
>
> Jim Dirkes 1631 Acacia Drive NW Grand Rapids, MI 49504 -
> 616 450 8653
>
> *Coffee Conversation:*
> *The "individual" is an impossible concept, conceived by
> the Enlightenment philosophers. It makes no sense to the
> Christian. In marriages, and families, in associations and
> friendships and religious orders, we are not
> individuals, but a communion of persons.
> *
>
> ------- Original Message -------
> On Tuesday, March 14th, 2023 at 2:10 PM,
> chris.malcolm.yates at gmail.com
> <chris.malcolm.yates at gmail.com> wrote:
>
>> I think I need to qualify this: informed by G14, but
>> definitely not compliant with it! There is some allowance
>> for repairing or “healing” data, but when the data has a
>> lot of holes or modes/ category variables then forget it.
>> This is my case, but the client still wants some kind of
>> representative simulation.
>>
>> Monthly models can be garbage. School holidays cut across
>> months at different times, combined heat and power is
>> popular which complicates gas usage especially when
>> metering is limited… did the heat by-product of
>> electricity generation go to the building, or was it
>> rejected? They can work for heating in our temperate
>> climate, but not for cooling.
>>
>> Nevertheless, we need some kind of representative
>> simulation model. We can’t make any ECM qualifying claims
>> but we can do something useful. This is where these
>> methods can give you a lot of insight before you start
>> modelling.
>>
>> I hadn’t tried the IMT previously. We tend to have
>> limited our regression analysis to monthly “degree day”
>> methods (your 2p model, I think). I plugged some project
>> specific daily electricity data into the MVR example
>> (multi variate regression) and it seemed to give decent
>> CVRMSE (~2-3%) but low R2 (~0.7). However, it appeared to
>> provide some insight on cooling usage (monthly 2p models
>> are meaningless for this in the UK’s temperate climate).
>>
>> I also made some 2p monthly models of gas. I thought
>> these were good until I compared successive years. *I
>> guess this is were understanding a range of statistical
>> indices is helpful.*
>>
>> Here’s the final rub, because the underlying data has so
>> many inconsistencies that can only be made sense of with
>> some regression models, it’s easier to “calibrate” the
>> simulation model to the regression models than the
>> original data. But I may use 2p monthly for gas, daily
>> MVR for electric…
>>
>> So I need to ask if I have wondered completely off-piste
>> with this!
>>
>> Chris
>>
>> *From:*David Eldridge <dancingdavide at hotmail.com>
>> *Sent:* Tuesday, March 14, 2023 12:53 PM
>> *To:* Jim Dirkes <jvdirkes2 at protonmail.com>
>> *Cc:* chris.malcolm.yates at gmail.com; bldg-sim at onebuilding.org
>> *Subject:* Re: [Bldg-sim] Lies, darn lies, and statistics
>>
>> Indirectly there probably isn’t a daily set of metrics in
>> the Guideline since the simulation programs aren’t
>> usually outputting daily results, but there’s no reason
>> there couldn’t be one statistically.
>>
>> You could make one if you had only daily utility data and
>> had to aggregate the simulation results to daily totals,
>> there isn’t a published target metric but you could still
>> show that you calculated one and why you think it was a
>> good or bad result.
>>
>> DSE Mobile
>>
>>
>>
>> On Mar 14, 2023, at 6:10 AM, Jim Dirkes via Bldg-sim
>> <bldg-sim at lists.onebuilding.org> wrote:
>>
>>
>>
>> Dear Chris,
>>
>> Kudos for appreciating a gap in your understanding.
>> (I'm in your camp)
>>
>> On the other hand, there are SO many variables in
>> building operation that, short of a highly
>> instrumented (and carefully calibrated) building for
>> everything from lights to people to plug loads to
>> HVAC - calibration is a fiction (and I'm confident
>> that no such building exists). Daily calibration is a
>> complete fiction, perhaps even a deception. On top of
>> that, a "calibrated" model is just a moment in time;
>> everything going forward is guaranteed to be
>> different than during the calibration time period.
>>
>> I think of "calibration" as more like a sensitivity
>> analysis - determine which variables matter more and
>> which matter less. GenOpt works nicely for that
>> purpose https://github.com/lbl-srg/GenOpt
>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flbl-srg%2FGenOpt&data=05%7C01%7C%7C0d63c1ff284443bc766608db247cc033%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638143890442545264%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hMX7ISkjm6aByq3ZaUXf5p2QKTf3nlOBYbgP7ALodWQ%3D&reserved=0>
>>
>> Jim Dirkes 1631 Acacia Drive NW Grand Rapids, MI
>> 49504 - 616 450 8653
>>
>> *Coffee Conversation:*
>>
>> The "individual" is an impossible concept, conceived
>> by the Enlightenment philosophers. It makes no sense
>> to the Christian. In marriages, and families, in
>> associations and friendships and religious orders, we
>> are not individuals, but a communion of persons.
>>
>> ------- Original Message -------
>> On Tuesday, March 14th, 2023 at 6:52 AM, Chris Yates
>> via Bldg-sim <bldg-sim at lists.onebuilding.org> wrote:
>>
>>
>> Hi All
>>
>> I do find ASHRAE Guideline 14 a little too
>> hardcore for my basic understanding of
>> statistics. I can plug any of the equations into
>> Excel, but I’ve realised my statistics
>> understanding is very limited! (I’m outed!)
>>
>> We don’t actually have to work to G14 in the UK
>> (probably good because my copy is a bit old). I
>> finally realised I didn’t know enough after I’d
>> been (lazily) using R2 in Excel on some monthly
>> data. I thought that R2 > 0.9 was generally ok…
>> yeah, it wasn’t.
>>
>> So, are there any easy to understand resources
>> available?
>>
>> I’ve been messing around with the IMT as well.
>> It’s been fun going back to DOS 😊. This got me
>> into daily methods, which leads to my next
>> question. Is there any reason why there isn’t a
>> daily calibration option specified in G14?
>>
>> Many thanks!
>>
>> Chris Yates
>>
>> _______________________________________________
>> Bldg-sim mailing list
>> http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
>> To unsubscribe from this mailing list send a blank
>> message to BLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG
>>
>
>
> _______________________________________________
> Bldg-sim mailing list
> http://lists.onebuilding.org/listinfo.cgi/bldg-sim-onebuilding.org
> To unsubscribe from this mailing list send a blank message toBLDG-SIM-UNSUBSCRIBE at ONEBUILDING.ORG
--
Jason Glazer, P.E., BEMP, GARD Analytics, 90.1 ECB chair
Admin for onebuilding.org building performance mailing lists
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20230314/6f2ce3dd/attachment.htm>
More information about the Bldg-sim
mailing list