[Equest-users] [Bldg-sim] Answers Re: Test your knowledge of simulation weather file formats Part 1: the DOE-2 *.BIN/*.BINM format

[Julien Marrec]

Christopher,

This is a jupyter notebook. See http://jupyter.org/.
You can do "pip install jupyter" then start a server with "jupyter
notebook" in a terminal, navigate to the location of the ipynb file and
open it up. Then you can just run it there, cell by cell interactively
(CTRL+ENTER to run a cell).
This is **hugely** helpful in anything that has to do with data analysis,
because it allows for interactive exploring. I strongly suggest you try it
out.
Note that if you have installed a scientific python distro such as
Anaconda, you should already have it installed.

You can also download this as a python file from the notebook, which I've
done out of convenience for you (see attached).

Best,
Julien



--
Julien Marrec, EBCP, BPI MFBA
Owner at EffiBEM <http://www.effibem.com>
T: +33 6 95 14 42 13

LinkedIn (en <https://www.linkedin.com/in/julienmarrec>) *| *(fr
<https://fr.linkedin.com/in/julienmarrec/fr>) :
<http://www.linkedin.com/in/julienmarrec>

2017-12-08 14:26 GMT+01:00 Jones, Christopher <Christopher.r.Jones at wsp.com>:

> Julien,
>
> Thank you for the Python code. One question – how do you download just the
> code without the .png data in the file?
>
>
>
>
>
> *Christopher R. Jones*, P.Eng.
>
> Technical Specialist
>
> Sustainability & Energy
>
>
>
> T +1 416-644-0252 <+1%20416-644-0252>
>
>
>
> 2300 Yonge Street, Suite 2300
> <https://maps.google.com/?q=2300+Yonge+Street,+Suite+2300%0D+Toronto,+ON+M4P+1E4+Canada&entry=gmail&source=g>
>
> Toronto, ON M4P 1E4 Canada
> <https://maps.google.com/?q=2300+Yonge+Street,+Suite+2300%0D+Toronto,+ON+M4P+1E4+Canada&entry=gmail&source=g>
>
> wsp.com
>
>
>
>
>
> *Please consider the environment before printing...*
>
>
>
>
>
> *From:* Bldg-sim [mailto:bldg-sim-bounces at lists.onebuilding.org] *On
> Behalf Of *Julien Marrec via Bldg-sim
> *Sent:* Thursday, December 07, 2017 7:02 AM
> *To:* Joe Huang <yjhuang at whiteboxtechnologies.com>
> *Cc:* EnergyPlus_Support <EnergyPlus_Support at yahoogroups.com>; Javed
> Iqbal <eee.javed at gmail.com>; eQUEST-users at onebuilding.org; BLDG-SIM <
> bldg-sim at lists.onebuilding.org>
> *Subject:* Re: [Bldg-sim] Answers Re: Test your knowledge of simulation
> weather file formats Part 1: the DOE-2 *.BIN/*.BINM format
>
>
>
> Joe,
>
> This was fun, thanks for putting it together, I'm looking forward to the
> next one.
>
> I have to say that I'm impressed by Aaron's knowledge and level of detail,
> and it's not the first time.
>
>
>
> I've used Python to parse your hourly dump and visualize it, as well as
> comparing with the Barrow EPW (converting your hourly dump to SI Units).
> I've posted that on Github, where you can already look at the code + graphs
> since it's in a jupyter notebook, but you could also download it and run it
> on your machine. My hope is that I'll convert one or two members of this
> list to coding :)
>
> For the record, the actual parsing of the hourly dump into a usable format
> (pandas.DataFrame) took me about 5 minutes and 16 lines of code (it was a
> very simple one though).
> https://github.com/jmarrec/BINM_Challenge/blob/master/Analyse_BINM.ipynb
>
>
>
> Cheers,
>
> Julien
>
>
> --
> Julien Marrec, EBCP, BPI MFBA
> Owner at EffiBEM <http://www.effibem.com>
> T: +33 6 95 14 42 13 <06%2095%2014%2042%2013>
>
> LinkedIn (en <https://www.linkedin.com/in/julienmarrec>) *| *(fr
> <https://fr.linkedin.com/in/julienmarrec/fr>) :
>
>
>
> 2017-12-07 10:40 GMT+01:00 Joe Huang <yjhuang at whiteboxtechnologies.com>:
>
> I got a total of five responses,  all of which are attached at the end of
> this e-mail.  Although I was focusing on the FORMAT of the *.BIN weather
> file, most of the respondents focused on the data instead, and to my
> chagrin more intensely than I have, esp. Julien who pointed out an anomaly
> in the direct normal radiation that made me gasp and go back into my file
> and processing procedures.  Kudos to Julien for spotting that, but I'll
> explain what happened there following his e-mail attached below.
>
> The three answers I was seeking are:
>
> (1) *The weather file is for a location above the Arctic Circle*, which
> would make DOE-2.1E crash due to a bug in the shading calculation but not
> in DOE-2.2 which fixed this bug in 2004.  The point here is that this is a
> problem in DOE-2.1E but NOT in the *.BIN format, except it might seem that
> way because the DOE-2 weather packer also crashes because it uses the same
> shading routine to generate the weather statistics.
>
> (2) *The weather file is for a leap year with the output file showing
> Feb. 29th*.  I've heard many people say that DOE-2 weather files contain
> only 365 days, but that's absolutely not true.  The *.BIN format stores
> data for 12 months of 32 days each, or 384 days!  The reason that Feb. 29th
> never shows up in a DOE-2 run is that the developers never bothered to
> reset the February day count to 29 on leap years, even though DOE-2
> calculates when that's the case.  Now that more people are running DOE-2
> with actual historical years, it's well past time for this little fix to be
> implemented.
>
> (3) *The weather file reports shows the weather parameters to an
> additional decimal of precision*, i.e., temperatures are to the 0.1F,
> pressures to 0.01 inches of mercury, solar radiation to 0.1 Btu/sqft, and
> wind speeds to 0.1 mph.  This required a modest change to the BIN format
> that I implemented as the *.BINM (M for Modified) starting in 2011.  This
> leads to what I think is the most fascinating part about the DOE-2 *.BIN
> format that was developed in the early 1980's when computer memory was very
> limited. Members of the original development team (Ender Erdem and maybe
> Fred Buhl) came up with the strategy of "packing" the data by converting
> all data to integers, pack four integers into one big integer, and then
> store them in the file in binary form. By so doing the *.BIN files are only
> 146K (70-80KB zipped), whereas other formats can be well over 1MB (200+KB
> zipped). DOE-2 also uses the *.BIN format to improve execution speed by not
> reading the data an hour at a time or 8760 ASCII reads, but by reading 16
> day chunks at a time or 24 binary reads  for the entire year (which also
> explains why the *.BIN format contains 24x16 or 384 days).
>
> So now let's look at how did our contestants do, listed in order of when I
> received their answers:
>
> Parag -  0 out of 3  (he looked almost entirely on the data, not on the
> format)
>
> Julien -  1 out of 3 (he noticed the leap day, but also focused his
> attention on the data and pointed out two problems that I will address
> following his e-mail  below)
>
> Aaron - 2 out of 3 (I was impressed that he knew about the packing and
> unpacking process, but did not notice the leap year)
>
> Javed -  0 out of 3 (but had a good question on why DNI (Direct Normal) is
> larger than GHI (Global Horizontal) that I will answer following his e-mail
> below)
>
> Nathan - 1 out of 3 (he also noticed the leap day, and had other questions
> that I will answer following his e-mail below).
>
> So, nobody noticed all three answers, but since it's the holiday season, I
> will make them all winners and provide a historical year weather file of
> their choosing. Just e-mail me if you're interested and tell me which one
> you want.
>
> This has been an interesting and insightful experience for me.  I hope
> others also found it entertaining and useful, as well.  I've learned that
> (1) think twice before coming out with a flawed contest rule, (2) look over
> more times whatever I put out on the Web.
>
> (please be sure to read the contestant's submittals and my responses below)
>
> Joe
>
> Joe Huang
>
> White Box Technologies, Inc.
>
> 346 Rheem Blvd., Suite 205A <https://maps.google.com/?q=346+Rheem+Blvd.,+Suite+205A%0D+Moraga+CA+94556&entry=gmail&source=g>
>
> Moraga CA 94556 <https://maps.google.com/?q=346+Rheem+Blvd.,+Suite+205A%0D+Moraga+CA+94556&entry=gmail&source=g>
>
> yjhuang at whiteboxtechnologies.com
>
> http://weather.whiteboxtechnologies.com for simulation-ready weather data
>
> (o) (925)388-0265
>
> (c) (510)928-2683
>
> "building energy simulations at your fingertips"
>
> Attached e-mails  follow in the same order (only final e-mails shown)
> -----------------------------------------------------------
>
> On 12/5/2017 11:24 PM, Parag Rastogi wrote:
>
> Hi Joe,
>
>
>
> Don't know anything about the DOE-2 platform, but here's my take on the meteorological "oddities". Thanks for organising this challenge - it's really cool!
>
>
>
> 1. A large part of the year has zero sun while the other part has sun for 24 hours.
>
> 2. The humidity is _really_ high - the WBT and DBT are mostly within a degree of each other. Don’t know if DOE-2 has a problem with that?
>
> 3. In the summer months, the direct normal is sometimes larger than global solar. That is really odd!
>
> 4. I would not live here - Glasgow winters are bad enough.
>
>
>
> Parag
>
> YJH answer:   Yes, I picked Barrow on purpose to show that the *.BIN
> format still works for places above the Arctic Circle or below the
> Antarctic Circle.  RH is generally very high in frigid climates because the
> cold air cannot hold much moisture (that's why there's always frost in your
> refrigerators!).  The DNI can often be greater than the GHI at low sun
> angles, since the DNI is calculated normal to the sun, while the GHI is
> calculated for a flat horizontal plane.  Yes, stay in Glasgow or at least
> don't go to Barrow in place of Glasgow.
>
> On 12/6/2017 6:35 AM, Julien Marrec wrote:
>
> Hi Joe,
>
>
>
> Replying off the mailing list, since it would kinda beat the point of
> "first 5 to find the error" otherwise.
>
>
>
> 1. There is a 29th of February, but 1912 was indeed a leap year, so I'm
> not sure whether that qualifies for a bug (can't remember how eQuest/DOE
> deal with leap year weather files).
>
>
>
> 2. There is an unsual peak of Global Solar radiation mid November (11 to
> 18th) while the Direct Normal stays at zero. The peak has values at 400
> BTU-HR/SQFT while the max is 250 for the rest of the year. 400 means 1250
> Wh/m2, which is just short of what the Extraterrestrial Direct Normal Solar
> Radiation is.
>
>
>
> In IP units:
>
> [image: Images intégrées 1]
>
>
>
>
>
> I don't see any problem with the Atmospheric pressure, not the relative
> humidity I computed with Db, Wb and Atm. Wetbulb and drybulb are where I
> expect them for this location (checked against the EPW file for the same
> location too).
>
>
>
> So I'm not sure what's the 3rd problem, I'd say that it's probably not a
> good idea to live there unless you *really* like cold weather?
>
>
>
> Best,
>
> Julien
>
> --
>
> Julien Marrec, EBCP, BPI MFBA
> Owner at EffiBEM <http://www.effibem.com>
> T: +33 6 95 14 42 13 <06%2095%2014%2042%2013>
>
> YJH answer: I was embarrassed that you noticed the input echo has 1912
> instead of 2012, which was because DOE-2.1E was not Y2K compliant, so I had
> to set the year to 1912 in order for DOE-2.1E to run.  I was further more
> astonished to see the wierd spike of GHI in November, which wasn't there in
> the actual weather file (see below).
>
>
> Since DOE-2.1E can't run with this weather file, I used a version that I
> modified that took care of the high latitude problem as well as adding the
> extra precision.  When I checked the Fortran source code, I found this
> extra line (<<<)  that I have no memory of ever inserting nor why I did so
> (sound like a politician, don't I ? :-) ).
>       DO 500 I=1,14
>       CALC(I) = FLOAT(IWTH(I))*XMASK(I,2) + XMASK(I,1)
>       IF (I.LE.2) CALC(I)=CALC(I)+IWTH2(I)/10.
>       IF (I.EQ.11.OR.I.EQ.12) CALC(I)=CALC(I)+IWTH2(I-8)/10.   <<< ???
>   500 CONTINUE
> So, if you believe my story, this spike in GHI was the result of my
> unpacking routine and not in the weather file itself.  A curious thing with
> the solar in such Arctic locations is that the GHI is above 0 for all hours
> over two months  (see central bottom of the left plot), indicating  the
> proverbial midnight sun during the summer.
>
> On 12/6/2017 9:05 AM, Aaron Powers wrote:
>
> Hey Joe,
>
>
>
> Here's what I could find.  It's really 2 things since the first 2 and last
> 2 are related.
>
>
>
> 1. Normal DOE2 binary weather files are made up of 24 records of 6208
> bytes each.  So each file is roughly 145 kB.  Your file is 182 kB, so each
> of the 24 records is 7744 bytes in length.  I'm not sure what you have in
> the extra bytes, but the first 6208 bytes of each record is exactly as you
> would expect in a DOE2 packed file.  So I am able to read the .bin file by
> just ignoring the bytes after 6208 of each record.
>
>
>
> 2. You have more precision for each data point.  I'm guessing this must be
> tied up to the extra bytes in each record.  For example, the drybulb,
> wetbulb, and pressure are usually packed into a single integer value.  The
> first and last 4 bits of the integer are junk due to the way Fortran writes
> binary data.  The atmospheric pressure takes up 15 bits and the drybulb and
> wetbulb take up 8 bits each.  After encryption, this allows the drybulb and
> wetbulb to be stored with 0 decimal places and the pressure with 1 decimal
> place.  Similarly, the direct normal and global horizontal are usually
> stored with 0 decimal places.  Your output file shows a higher precision
> for each of these data points.  Again, I'm guessing that this extra
> precision is in the extra bytes in each record.
>
>
>
> 3. Since Barrow is above the Arctic Circle, it gets the midnight sun and
> polar night.
>
>
>
> 4. Related to previous, in the peak of summer, radiation has very little
> diffuse component and during winter is almost all diffuse (when it exists).
>
>
>
> Aaron
>
> YJH answer: You're completely correct in all your suppositions, Sherlock
> Holmes !  What I've done is to add another array of 384, each containing
> the extra precision for the two temperatures, pressure, wind speed, and the
> two solar irradiances.  It was my former colleague at LBNL Ender Erdem who
> assured me that in a binary read, this extra integer would be ignored,
> thereby guaranteeing compatibility between the *.BIN and *.BINM formats.
> However, once the extra read is added, there has to be a flag telling DOE-2
> NOT to do that for the older *.BIN format.  That's the main reason why I
> haven't pushed very hard on  this.
>
> On 12/6/2017 9:18 AM, Javed Iqbal wrote:
>
> Dear Sir Joe,
>
>
>
> I am able to find something though NOT all which looks to me unusual and
> hence reporting here.
>
>
>
> 1) The following screenshot for *22nd April* shows that Direct Normal is
> greater than the Global solar. As per my limited knowledge in this
> direction, usually Direct Normal Solar can be greater than Global solar
> (Direct+diffuse) or GHI during certain periods such as morning or afternoon
> hours. Whereas, the hourly report shows Direct Normal Solar is always
> higher (~8-10 times) than the Global Solar. If this is one of the
> acceptable discrepancy than there are many such instances could be found in
> in the hourly weather file.
>
>
>
> [image: Inline image 1]
>
>
>
> Let me know your thought on this.
>
>
>
> Sorry I could do this much only out of my busy schedule.
>
>
>
> Thanks,
>
> YJH answer:  The DNI can be larger than the GHI frequently in the Arctic
> locations because of persistently low sun angles.  However, the numbers
> that you've highlighted above have DNI/GHI ratios that do seem excessive.
> All these solar irradiances are calculated using several empirical and
> analytical models and so could always be wrong.  However, the formulation
> of the direct solar model calculates the DNI as a sigmoid function of the
> ratio between the GHI and the extraterrestrial global horizontal radiation,
> so it would seem difficult to produce a DNI that would violate physical
> reality, i.e., GHI = DNI*sin(solarZ) + DHI.  Time permitting, I may look
> more into this.
>
> On 12/6/2017 2:01 PM, Nathan Brown wrote:
>
> Ok, so here’s my revised list:
>
>
>
> 1. Hour index is 1, not zero (not different from .epw, but still annoying
> to deal with while post processing)
> 2. Solar noon is not aligned to the noon timestamp. Does this mean
> permanent DST?
> 3. This file contains data for 2/29
>
>
>
>
>
> Nathan Brown, BEMP, LEED AP ~ Associate
>
> *LOISOS *+* UBBELOHDE *
>
> - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> www.coolshadow.com
>
>
>
> YJH answer:  1 is not a correct answer since, as you note, the convention
> is the same for both *.BIN  and *.BINM, as well as *.epw files.  The
> annoyance you feel msy be due to the mixed use of cardinal and ordinal
> numbers in the weather files.  The standard convention on simulation
> weather files is the formal, i.e., Hour 1, Hour 2, etc., whereas the
> convention on weather station reports is the latter, Hour 00:00, Hour
> 01:00, etc.  In my opinion there is an inconsistency on simulation weather
> files where most of the measurements (temperatures, etc.) are reported by
> the timestamp, i.e., 00:00, but incorporated as Hour 1, etc., while
> others,  notably solar and rainfall, are both measured and reported
> cardinally as Hour 1, etc.   I'll talk more about this in the next contest
> on the epw file format.
>
> How are you deducing that solar noon is not aligned to the noon
> timestamp?  As for DST,  I feel that weather files should NEVER incorporate
> any such human-made artifacts.  There are several factors you should
> consider when comparing solar noon to local standard noon: (1) difference
> in longitude between the standard meridian and the location (Barrow); in
> this case, they are 150.00 West and 156.782 West respectively, or a 27
> minute effect,  (2) the solar irradiance is given over the past hour, so
> the average solar position for Hour 12 is at 11:30 AM local time rather
> than 12:00 PM, another 30 minute effect, and (3)  "Equation of time" that
> makes local time faster or slower than solar time by +- 15 minutes at the
> most. I don't know if these effects combined accounts for the difference
> between solar and local noon that you see.
>
>
>
> ------------------------------
>
>
> NOTICE: This communication and any attachments ("this message") may
> contain information which is privileged, confidential, proprietary or
> otherwise subject to restricted disclosure under applicable law. This
> message is for the sole use of the intended recipient(s). Any unauthorized
> use, disclosure, viewing, copying, alteration, dissemination or
> distribution of, or reliance on, this message is strictly prohibited. If
> you have received this message in error, or you are not an authorized or
> intended recipient, please notify the sender immediately by replying to
> this message, delete this message and all copies from your e-mail system
> and destroy any printed copies. You are receiving this communication
> because you are listed as a current WSP contact. Should you have any
> questions regarding WSP's electronic communications policy, please consult
> our Anti-Spam Commitment at www.wsp.com/casl. For any concern or if you
> believe you should not be receiving this message, please forward this
> message to caslcompliance at wsp.com so that we can promptly address your
> request. Note that not all messages sent by WSP qualify as commercial
> electronic messages.
>
> AVIS : Ce message, incluant tout fichier l'accompagnant (« le message »),
> peut contenir des renseignements ou de l'information privilégiés,
> confidentiels, propriétaires ou à divulgation restreinte en vertu de la
> loi. Ce message est destiné à l'usage exclusif du/des destinataire(s)
> voulu(s). Toute utilisation non permise, divulgation, lecture,
> reproduction, modification, diffusion ou distribution est interdite. Si
> vous avez reçu ce message par erreur, ou que vous n'êtes pas un
> destinataire autorisé ou voulu, veuillez en aviser l'expéditeur
> immédiatement et détruire le message et toute copie électronique ou
> imprimée. Vous recevez cette communication car vous faites partie des
> contacts de WSP. Si vous avez des questions concernant la politique de
> communications électroniques de WSP, veuillez consulter notre Engagement
> anti-pourriel au www.wsp.com/lcap. Pour toute question ou si vous croyez
> que vous ne devriez pas recevoir ce message, prière de le transférer au
> conformitelcap at wsp.com afin que nous puissions rapidement traiter votre
> demande. Notez que ce ne sont pas tous les messages transmis par WSP qui
> constituent des messages electroniques commerciaux.
>
>
>
> -LAEmHhHzdJzBlTWfa4Hgs7pbKl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.onebuilding.org/pipermail/equest-users-onebuilding.org/attachments/20171208/3b0615d6/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 25844 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/equest-users-onebuilding.org/attachments/20171208/3b0615d6/attachment-0020.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 7978 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/equest-users-onebuilding.org/attachments/20171208/3b0615d6/attachment-0021.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 8656 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/equest-users-onebuilding.org/attachments/20171208/3b0615d6/attachment-0022.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 130605 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/equest-users-onebuilding.org/attachments/20171208/3b0615d6/attachment-0023.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 73807 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/equest-users-onebuilding.org/attachments/20171208/3b0615d6/attachment-0024.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Analyse_BINM.py
Type: text/x-python
Size: 6696 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/equest-users-onebuilding.org/attachments/20171208/3b0615d6/attachment-0004.py>