[Bldg-sim] Answers Re: Test your knowledge of simulation weather file formats Part 1: the DOE-2 *.BIN/*.BINM format
Joe Huang
yjhuang at whiteboxtechnologies.com
Thu Dec 7 01:40:31 PST 2017
I got a total of five responses, all of which are attached at the end of this e-mail.
Although I was focusing on the FORMAT of the *.BIN weather file, most of the respondents
focused on the data instead, and to my chagrin more intensely than I have, esp. Julien who
pointed out an anomaly in the direct normal radiation that made me gasp and go back into
my file and processing procedures. Kudos to Julien for spotting that, but I'll explain
what happened there following his e-mail attached below.
The three answers I was seeking are:
(1) *The weather file is for a location above the Arctic Circle*, which would make
DOE-2.1E crash due to a bug in the shading calculation but not in DOE-2.2 which fixed this
bug in 2004. The point here is that this is a problem in DOE-2.1E but NOT in the *.BIN
format, except it might seem that way because the DOE-2 weather packer also crashes
because it uses the same shading routine to generate the weather statistics.
(2) *The weather file is for a leap year with the output file showing Feb. 29th*. I've
heard many people say that DOE-2 weather files contain only 365 days, but that's
absolutely not true. The *.BIN format stores data for 12 months of 32 days each, or 384
days! The reason that Feb. 29th never shows up in a DOE-2 run is that the developers
never bothered to reset the February day count to 29 on leap years, even though DOE-2
calculates when that's the case. Now that more people are running DOE-2 with actual
historical years, it's well past time for this little fix to be implemented.
(3) *The weather file reports shows the weather parameters to an additional decimal of
precision*, i.e., temperatures are to the 0.1F, pressures to 0.01 inches of mercury, solar
radiation to 0.1 Btu/sqft, and wind speeds to 0.1 mph. This required a modest change to
the BIN format that I implemented as the *.BINM (M for Modified) starting in 2011. This
leads to what I think is the most fascinating part about the DOE-2 *.BIN format that was
developed in the early 1980's when computer memory was very limited. Members of the
original development team (Ender Erdem and maybe Fred Buhl) came up with the strategy of
"packing" the data by converting all data to integers, pack four integers into one big
integer, and then store them in the file in binary form. By so doing the *.BIN files are
only 146K (70-80KB zipped), whereas other formats can be well over 1MB (200+KB zipped).
DOE-2 also uses the *.BIN format to improve execution speed by not reading the data an
hour at a time or 8760 ASCII reads, but by reading 16 day chunks at a time or 24 binary
reads for the entire year (which also explains why the *.BIN format contains 24x16 or 384
days).
So now let's look at how did our contestants do, listed in order of when I received their
answers:
Parag - 0 out of 3 (he looked almost entirely on the data, not on the format)
Julien - 1 out of 3 (he noticed the leap day, but also focused his attention on the data
and pointed out two problems that I will address following his e-mail below)
Aaron - 2 out of 3 (I was impressed that he knew about the packing and unpacking process,
but did not notice the leap year)
Javed - 0 out of 3 (but had a good question on why DNI (Direct Normal) is larger than GHI
(Global Horizontal) that I will answer following his e-mail below)
Nathan - 1 out of 3 (he also noticed the leap day, and had other questions that I will
answer following his e-mail below).
So, nobody noticed all three answers, but since it's the holiday season, I will make them
all winners and provide a historical year weather file of their choosing. Just e-mail me
if you're interested and tell me which one you want.
This has been an interesting and insightful experience for me. I hope others also found it
entertaining and useful, as well. I've learned that (1) think twice before coming out with
a flawed contest rule, (2) look over more times whatever I put out on the Web.
(please be sure to read the contestant's submittals and my responses below)
Joe
Joe Huang
White Box Technologies, Inc.
346 Rheem Blvd., Suite 205A
Moraga CA 94556
yjhuang at whiteboxtechnologies.com
http://weather.whiteboxtechnologies.com for simulation-ready weather data
(o) (925)388-0265
(c) (510)928-2683
"building energy simulations at your fingertips"
Attached e-mails follow in the same order (only final e-mails shown)
-----------------------------------------------------------
On 12/5/2017 11:24 PM, Parag Rastogi wrote:
> Hi Joe,
>
> Don't know anything about the DOE-2 platform, but here's my take on the meteorological "oddities". Thanks for organising this challenge - it's really cool!
>
> 1. A large part of the year has zero sun while the other part has sun for 24 hours.
> 2. The humidity is _really_ high - the WBT and DBT are mostly within a degree of each other. Don’t know if DOE-2 has a problem with that?
> 3. In the summer months, the direct normal is sometimes larger than global solar. That is really odd!
> 4. I would not live here - Glasgow winters are bad enough.
>
> Parag
YJH answer: Yes, I picked Barrow on purpose to show that the *.BIN format still works
for places above the Arctic Circle or below the Antarctic Circle. RH is generally very
high in frigid climates because the cold air cannot hold much moisture (that's why there's
always frost in your refrigerators!). The DNI can often be greater than the GHI at low
sun angles, since the DNI is calculated normal to the sun, while the GHI is calculated for
a flat horizontal plane. Yes, stay in Glasgow or at least don't go to Barrow in place of
Glasgow.
On 12/6/2017 6:35 AM, Julien Marrec wrote:
> Hi Joe,
>
> Replying off the mailing list, since it would kinda beat the point of "first 5 to find
> the error" otherwise.
>
> 1. There is a 29th of February, but 1912 was indeed a leap year, so I'm not sure whether
> that qualifies for a bug (can't remember how eQuest/DOE deal with leap year weather files).
>
> 2. There is an unsual peak of Global Solar radiation mid November (11 to 18th) while the
> Direct Normal stays at zero. The peak has values at 400 BTU-HR/SQFT while the max is 250
> for the rest of the year. 400 means 1250 Wh/m2, which is just short of what the
> Extraterrestrial Direct Normal Solar Radiation is.
>
> In IP units:
> Images intégrées 1
>
>
> I don't see any problem with the Atmospheric pressure, not the relative humidity I
> computed with Db, Wb and Atm. Wetbulb and drybulb are where I expect them for this
> location (checked against the EPW file for the same location too).
>
> So I'm not sure what's the 3rd problem, I'd say that it's probably not a good idea to
> live there unless you *really* like cold weather?
>
> Best,
> Julien
> --
> Julien Marrec, EBCP, BPI MFBA
> Owner at EffiBEM <http://www.effibem.com>
> T: +33 6 95 14 42 13
YJH answer: I was embarrassed that you noticed the input echo has 1912 instead of 2012,
which was because DOE-2.1E was not Y2K compliant, so I had to set the year to 1912 in
order for DOE-2.1E to run. I was further more astonished to see the wierd spike of GHI in
November, which wasn't there in the actual weather file (see below).
Since DOE-2.1E can't run with this weather file, I used a version that I modified that
took care of the high latitude problem as well as adding the extra precision. When I
checked the Fortran source code, I found this extra line (<<<) that I have no memory of
ever inserting nor why I did so (sound like a politician, don't I ? :-) ).
DO 500 I=1,14
CALC(I) = FLOAT(IWTH(I))*XMASK(I,2) + XMASK(I,1)
IF (I.LE.2) CALC(I)=CALC(I)+IWTH2(I)/10.
IF (I.EQ.11.OR.I.EQ.12) CALC(I)=CALC(I)+IWTH2(I-8)/10. <<< ???
500 CONTINUE
So, if you believe my story, this spike in GHI was the result of my unpacking routine and
not in the weather file itself. A curious thing with the solar in such Arctic locations
is that the GHI is above 0 for all hours over two months (see central bottom of the left
plot), indicating the proverbial midnight sun during the summer.
On 12/6/2017 9:05 AM, Aaron Powers wrote:
> Hey Joe,
>
> Here's what I could find. It's really 2 things since the first 2 and last 2 are related.
>
> 1. Normal DOE2 binary weather files are made up of 24 records of 6208 bytes each. So
> each file is roughly 145 kB. Your file is 182 kB, so each of the 24 records is 7744
> bytes in length. I'm not sure what you have in the extra bytes, but the first 6208
> bytes of each record is exactly as you would expect in a DOE2 packed file. So I am able
> to read the .bin file by just ignoring the bytes after 6208 of each record.
>
> 2. You have more precision for each data point. I'm guessing this must be tied up to
> the extra bytes in each record. For example, the drybulb, wetbulb, and pressure are
> usually packed into a single integer value. The first and last 4 bits of the integer
> are junk due to the way Fortran writes binary data. The atmospheric pressure takes up
> 15 bits and the drybulb and wetbulb take up 8 bits each. After encryption, this allows
> the drybulb and wetbulb to be stored with 0 decimal places and the pressure with 1
> decimal place. Similarly, the direct normal and global horizontal are usually stored
> with 0 decimal places. Your output file shows a higher precision for each of these data
> points. Again, I'm guessing that this extra precision is in the extra bytes in each record.
>
> 3. Since Barrow is above the Arctic Circle, it gets the midnight sun and polar night.
>
> 4. Related to previous, in the peak of summer, radiation has very little diffuse
> component and during winter is almost all diffuse (when it exists).
>
> Aaron
YJH answer: You're completely correct in all your suppositions, Sherlock Holmes ! What
I've done is to add another array of 384, each containing the extra precision for the two
temperatures, pressure, wind speed, and the two solar irradiances. It was my former
colleague at LBNL Ender Erdem who assured me that in a binary read, this extra integer
would be ignored, thereby guaranteeing compatibility between the *.BIN and *.BINM formats.
However, once the extra read is added, there has to be a flag telling DOE-2 NOT to do that
for the older *.BIN format. That's the main reason why I haven't pushed very hard on this.
On 12/6/2017 9:18 AM, Javed Iqbal wrote:
> Dear Sir Joe,
>
> I am able to find something though NOT all which looks to me unusual and hence reporting
> here.
>
> 1) The following screenshot for *_22nd April_* shows that Direct Normal is greater than
> the Global solar. As per my limited knowledge in this direction, usually Direct Normal
> Solar can be greater than Global solar (Direct+diffuse) or GHI during certain periods
> such as morning or afternoon hours. Whereas, the hourly report shows Direct Normal Solar
> is always higher (~8-10 times) than the Global Solar. If this is one of the acceptable
> discrepancy than there are many such instances could be found in in the hourly weather
> file.
>
> Inline image 1
>
> Let me know your thought on this.
>
> Sorry I could do this much only out of my busy schedule.
>
> Thanks,
YJH answer: The DNI can be larger than the GHI frequently in the Arctic locations because
of persistently low sun angles. However, the numbers that you've highlighted above have
DNI/GHI ratios that do seem excessive. All these solar irradiances are calculated using
several empirical and analytical models and so could always be wrong. However, the
formulation of the direct solar model calculates the DNI as a sigmoid function of the
ratio between the GHI and the extraterrestrial global horizontal radiation, so it would
seem difficult to produce a DNI that would violate physical reality, i.e., GHI =
DNI*sin(solarZ) + DHI. Time permitting, I may look more into this.
On 12/6/2017 2:01 PM, Nathan Brown wrote:
> Ok, so here’s my revised list:
>
> 1. Hour index is 1, not zero (not different from .epw, but still annoying to deal with
> while post processing)
> 2. Solar noon is not aligned to the noon timestamp. Does this mean permanent DST?
> 3. This file contains data for 2/29
>
>
> Nathan Brown, BEMP, LEED AP ~ Associate
> LOISOS + UBBELOHDE
> - - - - - - - - - - - - - - - - - - - - - - - - - - -
> www.coolshadow.com <http://www.coolshadow.com/>
>
YJH answer: 1 is not a correct answer since, as you note, the convention is the same for
both *.BIN and *.BINM, as well as *.epw files. The annoyance you feel msy be due to the
mixed use of cardinal and ordinal numbers in the weather files. The standard convention
on simulation weather files is the formal, i.e., Hour 1, Hour 2, etc., whereas the
convention on weather station reports is the latter, Hour 00:00, Hour 01:00, etc. In my
opinion there is an inconsistency on simulation weather files where most of the
measurements (temperatures, etc.) are reported by the timestamp, i.e., 00:00, but
incorporated as Hour 1, etc., while others, notably solar and rainfall, are both measured
and reported cardinally as Hour 1, etc. I'll talk more about this in the next contest on
the epw file format.
How are you deducing that solar noon is not aligned to the noon timestamp? As for DST, I
feel that weather files should NEVER incorporate any such human-made artifacts. There are
several factors you should consider when comparing solar noon to local standard noon: (1)
difference in longitude between the standard meridian and the location (Barrow); in this
case, they are 150.00 West and 156.782 West respectively, or a 27 minute effect, (2) the
solar irradiance is given over the past hour, so the average solar position for Hour 12 is
at 11:30 AM local time rather than 12:00 PM, another 30 minute effect, and (3) "Equation
of time" that makes local time faster or slower than solar time by +- 15 minutes at the
most. I don't know if these effects combined accounts for the difference between solar and
local noon that you see.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20171207/9a0f726e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 73807 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20171207/9a0f726e/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: holbnogkefilobad.png
Type: image/png
Size: 8656 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20171207/9a0f726e/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bnekkanonlhggemb.png
Type: image/png
Size: 7978 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20171207/9a0f726e/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 38847 bytes
Desc: not available
URL: <http://lists.onebuilding.org/pipermail/bldg-sim-onebuilding.org/attachments/20171207/9a0f726e/attachment-0003.png>
More information about the Bldg-sim
mailing list