model

The Engerer2 Diffuse Fraction Model (Global Radiation Separation Model)


August 2016 Update: You can learn how to compute the Engerer2 model in my Rpackage here!

This blog post explores the success of the Engerer 2 model as laid out in: Engerer, Nicholas A.  Minute resolution estimates of the diffuse fraction of global irradiance for southeastern Australia. Solar Energy. 116, June 2015, pp. 215-237.

[download the paper here]


One of the key outcomes from my PhD thesis was the validation of two different types of radiation models: clear sky and separation, against one minute resolution data. For the clear sky validation, I found suitable performance from several models for use in Australia, but the available separation models, however, did not have acceptable performance

The main issue with the available separation models (models that take a global radiation measurement from a pyranometer and separate it into its direct and diffuse components), is that they are regression based, with the original data being hourly averages of radiation.  At minute-resolution timescales, the relationship with global and diffuse/direct radiation is very different.  For one, there are very rapid fluctuations in the incoming radiation budget across these timescales.  Another big difference is the influence of cloud enhancement which is where radiation arriving at the surface exceeds the clear sky value because of non-linear interactions with some types of cloud decks. 

The Engerer2 model, with the diffuse fraction (Kd), plotted against the clearness index (Kt)

Making big changes, in the name of science

Thus, when I formulated my model, I knew that I had to make some significant advancements upon the existing methods/literature.  The principal improvements made with this model are four-fold:

  1. Inclusion of a physical model (REST2 clear sky model), making the model 'quasi-physical', much like the DISC model written by Eugene Maxwell (Maxwell 1987)
  2. The model is the only one of its class (as of the time of publication) that has been fit to minute resolution data (most other models have been designed for hourly data)
  3. There are two new variables, which have not previously been utilised in a separation model.  These are delta_Ktc (deviation of observed clearness index from clear sky value of clearness index)
  4. and K_de (the portion of the diffuse fraction that is attributable to cloud enhancement events)
The Engerer2 model formulation.  Please  read the paper  for more information.

The Engerer2 model formulation.  Please read the paper for more information.

independent assessment of the model: it works very, very well

The result is an impressive performance of the model against the current suite representing the state-of-the-art.  In a recent study, Gueymard and Ruiz-Arias 2015, radiation data from 54 sites around the globe were used to validate 140 separation models.  In this study, the Engerer2 model was the best!  Here it is, as described in the text:

“It is found that two models stand out over the arid, temperate and tropical climate zones: ENGERER2 and PEREZ2. These two models share two important features: (i) They include a variability predictor; and (iii) They leverage clear-sky irradiance estimates. The reason why ENGERER2 performs consistently better than PEREZ2 or other models is most likely because it was actually derived from 1-min data (compared to hourly data for PEREZ2 or most other models tested here). Based on the ensemble of statistical results obtained here, it is concluded that ENGERER2 has the best generalization skill, and can thus be considered a ‘‘quasi-universal” 1-min separation model, wherever and whenever low-albedo conditions prevail.”
Figure 2 in Gueymard & Ruiz-Arias 2015. Displaying the stations at which radiation data was used for model evaluation.

Figure 2 in Gueymard & Ruiz-Arias 2015. Displaying the stations at which radiation data was used for model evaluation.

And well... was that ever quite the compliment (especially coming from a scientist whom I've looked up to for so long)!  I am very pleased with this result, because now, my Engerer2 model is the ‘'quasi-universal' 1-min separation model" and has been accepted to be of global standard.  That makes my inner nerd quite happy, I'll admit. "Chuffed" as my Aussie friends would say :-).  And now I've have been given a reason to write a long overdue blog post about this research work.  As well as deliver a little surprise...

NOw, the Engerer2 model is in demand

As a result of this excellent outcome, I have several researchers in the community who would like to use my model, and I am quite happy to oblige.  So with this post, I am also announcing that a beta version of my Rpackage "anusolar" is now available, on request.  You can read more about this software at nickengerer.org/rpackage and where you can find out how to use the Engerer 2 separation model!  This package will allow you to do more than that, including PV simulation, KPV calculation and creating output from clear-sky radiation models.  So go check it out!

#MODSIM2015 - Extracting Solar Radiation Measurements from PV System Power Output

 A bit of background info on my 21st International Congress on Modelling and Simulation (MODSIM2015) paper

[download the paper]

#MOSIM2015 is underway! And lucky me, I’m here at our wonderful sunny Gold Coast location awaiting my opportunity to present an innovative modeling approach which estimates solar radiation using PV system power output.

Without further delay, let’s dig right in...

The Premise

A pyranometer!

A pyranometer!

Here’s the premise:  photovoltaic solar panels are not altogether different from pyranometers.  In most cases, pyranometers – the global ‘go-to’ solar radiation sensor – use a silicon wafer to measure the global horizontal irradiance at a given location.  Interestingly, this silicon wafer is based on the same technology as most PV modules.  The result is that the power output from a PV system has a first order relationship with the global solar radiation arriving in the plane-of-array.  As a scientist, this creates a sense of curiousity within me!  I wonder, can we use the power output from a PV system to then work our way back to a standard radiation measurement?

The Hypothesis

Let’s put that pondering within the framework of the scientific method, forming a hypothesis:

“Solar panels are not altogether different from pyranometers, thus one should be able to use their power output as an input to a ‘separation’ model to estimate the diffuse and beam components of solar radiation”

What is a separation model?

Over the past several decades, there have been many (as in hundreds) of models developed for the purpose of separating the diffuse and direct (beam) components from a global horizontal irradiance measurement.  In my recent paper (link), I describe this more thoroughly, so dig into that if you’re interested in learning more (or if you are a bit lost on the above bold terms)

The Engerer 2 model

The Engerer 2 model

What I’ve done with my MODSIM2015 paper, is use my own separation model format – the “Engerer 2” model - and modified it to accept PV power output measurements as the primary input, in place of pyranometer measurements.  The overall goal here, is to estimate the diffuse component of radiation using only the measured PV power output through this model and then test how well it performed.

In this paper, we use a scientific approach to test this.  Two different model formats were tested.  One with a single input, and another with four inputs.  We compare a standard pyranometer based model (Kt), fit to the same data, to the PV power output based model (KPV).  This allows us to directly compare the two approaches in a controlled, scientific fashion. 

Now, in order to do this, we need data.  So, I turned to my usual sources of data, PVOutput.org and the Bureau of Meteorology solar radiation monitoring sites.  Fortunately, I was able to identify 18 PV sites in Adelaide and Melbourne which were of very good quality and located within 10km of a solar radiation monitoring site. 

Next, I processed these data through quality control algorithms (read more in my PhD thesis) and then averaged them to hourly values.  With this QC complete, we’re able to start fitting and testing the models. 

There are a lot of details here, which I’ll glaze over in this blog post, but you can review in our paper.  Such as what QC methods I used, how I chose the PV sites, etc, etc.  I’ll leave it to you to investigate this further!  I’d rather spend my time here talking about what we found, because it is pretty cool.

The Results

First, in fitting and testing the single parameter model, the general approach undertaken was validatedThe idea works.  The relationship used in my diffuse fraction model (Engerer 2) relies on a generalized logistic function based on the relationship between the Diffuse Fraction (Kd) and the Clearness Index (Kt).  However, in the PV power output based model, we use KPV in place of Kt,  KPV is the clear-sky index for photovoltaics, and allows for the removal of all the individual nuances from the PV power output time series (use the link above to learn more)

This is a great start, but the results for a single input model aren’t very good.  This is because the model is too simple for the complex relationship between radiation and cloud cover.  But never fear, that’s where the Engerer 2 model comes in.

For those of you who are not familiar with this field, it's worth mentioning that the Engerer 2 model was recently declared the "quasi-universal" separation model in a global study using 54 sites to test 140 different models in a study by Chris Gueymard, a leader in the solar radiation modelling field.  So we're taking a modern approach to this problem.

Herein, we use the Engerer 2 model format to further test our hypothesis.  Except for the solar PV power output based model, we change another one of the input variables so that it is based on PV power output instead of a pyranometer measurement. This is the deviation from clear-sky variable (read more in the paper).

Kt versus KPV based modelling within the Engerer 2 model framework. 

Kt versus KPV based modelling within the Engerer 2 model framework. 

The results were very encouraging! The figure above demonstrates that by using only the PV power output, we are able to use our generalised logistic function based model to estimate the diffuse fraction of solar radiation with only a 5% increase in rRMSE and 0.7% increase in rMBE over a pyranometer based model.  Moreover, the observed relative errors are within the ‘good’ modeling requirements established by Gueymard and Meyers 2008.

Great, now what?

Moving forward with this result, there is now more work to do! For example, with a pyranometer measurement, once we have the diffuse component, we get the direct component through the closure equation. With PV system power output, this isn't possible.   So we'll need to test our ability to do extract the beam component further.  Possibly, this might require the joint development of another model to extract it

Despite the further work required, the results herein remain exciting. The approach used in our paper show that extracting solar radiation measurements from PV system power output can be done. This is a world first for such a result, and it is exciting to know that further work in this space is realistic. 

I am very hopeful that this result will encourage others in the field to further explore this conceptual work.  If we can extract solar radiation measurements from PV power output, we suddenly can create a very rich dataset of radiation measurements globally, greatly augmenting our ability to anlayse the spatio-temporal evolution of incoming solar radiation.  Such a dataset can assist in the validation of satellite measurements or global climate/meteorological modeling tools. 

So, with this very awesome scientific finding, I’ll leave you with the above thoughts, and hope that you’ll go on to read the paper, cite the results and further develop the prospect of having millions of additional solar radiation sensors globally.

[download the paper]

[download the presentation]

View the presentation below:


What's Happening at #MODSIM2015

Find Stuff!


PhD Thesis: What I did, What I found, Why it Matters

Several years ago, I decided I wanted to write a Masters Thesis that didn't just sit on a shelf, collecting dust.  Call me idealistic, but I wanted to adopt a philosophy of "science off the shelf" 

[April 2016 Update: My PhD Thesis is now finalised! Download it here: Part 1  |  Part 2 ]
 


[download my submitted thesis - part 1]

[download my submitted thesis - part 2]

Back at the University of Oklahoma School of Meteorology, there was a running joke amongst graduate students that they could stick a $20 bill in their thesis, check back in 5-10 years, and find it still nestled between the pages, safe and sound.  While it was humorous, and mostly in jest, it reflected the mentality of postgraduate education/research work - what we were doing was mostly just to tick boxes & get our degree so that we could one day do something that mattered. In reality, our 'research' would sit on a library shelf, acquiring dust.

Yearly kWh production for a simulated 2kW Sanyo array in Oklahoma

Yearly kWh production for a simulated 2kW Sanyo array in Oklahoma

But I wasn't content to resign myself to this fate. I felt compelled to produce something that mattered, that would have real-world relevance.  So I brought on an industry partner, a local solar installer in Oklahoma, and put together a solar map for Oklahoma, not based on solar radiation - but on kWh hours of electricity generation, which immediately translates to payoff times and dollars. [download my M.S. Thesis]

It was a tremendously successful experience, and as I moved forward into my PhD project, I was very determined to not only maintain that approach, but to expand it.  Now, looking back, I can see that approach has paid off in great ways, which I'd like to share with you.  Because what I've discovered, what I've developed and what I am now positioned to do with the technology I created, is all highly relevant, commercially viable and ready to hit the ground running.

What I did and why

Australian PV installations by year, as provided by the Australian PV Institute

There is ~4GW of solar energy installed in Australia, which is dominated by ~1.4 million small-scale photovoltaic (PV) arrays.  These arrays are relatively small (average size 1.5kW in 2011, growing to 4.5kW by the end of 2014), and the vast majority of these installations are un-monitored.  This means that their minute-by-minute performance is not recorded.  In fact, the only information collected for most of these systems is the total quarterly production as reported on electricity bills.  Long story short, this lack of information limits the number of PV systems that can be installed in a given region (like here).

Measured PV power output, divided by that PV systems clear sky power output

Measured PV power output, divided by that PV systems clear sky power output

So what did I do about it? I developed a method for estimating the power output from many thousands of PV systems using a small selection of monitored PV systems (which report their minute-by-minute or hour-by-hour generation).  This method is called  “the clear-sky index for photovoltaics”, KPV. [read more]

Along the way, I had to do significant work in the field of solar radiation modelling, validating clear sky radiation models and developing a new type of solar radiation model fit to Australian radiation data (a "separation model").  I also had to show that my new method (using solar panels as a sensor network) was able to do the job just as well as professional grade radiation equipment.  

Pyranometer based methods versus KPV based methods.  The takeaway? Get the sites close enough (within 5km) and my new method is just as good as using a more sparse network of professional grade solar radiation sensors.  

It was also necessary to develop new quality control routines for the PV system power output data.  This type of data is messy, often provided by non-experts and hasn't been used in the way I've proposed before.  This was accomplished through the development of a new quality control routine called QCPV (now going through the review process in a major journal).

~200 solar PV systems installed in Canberra, which are reporting their data in real-time

Once I sorted out the quality control work, it became possible to work with data from many hundreds of solar PV systems, as well as scale my KPV method up to many thousands of systems.  So I forged ahead, using Canberra as a proof-of-concept, creating a city-wide distributed PV simulation of its 12,000+ embedded PV generators (based on December 2012 installation data).  

I then paired this simulation system with weather events that cause broad-scale, rapid changes in the power output of all of the PV systems at the same time [check it out].  It is these types of events which are the most likely to cause future grid stability problems.  The basic idea is that, when you have a wide-spread solar network, the negative effects of partly cloudy days are "smoothed out" by those systems being positioned over broad region, but during certain weather events, it is not possible to smooth out these impacts, because the cloud features are too widespread, sudden and thick.

"when you have a wide-spread solar network, the effects of partly cloudy days are "smoothed out" by those systems being positioned over large region, but during certain weather events, it is not possible to smooth out these impacts, because the cloud features are too widespread, sudden and thick"

What I did find?

Let me keep this as focused and brief as possible...

KPV estimates (color) versus measurements (black) under a positive ramp event

Firstly, I demonstrated that my newly proposed KPV method was much better than existing methods, showing that it was well-behaved under all cloud cover conditions, and performed well under positive and negative ramp events. [download publication][read blog post]

Next I found that for clear sky radiation models, operating in Australia, global clear sky simulations, are best computed by the Solis, Esra and REST2 approaches, while the Iqbal, Esra and REST2 methods are the most proficient clear sky beam models. [download publication][read blog post]

The Engerer 2 separation model at work (blue model estimates, grey observations)

 After that, I found that only the Perez separation model performed satisfactorily for high resolution (one minute) solar radiation data.  In response to this, I developed three new separation models, which gave slight improvements over the Perez model and greatly exceeded the performance of all other existing model techniques. [download publication]

Once that was handled, I compared radiation sensor based methods to my PV data based approaches, with a student project.  This study found that the approaches were equally as good for separation distances of 5km or less. Given that PV sensors are "cheap" (someone else pays for them) - this was a great finding. [download publication][read blog post]

Post QC KPV estimates, very tight correlation, great results

Then I dug into the development of the QCPV algorithm (quality control), demonstrating that the method I created can result in a 43% reduction in Mean Absolute Percent Error (MAPE) over the raw data. [pre-print coming soon]

Second to last, with another student project, we categorised the weather events that cause those large scale, collective changes in PV power output discussed earlier.  Positive collective ramp events (sudden clearing) were caused by Australian northwest cloud bands and radiation fog dissipation. Negative collective ramp (sudden cloud cover arrival) events were caused most frequently by the passage of cold fronts and thunderstorms.[download manuscript][read blog post]

Finally, I put it all together, with the city-wide PV simulation system, using it to simulate the changes in total power and energy output from these collective ramp events.  I was able, for the first time, to quantify (aka determine a representative number) the amount of power that (dis)appears on the electrical grid during these events.  For example, a thunderstorm event on 19 February 2014 removed 20.78 MW of power generation from the local grid over an 85 minute period, which equates to approximately 14.54 MWh of energy generation forgone over that period.  That's probably enough to change prices on the energy market - not very much, but as the solar installation numbers continue to grow, that influence will grow significantly.  

Here's the thunderstorm ramp event from 19 February 2014, along with some satellite imagery.

What is my overall conclusion?

The overall conclusion, is that the developed regional simulation system for distributed solar PV, made possible by an upscaling of my KPV methodology, represents a significant, unique and promising tool for scientific, engineering and operational purposes.  

In the simplest of terms: I built a very handy tool, with cheap inputs that can be run anywhere that solar PV systems are reporting their power output data.

Where next?

I have a full-time lecturer position ("professor" in the American use of the word) at The Australian National University, where I work in the Fenner School of Environment and Society (employed since July 2013).  I am using the freedom and security this position provides me with, to apply for funding to scale this simulation system up, Australia-wide.  I'll join it with the new Himawari 8/9 satellite data, and pair up with the energy market/utilities in Australia, in order to help large amounts of distributed solar to be added to the grid.  

You could say the future is sunny and bright (#punny).  And with HUGE amounts of solar being installed globally, the solar century is before us.  There are plenty of opportunities for this science to stay off of that dust shelf.  So I'd say, overall, this whole PhD thing has been a smashing success! Even if it was a bit of a wild ride.  I hope to have more exciting news soon - for now, I'll get back to working getting this simulator to run real-time in Canberra...

Solar 2014: Which Clear-Sky Radiation Model is Best for Use in Australia?


Update 2015: This research is now available as a paper in Solar Energy journal!

[download paper]

Citation: Engerer, Nicholas A and Franklin P. Mills. Validating Nine Clear Sky Radiation Models in Australia. Solar Energy. 120, October 2015, pp. 9-24.


Have you ever wondered which clear-sky solar radiation model you should use in your research project or solar energy simulation?  When I was designing my KPV method for estimating PV system power output I needed to figure out which clear-sky model would be the best one to use.  But there was a problem - I couldn't find a single validation study for clear-sky radiation modelling for Australia!

So in the paper, I had to do a quick model validation using one year of radiation data from Wagga Wagga, from which I decided on the Esra model.  But that simple validation left me wondering which model REALLY was the best?  

Thus I embarked on a scientific journey to discover which model was the best for use in Australia using the solar radiation data from 14 sites in Australia:

 

First, I set a few ground rules.  I wasn't going to use any radiation model that was overly complicated, nor was I going to use atmospheric variables that were difficult to obtain.

This meant using climatological values for input values such as the Linke Turbidity coefficient or ozone content - rather than using direct measurements from a photometer (because who honestly has spectral data?).  I think this is important, because a validation study should focus on models that are widely applicable so that it is widely useful.

In the end, I chose nine models from the options from both beam (direct) and global radiation: 

In the many Australian presentations and publications I've read and attended over the past few years, the most common clear-sky radiation model used is the Ineichen-Perez model.  

However, my research shows that the Ineichen-Perez model is not even in the top-three best choices.  So we really shouldn't be using it in our research as it is introducing unnecessary errors into our collective research knowledge.

What models are the best to use?  For the beam models, the top three choices are the Iqbal-C, Esra and REST2 models.  And for the global models, they are the Solis, Esra and REST2 models.

If we were to chose the best overall model, that would be the Esra model.  Which edges out the REST2 model, due to is large errors at high zenith angles.

You can read more about this in my Solar 2014 Poster Presentation [direct download] and the hopefully an upcoming article in Solar Energy journal (fingers crossed).

Until then, the quick answer is...

The best clear-sky radiation model for use in Australia is the Esra model!

 
/*