Launching Our Grid-based Solar Forecasting API at #CIDER17


Today, I’m presenting at the #CIDER17 conference in Sydney.

This conference, is focused on DER integration with networks in the Asia-Pacific, and nowhere is this more relevant than here in Australia.  In my presentation, I am launching of our grid-based solar forecasting API, which has now gone live in beta with a total of 8 Australian distribution networks now available (3 more added soon...).

This API deployment is a part of our ARENA funded distributed solar forecasting and modelling project where we are tasked with the integration solar forecasts into LV network operations.  Here, we have two key outcomes. First, to make accessing solar forecasts easy, scalable and to directly integrate them with the existing software in use by the DNSPs.  And second, raise the allowable penetrations of solar PV in these networks through #orchestration and proactive (rather than reactive) LV network management.

Integration via an API Framework

With that first outcome in mind, we have chosen an API framework for the delivery of our solar forecasting service.  This API is straightforward and built to start the integration process. Data from the API can be previewed in an HTML page for ease of use or testing, but can also be downloaded in specific data formats including JSON, XML and CSV via your own scripts.  Over the next year, we’ll be focusing in on building integrations for your load flow & network modelling software of choice – so be sure to provide us your feedback on what development you’d like to see in this space.

Getting to the grid-based API specifics, lets start off with “the how” we create the forecasting aggregations exposed in this API service.  Using the installed DER metadata for small-scale solar PV sites, provided by DNSPs, we have been able to:

  1. Generate PV power output forecasts & estimated actuals by matching the radiation estimates from our satellite nowcasting system to the locations and characteristics of the installed PV sites
  2. We then aggregate the PV power output forecasts to the upstream distribution transformers and zone substations; providing the total PV power output forecast for those networks assets
  3. Finally, we expose this forecast data via an API framework, which currently makes these aggregations available the zone substation level

Once these aggregations are completed, participating DNSPs are able to retrieve forecasts (as well as estimated actuals, more on that soon) via a wide variety of methods and file formats (as discussed above).  DNSPs first must register for the Solcast API services, and then are granted access to these aggregations for their network via their unique API key. We have instructions available on how to retrieve these forecasts & use the API, which is available on request (just send us an email!).

At this time, we have a total of 3 API endpoints unique to each DNSP:

  1. The ability to retrieve the API’s list of zone substation names available for forecast retrieval. This makes it easy to access the next two endpoints…
  2. An endpoint which provides a 0-7 day forecast, at 30 minute intervals (updated behind the scenes every 10 minutes) for the specified zone substation
  3. An endpoint which allows the DNSP to retrieve the estimated actuals for the downstream PV sites at that asset.  The estimated actuals are a look back at the past 7 days, at 30 minute intervals using the valid satellite scan from that time [no forecasting involved] (Note: this will be expanded to reach back further soon!)

#CIDER17 Demo of the API:

In the interest of showing off the capabilities of this API system, I am attaching the following screenshots and images of the API in actions. Several of these images are from forecasts & estimated actuals retrieved just prior to this talk, the remainder are from other events of interest. 

(Follow-up to Solar Analytics talk yesterday - ANU solar PV Simulations from 2014....)


Help Us Improve: Our Priority Here is DNSP Testing & Feedback

At this stage, we are asking DNSPs to access, analyse and critically review this API offering.  We want to know how they will use it, what features they’d like to see added and where we can integrate with existing software and operational strategies.

Looking Ahead: LV Network Orchestration

Getting back to outcome #2, raising the allowable penetration levels of solar PV in our LV networks, we turn to what I believe becomes possible with this grid-based solar forecasting API: using this service in the #orchestration of DER services to LV networks.  By mapping solar variability to our LV networks, we can pair it up with solutions in demand management, energy storage and smart inverters.  With 10GW of small-scale solar, 4GW of large scale solar and 500k energy storage systems on the way by 2020 – I believe this our API can directly contribute to the successful integration of the incoming high penetrations of DER. 

Zooming out, orchestration is really the forefront of the renewables powered transition now underway across the Asia Pacific.  In this effort, we must look to turn intermittency into opportunity (on that topic, watch the video embedded here). Our ANU/Solcast team is working on precisely that! And we plan to make the API open to collaboration in that regard (have you seen our data sharing campaign for researchers?), wherever our DNSP partners determine that to be beneficial.

In Closing, Our Message: Let's Collaborate!

Thanks for taking the time to read this blog post and review the above information. If you are an Australian DNSP project partner, all of the API related services are open to use across your entire organisation. Please share it widely and freely – we can handle any amount of requests that you throw at the API, so have fun.

 Himawari 8 domain/imagery

Himawari 8 domain/imagery

If you are interested in deploying a similar solution (bottom up DER mapping of solar, deployment of a solar forecasting API service for solar farms and small-scale solar sites) elsewhere across the Asia-Pacific, get in touch with Solcast directly.  We’re looking for opportunities to collaborate across the Himawari 8 domain!

It’s all part of teaming up for the solar powered future!

-Dr. Nick Engerer-

#MODSIM2015 - Extracting Solar Radiation Measurements from PV System Power Output

 A bit of background info on my 21st International Congress on Modelling and Simulation (MODSIM2015) paper

[download the paper]

#MOSIM2015 is underway! And lucky me, I’m here at our wonderful sunny Gold Coast location awaiting my opportunity to present an innovative modeling approach which estimates solar radiation using PV system power output.

Without further delay, let’s dig right in...

The Premise

 A pyranometer!

A pyranometer!

Here’s the premise:  photovoltaic solar panels are not altogether different from pyranometers.  In most cases, pyranometers – the global ‘go-to’ solar radiation sensor – use a silicon wafer to measure the global horizontal irradiance at a given location.  Interestingly, this silicon wafer is based on the same technology as most PV modules.  The result is that the power output from a PV system has a first order relationship with the global solar radiation arriving in the plane-of-array.  As a scientist, this creates a sense of curiousity within me!  I wonder, can we use the power output from a PV system to then work our way back to a standard radiation measurement?

The Hypothesis

Let’s put that pondering within the framework of the scientific method, forming a hypothesis:

“Solar panels are not altogether different from pyranometers, thus one should be able to use their power output as an input to a ‘separation’ model to estimate the diffuse and beam components of solar radiation”

What is a separation model?

Over the past several decades, there have been many (as in hundreds) of models developed for the purpose of separating the diffuse and direct (beam) components from a global horizontal irradiance measurement.  In my recent paper (link), I describe this more thoroughly, so dig into that if you’re interested in learning more (or if you are a bit lost on the above bold terms)

 The Engerer 2 model

The Engerer 2 model

What I’ve done with my MODSIM2015 paper, is use my own separation model format – the “Engerer 2” model - and modified it to accept PV power output measurements as the primary input, in place of pyranometer measurements.  The overall goal here, is to estimate the diffuse component of radiation using only the measured PV power output through this model and then test how well it performed.

In this paper, we use a scientific approach to test this.  Two different model formats were tested.  One with a single input, and another with four inputs.  We compare a standard pyranometer based model (Kt), fit to the same data, to the PV power output based model (KPV).  This allows us to directly compare the two approaches in a controlled, scientific fashion. 

Now, in order to do this, we need data.  So, I turned to my usual sources of data, and the Bureau of Meteorology solar radiation monitoring sites.  Fortunately, I was able to identify 18 PV sites in Adelaide and Melbourne which were of very good quality and located within 10km of a solar radiation monitoring site. 

Next, I processed these data through quality control algorithms (read more in my PhD thesis) and then averaged them to hourly values.  With this QC complete, we’re able to start fitting and testing the models. 

There are a lot of details here, which I’ll glaze over in this blog post, but you can review in our paper.  Such as what QC methods I used, how I chose the PV sites, etc, etc.  I’ll leave it to you to investigate this further!  I’d rather spend my time here talking about what we found, because it is pretty cool.

The Results

First, in fitting and testing the single parameter model, the general approach undertaken was validatedThe idea works.  The relationship used in my diffuse fraction model (Engerer 2) relies on a generalized logistic function based on the relationship between the Diffuse Fraction (Kd) and the Clearness Index (Kt).  However, in the PV power output based model, we use KPV in place of Kt,  KPV is the clear-sky index for photovoltaics, and allows for the removal of all the individual nuances from the PV power output time series (use the link above to learn more)

This is a great start, but the results for a single input model aren’t very good.  This is because the model is too simple for the complex relationship between radiation and cloud cover.  But never fear, that’s where the Engerer 2 model comes in.

For those of you who are not familiar with this field, it's worth mentioning that the Engerer 2 model was recently declared the "quasi-universal" separation model in a global study using 54 sites to test 140 different models in a study by Chris Gueymard, a leader in the solar radiation modelling field.  So we're taking a modern approach to this problem.

Herein, we use the Engerer 2 model format to further test our hypothesis.  Except for the solar PV power output based model, we change another one of the input variables so that it is based on PV power output instead of a pyranometer measurement. This is the deviation from clear-sky variable (read more in the paper).

 Kt versus KPV based modelling within the Engerer 2 model framework. 

Kt versus KPV based modelling within the Engerer 2 model framework. 

The results were very encouraging! The figure above demonstrates that by using only the PV power output, we are able to use our generalised logistic function based model to estimate the diffuse fraction of solar radiation with only a 5% increase in rRMSE and 0.7% increase in rMBE over a pyranometer based model.  Moreover, the observed relative errors are within the ‘good’ modeling requirements established by Gueymard and Meyers 2008.

Great, now what?

Moving forward with this result, there is now more work to do! For example, with a pyranometer measurement, once we have the diffuse component, we get the direct component through the closure equation. With PV system power output, this isn't possible.   So we'll need to test our ability to do extract the beam component further.  Possibly, this might require the joint development of another model to extract it

Despite the further work required, the results herein remain exciting. The approach used in our paper show that extracting solar radiation measurements from PV system power output can be done. This is a world first for such a result, and it is exciting to know that further work in this space is realistic. 

I am very hopeful that this result will encourage others in the field to further explore this conceptual work.  If we can extract solar radiation measurements from PV power output, we suddenly can create a very rich dataset of radiation measurements globally, greatly augmenting our ability to anlayse the spatio-temporal evolution of incoming solar radiation.  Such a dataset can assist in the validation of satellite measurements or global climate/meteorological modeling tools. 

So, with this very awesome scientific finding, I’ll leave you with the above thoughts, and hope that you’ll go on to read the paper, cite the results and further develop the prospect of having millions of additional solar radiation sensors globally.

[download the paper]

[download the presentation]

View the presentation below:

What's Happening at #MODSIM2015

Find Stuff!

#SWC2015 - Launching the Regional PV Simulation System in Daegu, Korea at the Solar World Congress

Our conference paper is entitled: Real-time Simulations of 15,000+ Distributed PV Arrays at Sub-Grid Level using the Regional PV Simulation System (RPSS)

[download it here]

It's an absolute pleasure to be writing to you from Daegu, Korea, where a few hundred solar energy scientists are gathered for the International Solar Energy Society's 2015 Solar World Congress.  I've been inspired by the work that I've seen here, and I'm quite proud to be a part of this community.  

At this conference, it is my great pleasure to officially launch my latest research outcome, which progresses my PhD thesis out of the laboratory, and into the real-world.  It's been a fun challenge to make this leap from science to technology, and I'm honestly quite enthused by the outcome.

From Science to Tech, How We Did It

RPSS stands for Regional PV Simulation System, which was developed by Nicholas Engerer during his PhD thesis, and is now undergoing further development to achieve operational deployment in Australia.

Over the past several months, I have been working with ANU Engineering student James Hansard to get the Regional PV Simulation System up and running, live, in the Canberra, Australia region.  This has been made possible through several mechanisms, the foremost of which was excellent ingenuity and determination from James (you can download his thesis here). 


 There are 100+ sites reporting their power output into the RPSS as of November 2015

There are 100+ sites reporting their power output into the RPSS as of November 2015

Using an allocation on the National Computational Infrastructure, the RPSS is now running in near real-time via a series of three virtual machines.  The first machine is gathering PV power output data in near real-time (about 10-15 minute delay) from 100+ sites via, a webpage where users publicly report their PV system's power output.  The second machine runs the RPSS algorithms and produce the output files/graphics.  The latter of these three VMs, is a web server, which hosts the latest simulations at  At this webpage, you can see the RPSS output for the day, mapped to ActewAGL's distribution network.  

RPSS version 2: 15,000+ PV Systems Simulated by Transformer Node

In the latest version of the RPSS, I am now simulating 15,000+ PV systems, grouped by transformer.  This is a significant advancement from the 12,000+ PV systems mapped by suburb in my PhD thesis, which progresses the RPSS towards industry relevant outcomes.  If we can quantify the contributions of PV to the grid at a high resolution, and forecast any significant changes ahead of time, then the intermittency of distributed PV can be mitigated.  Getting the RPSS running in near real-time is the first step in this process.

Quantifying PV Power Output by Distribution Node (Transformer)

When we break down our simulations by transformer, we start to observe very interesting behaviours from the collectives of distributed PV installed behind them that have not previously been observed in Australia.  Let me show you what I mean!  First, let's watch a simulation (RPSS output) from 5 March 2014:

Now, if you don't think that video is pretty cool, then you must be much less nerdy than I am.  I think its fantastic how, using only input PV data from a 100+ sites, we can observe cloud-shadow like features in the simulation! (That's the power of KPV!)

Now, let's look in detail, at the simulated power output grouped by transformer node (below)..

The three different coloured lines above, each correspond to the simulated power output at a given transformer (node).  These are shown on the right side of the simulation video above.  What is very interesting, is the observed differences in ramp event timing that occur between the three nodes during the strong negative collective ramp event that occurs around 11AM.  From this, we can see that the timing of these events are very important to predict precisely, in both space and time, as the strong ramping can occur at different times for each of them.  In the strong positive ramp that follows, there is a similar mismatch in the timing of the ramp peaks.  For a scientist, this is fascinating.  For a distribution company, its a clear example of the challenge of PV intermittency.  

And all of it motivates further development and deployment of this tool

If you found this post interesting, you can dig into the science a bit deeper by downloading our paper below, or by viewing the slides from the presentation.  There is also a tag cloud below, to help you navigate through other content in my webpage.

[You can download our ISES SWC 2015 paper here]

[Or you can download a PDF of the talk]

Or view it in the slideshow below:

SWC2015 Action:

Tag Cloud (Find More Stuff)

PhD Final Seminar - City-wide Simulations of Distributed Photovoltaic Power Production

I've just successfully completed my PhD defence seminar, and am excited to share with you the results of my thesis project.

It sure is tough to fit a description of everything that I've accomplished over the past four years into an hour long seminar.  Well, in fact, it's impossible!  So much goes into a PhD thesis (dissertation), and only the author themselves will truly understand all the work that went into it.

But I think I've managed to do a good job getting the best of the best information into a manageable format - and I'll be able to share a video of the final seminar with you in a few weeks.  But for now, I'd like to get the slides up on the web, and post some of the simulation videos.

In the shortest summary possible: I've created a simulation system for Canberra PV installations, which uses a subset of monitored PV systems (approximately 80-160 systems, depending on the site availability/date) to simulate 12,500+ PV arrays.  The installed PV arrays are based on data provided by ActewAGL (local utility/distributor) about the rated capacity and suburb-level location of each install through December 2012. 

The heart of the simulation system is an application of the KPV methodology, which uses the power output from one PV system to simulate the performance of another nearby PV system. I've published a paper on the topic in Solar Energy  for those of you who are interested in the more technical side of things.

For now, I'm going to stop writing, post the slides and simulation videos, and let them speak for themselves.  I hope you enjoy them:

[download presentation]

Presentation Slides:

Presentation Videos: Simulations

Video of a a clear sky day for all PV systems installed in Canberra. The images you see are the result of using ~70 PV systems to simulate the remaining 12,500. 

Video of a high variability day where broad convective clouds moved through Canberra. 

Video of a positive ramp event resulting from fog dissipation in Canberra. The images you see are the result of using ~80 PV systems to simulate the remaining 12,500. 

Video of a negative ramp event resulting from convection moving into Canberra. The images you see are the result of using ~140 PV systems to simulate the remaining 12,500. 

The Impact of Weather Events on Solar Energy Generation: Recorded Presentation

Happy new year! As promised, here is a recorded version of our November 2014 presentation on weather events and solar energy generation in Canberra, Australia.

Please use the below recording from SoundCloud (press play!) and the below gallery of slides (click through them manually) to review our presentation.  There are two videos that will not work within the Gallery, but that's okay!  You can find them here, along with a summary of our research findings.

[download the presentation here]