PVSC44: Himawari 8 enabled real-time distributed PV simulations for distribution networks

Just over a year ago, I announced an exciting new project focused on delivering real-time distributed PV simulations to distribution network service providers (DNSPs). This is an Australian Renewable Energy Agency (ARENA) funded research project, led by the Australian National University. As the Chief Investigator on this research project, I have spent the last year regularly meeting with and networking with DNSPs across the country. As a result, we now have 12 of the 15 DNSPs on board, which is a tremendous outcome.

Each of theses DNSPs is providing us with the metadata for the installed PV system within their networks, which we are using to produce simulations of their power output.  This work extends the efforts of my PhD, where I built the Regional PV Simulation System (RPSS) through collaboration with locality utility ActewAGL, but adds new capability through real-time PV monitoring and derived irradiance estimates from the Himawari 8 satellite.

This week, I am at the PVSC44 conference, co-chairing the session on solar forecasting for grid integration of PV, as well as presenting a paper which outlines our modelling methodology for this new tool. I wanted to take the chance to put up an accompanying blog post, to help make it easily accessible to my fellow researchers. Of course if you want the real details, you should check out the paper :-).  It is important to note that the following methods are the result of a close industry collaboration with industry partner Solcast, and represented the state-of-the-art of its satellite nowcasting technologies from early 2017

[Download the Paper]

Here's how it all works..

Flow diagram of our methodology

Flow diagram of our methodology

We take images from the Himawari-8 satellite, in particular the visible and infrared channels, and apply the general principles of the Heliosat-2 approach for turning these into cloud opacity estimates. What's very interesting about this process, is that the spatial and temporal scales of this data are quite advanced, as compared to other geostationary weather satellites (e.g. GOES 13/15 over the US). At 1-2km^2 resolution with 10 minute update cycles, the radiation modelling enabled by this satellite is quite an exciting advancement for our field.

We use the Himawari 8 data in combination with the GFS NWP model to decompose clouds into lower, middle and upper troposphere layers, and then aggregate these layers to estimate total cloudiness. Total cloud opacity (represented as an index between 0 and 1) is then derived using differences between the lowest visible return value of albedo. With this value in hand, we apply a linear reduction to the Esra clear sky radiation model to produce an estimate of global horizontal irradiance (Gh) for each pixel. Estimates of Gh are afterwards decomposed into direct normal irradiance (Bn) and diffuse horizontal irradiance (Dh) through the Engerer2 separation model.

Satellite imagery decomposition

Satellite imagery decomposition

Radiation Validation on "Estimated Actuals"

estimated actuals

For this conference paper, we have validated the Gh values produced by this system. These "Estimated Actuals" are produced using the satellite scan for each time stamp (no forecasting involved). It is necessary to compare these outputs to the observed radiation conditions at the surface, which we accomplish using data from the Australian Bureau of Meteorology.  Using 7 sites and 2 months of data from the BoM, we were able to determine these estimated actuals to have a Mean Bias Error (MBE) of -7 W m-2 and Root Mean Squared Error (RMSE) of 55 W m-2.  This accomplishes the industry standard of a 'good' radiation model, which is defined in the literature as having an relative MBE less than 5% (1.8% reported) and relative RMSE less than 15% (10.7% reported).

PV Power Estimated Actuals + Validation

The next step in the methodology is to compute PV power output calculations, based on the installed characteristics of the PV systems considered.  This requires information such as the azimuth & tilt of the system, as well as the total installed capacity. We apply the Reindl transposition model to estimate the available plane of array irradiance, followed by a quadratic PV power model to compute the estimated power output. By combining the Estimated Actuals from the satellite system with this model, we also can compute Estimated Actuals for PV system power output.

Screen Shot 2017-06-27 at 14.47.55 pm.png

In order to accomplish this step & validate it, we collected data from 78 PV systems across the Canberra, Australia region for a period of 6 months.  Using these time-series data, we then applied our QCPV methodology to both quality control the reported power outputs, but also to determine the actual azimuths & tilts for the POA radiation modelling step.  For this validation, we report an MBE of 0.04 W/Wp and RMSE of 0.15 W/Wp

Overall scatter in the validation data shows well behaved results, but with accuracy losses nearly double that of the radiation validation. We note that this validation reports uneven bias across the distribution of PV system measurement values, suggesting further refinement of the quadratic PV model coefficients could be required.

Scaling it up for DNSPs

Given the relatively good performance of the modelling system (much improvement needed still for operational purposes) we apply this system to a network wide simulation.  By utilising the metadata for 15,500+ embedded solar PV systems in the ActewAGL distribution network, we produce Estimated Actuals for the PV power systems in this network via this methodology. Additionally, we implement a simple cross-validation using 6 PV systems in a real-time scenario and provide the demo in video format:

In the provided image, we see the Estimated Actuals for each individual PV system in the ActewAGL network, as provided for the 23rd of June 2016. This day was selected based upon its complicated, multi-layered cloud conditions to best demonstrate the capability of this system in a challenging circumstance. A heat map colour profile shows the relative power output from each PV system based on its installed capacity.  This is accompanied by the data from the 6 PV cross-validation sites in the circles which are colour filled according to their observed power output.

At bottom right of this image, the Estimated Actuals for these systems are displayed in grey, with the red line showing the observed power output values. Overall, we observe a slightly positive bias in the Estimated Actuals of 0.038 W/Wp and a relatively lower RMSE value of 0.12 W/Wp.

A Great Start: Where Next?

Overall, our team is quite happy with this first set of results, but we've many improvements to make to this system moving forward. As a part of this validation, we've already found & corrected issues with positive cloud opacity bias during overcast conditions, and we have implemented a dynamic kernel which corrects the cloud shadow displacement at low sun angles. Up next for us, is improving the radiation modelling routines. For one, Engerer2 was not designed for this purpose and our linear reduction on the Esra model needs to be updated.

The good news is, we've filled up our team with postdocs, PhDs, honours students and software developers, and are working hard to make these updates.  Soon, we'll even highlight the forecasting capability we've added to the system in an upcoming issue of Progress in Solar Energy.  In the meantime, we're deploying this system for our participating DNSP partners right now, and will iterate forward based on their feedback. So check back soon and connect with us on Research Gate to stay up to date & to collaborate.

It's all a part of teaming up for that solar-powered future! And in that spirit be sure to check out our data sharing campaign, where we are making the output of this system available for your research purposes. Read more at this link!

Lastly, the ANU team would like thank & acknowledge our industry partner Solcast's contributions to this project. 

Announcing My ARENA Project: Real-time Operational PV Simulations for Distribution Network Service Providers

ARENA Project Launch - 13 April 2016

Members of the successful ARENA projects - who is that handsome guy front right? ;-)

Members of the successful ARENA projects - who is that handsome guy front right? ;-)

This has been a truly incredible week.  I have just returned from Perth, Australia where the Australian Renewable Energy Agency (ARENA) announced the outcomes of their Industry-Researcher development funding round.  All up, 9 projects were funded, totaling over $17M in funding, comprising a very impressive array of projects.  Proudly, the ANU is leading 3 projects and pulled in $4.7M from the scheme!  And looking across the remaining 6 projects, I can't help but feel incredibly proud of the innovative work Australian researchers are doing, as well as be excited about the future of renewable energy Down Under!

What's great about this funding round, is that it required industry and researchers to work together, and put up projects which have clear commercialisation potential.  That means projects which are "real-world relevant" were required, which, if you know me, meant that I was pretty darn excited!  This funding round is exactly where my passions and interest lie, and it lines up very well with the government's desired outcomes from universities in Australia, and is well aligned with the big push for innovation across the higher education sector

So, Why was I there?

Why do this? Lemme explain..

Why do this? Lemme explain..

To answer that question, we need some background.

I have spent the last few years working on another ARENA project which has focused on distributed small scale solar forecasting via machine learning and computer vision techniques.  This was an ANU-NICTA collaboration, which will wrap up mid-2016, (just before NICTA is subsumed into Data61/CSIRO - cue the doomsday music!).  One of the outcomes of this project, was my PhD work on the Regional PV Simulations System (RPSS).

RPSS v1 during a positive collective ramp event

RPSS v1 during a positive collective ramp event

Version 1 of the RPSS was developed within my thesis, and produced a modelling environment which simulated 12,000+ small-scale solar PV systems in Canberra (based on December 2012 installation data from ActewAGL).  This used data from PV systems that reported their power output on, which through the magic of KPV, I was able to upscale into city-wide simulations

In my thesis, I used several critical collective ramping events to demonstrate how quickly the power output from 12,000+ PV systems can change across an entire region.  This was a fun tool, but only worked with historical data and at surburb-level.  Read between the lines here: it was cool, but ultimately not directly useful. This is the type of science that would stay 'on-the-shelf'.

Enter RPSS Version 2

After I submitted my PhD, I continued to work on the RPSS, developing it up to version 2, which I launched at the Solar World Congress in Daegu, Korea in November 2015. This version featured two major upgrades:

  1. It worked with near real-time data from the live solar API, updating every 5 minutes (check out the beta at
  2. It mapped the simulated PV systems to the transformer nodes on ActewAGL's distribution network (now it's actually useful!)
RPSS v2, live data and mapped to ActewAGL's network

RPSS v2, live data and mapped to ActewAGL's network

These significant advancements demonstrated my ability to take the science off the shelf, get it out of the lab, and into relevance.  Working with live data is not easy (which is why the current version is merely a clunky beta!), nor is convincing an electrical utility to give you detailed data about their distribution network, but it teaches you about the many things required to advance a science to a technology.  And if you're like me, you'll fall in love with the challenge.

It was this second version of the RPSS that I took forward to ARENA, requesting funding to deploy it to distribution networks around Australia in order to address the challenges with integrating high-penetrations of solar PV into Australian electricity networks.  Does that sound like fun to you? Because it sure as hell does to me!

The Challenge: High penetration solar PV

High penetration solar in a neighbourhood in Canberra - solar PV on every roof!

High penetration solar in a neighbourhood in Canberra - solar PV on every roof!

With more than 1.5 million solar PV systems installed to date, totaling more than 4.5GW of capacity, the maximum penetration levels of solar PV are being reached in some areas of Australia. (e.g. Ergon 3.5 kW per system limits, Horizon Power PV limits on radial style networks). 

This 'maximum penetration' level refers to the maximum allowable amount of solar PV a utility will allow on a given part of their network.  The key word here is 'allow'.  For the most part, Distribution Network Service Providers (DNSPs) are taking preventative approaches to distributed PV intermittency, imposing maximum penetration levels that are significantly lower than are technically achievable.

Ergon is limiting the size of individual solar PV systems on parts of their distribution network

Ergon is limiting the size of individual solar PV systems on parts of their distribution network

The Key issue: solar PV = ?

There is one key item that is holding DNSPs back from being more liberal with their maximum PV penetration levels: most Australian DNSPs have no active feedback quantifying how much electricity their embedded PV generators are currently producing at any given time.  On that note, they also don't know how much PV variability will occur in the near future, nor what has occurred in the past.  So how could they possibly manage the inevitable solar future where everyone has solar PV on their roof if they have no idea how much power they are generating at any given time?

This is the key knowledge gap my project will address:  quantifying the current and expected distributed PV power production across distribution networks in near real-time.  That means quantifying distributed PV power output with enough lead-time to do something about PV-induced voltage fluctuations.  In other words, using technology to enable proactive, rather than reactive, grid management. 

What We'll do in response to the Challenge

The solar knowledge gap: how much solar & when/where?

The solar knowledge gap: how much solar & when/where?

This project will take the Regional PV Simulation System (RPSS v2) and develop and deploy it as an operational software that provides distribution network service providers (DNSPs) with real-time distributed photovoltaic (PV) simulations that are mapped to their distribution network.  The output of these simulations will be directly aimed at the knowledge gap that exists between distributed PV integration challenges and their solutions (e.g. energy storage technologies and/or remote demand/supply management). 

HOW we'll do it: Key PArtnerships

Distributed solar PV simulations are not totally unique to my research, and are used by many other researchers in Australia and the rest of the world.  They are integrated into the APVI Solar Map, they are used in Clean Power Research's PV FleetView product, as well as in many related projects around the world.  Where this project sets itself apart from all others is in the consortium I pulled together to accomplish the task and the unique data that they will provide.

This project will be led by myself (Chief Investigator), with the ANU being the sole research partner. Everyone else is an industry partner, and I'm excited to tell you about them.  They fall into a few key categories:

1. Distribution Network Service Providers (DNSP)

DNSP partners ARENA

This project includes active participation from 6 DNSPs, who will provide information about the solar PV systems installed on their networks, so that we can deploy the RPSS to their service region. I am hopeful that more DNSPs will join this project in the near future.

Photo taken in lobby of Western Power's Head Office in Perth CBD

Photo taken in lobby of Western Power's Head Office in Perth CBD

Photo taken in Horizon Power's Head Office in Bentley, WA

Photo taken in Horizon Power's Head Office in Bentley, WA

2. TWo Inverter companies, an inverter wholesaler & a solar installer

One of the primary inputs to the RPSS is monitored solar PV data, which we currently get from in the online beta version.  This isn't quite good enough, as this data is slow and unreliable, mostly because it comes from non-professional sources. 

This project will work with two inverter companies, SMA Australia and Fronius, to develop real-time, rapid update monitored PV data inputs to the RPSS.  The number of internet connected PV inverters is sky-rocketing, with Fronius installing 50/week at present.  This is an excellent and rapidly growing source of data.

Key project supporter & CTO/Owner of Si Clean Energy, Peter Bulanyi

Key project supporter & CTO/Owner of Si Clean Energy, Peter Bulanyi

This project is receiving significant support from inverter wholesaler, Si Clean Energy, with its involvement led by Owner/CTO Peter Bulanyi.  Peter has been the project's number one supporter from day one, and I would like publicly thank him for his adamant, steadfast support!  Si Clean Energy will also grant access to the AllSolus monitoring network of 1000+ irradiance sensors and PV sites across Australia. 


Benn Masters, Director of SolarHub & key project supporter

Benn Masters, Director of SolarHub & key project supporter

We will also be working with solar & battery installer SolarHub, who are a progressive, high-quality company based in Canberra.  SolarHub has been connecting a large number of monitoring systems to PV installations across Canberra, and will grant access to these data over the course of the project.  They have also been a key supporter of the project from its very beginnings, and I would like to also publicly thank Benn Masters, Director at SolarHub for his significant & strategic contributions to the project.

3. ADvisory support

Patrick Dale, Director, Aeris Capital

Patrick Dale, Director, Aeris Capital

Jesse Warburg, Associate Director Aeris Capital

Jesse Warburg, Associate Director Aeris Capital

We have two companies providing in-kind support to the project.  The first is Aeris Capital, whose Director Pat Dale and Associate Director Jesse Warburg are both very excited about the consortium in place and the technology we plan to develop.  Their role in the project is to help drive it toward valuable outcomes for industry and guide us through the commercialisation of the technology. 

Second, we'll have some input from local solar installer and owner of a solar PV monitoring device called the 'esquid', Soly Ltd., whom will provide technical advice into the project.

4. outreach support: Australian Photovoltaic Institute

Finally, we have negotiated a partnership with the Australian Photovoltaic Institute (APVI) to work simulations from the RPSS into the APVI Live Solar Map, which will expand upon the incredible work being done by this progressive and very important voice for solar in Australia.  This is a great place to insert kudos to Dr. Anna Bruce at UNSW for her excellent work on this APVI Live Solar Map, which is an invaluable contribution to solar PV science outreach.  I am very excited to help build upon their Live Solar Map tool, and contribute to improving on this impressive work out of UNSW. 

What we'll do: merging datasets, Deploying RPSS

RPSS modeled ramp events across 3 transformer nodes in ActewAGL's network.

RPSS modeled ramp events across 3 transformer nodes in ActewAGL's network.

Using our unique consortium of partners, we'll be deploying increasingly advanced versions of the RPSS to the distribution networks of each DNSP.  In the first year of the project (2016-2017), we'll deploy the RPSS v2, based on currently available PV monitoring data-streams like or the AllSolus network.  This version will be about 10 minutes behind 'real-time' and will be based heavily off the existing work that I've done during my PhD.

In 2017-2018, we will advanced the RPSS to include real-time input data from the Himawari 8 satellite as well as rapid-update PV monitoring input from SMA Australia and Fronius, closing the gap between 'near real-time' and 'real-time' operations.  This version will include updates from the latest in PV simulation and solar radiation radiation research.

A selfie from my April 2016 visit with Western Power

A selfie from my April 2016 visit with Western Power

Then finally, in 2018-2019, we'll have all the kinks worked out of the simulation system, having advanced it to true real-time status across all participating DNSP networks.  This version will then be used in the control rooms of these DNSPs to appropriately manage intermittent, distributed PV power production. 

It is highly likely that they will be pairing distributed PV modelling data with technologies like energy storage, to raise the maximum penetration levels of PV across their electricity network.  After all, that is the entire goal!  I look forward to updating this blog throughout the course of the project.

Up Next: Putting together an enthusiastic talented Team!

This project will kick off from July 2016, so I am now looking to put together a team of creative and inspired Originals, who are ready to dedicate their energy to pulling off a fantastic, influential and real-world driven project.  At this time the team I envision looks something like this:

Are you a unique, passionate person with awesome ideas? Then I want YOU participating on this project!

Are you a unique, passionate person with awesome ideas? Then I want YOU participating on this project!

  • A project manager/power systems engineer
  • 1-2 software engineers
  • 2 postdoctoral researchers
  • 6-8 ANU PhD candidates
  • 6-8 International PhD candidate visiting students
  • 8-10 ANU Masters research students
  • 20-30 ANU undergraduate research students

The core team consists of the engineer, programmers & postdoctoral researchers. Each of this persons will be fully funded by the project. Each of these positions will be advertised via the ANU over the next few months.

The secondary team will consist of promising PhD candidates, masters students and undergraduates.  The project has limited funds for scholarships, so I am searching for students who can self-fund themselves.  This would mean an APA scholarship for domestic PhD students, or country of origin funding sources for international PhD students.  Masters and undergraduate students who are enrolled at The ANU can participate in the project through coursework credit. 

I will be putting together a "How to Work with Me" webpage for all ranges of students in the next few days! Stay tuned!

I am very excited to see what our team will look like in the near future and sincerely hope that anyone who wants to participate will get in contact with me.

That's enough for now!

Thanks for reading about this exciting project, and looking into my latest research update.  I am excited to share more information with you, as this project moves forward, and plan to ensure our experiences from this project are shared with research scientists Australia-wide.  Stay tuned!

See ya next time!

See ya next time!

Find More Stuff:

CWS Lab - Part 1: An Excited Nerd Gets Access

Two weeks ago, I attended the AMOS 2015 Annual Conference - that's the Australian Meteorological & Oceanographic Society for those of you unfamiliar to this yearly weather & climate geek gathering down under (it's awesome). 

And while there are many, many great things that happened at this conference that I'd like blog about, I'm going to keep this very focused on one topic: a great new tool called CWS Lab.

During the conference, there was a demonstration setup for CWS Lab, manned by a charming bloke name Tim Bedin (aka captainceramic).  Tim is a Bureau of Meteorology employee who has dedicated a significant amount of his time (along with many other team members whom I'm now getting to know quite well) to developing this virtual machine environment.

#CWSLab on Twitter!

Now, I was ready to check out CWS Lab prior to heading up to Brisbane for the conference, having heard about it from Dr. Damien Irving (@DrClimate) via Twitter - where all great things happen in 140 characters or less.  He's been one of the pioneers on this project too, and another very helpful, tech-savvy guru there to help you get started (you'll find that people helping me is a common theme on this adventure). So I was actively looking for this demonstration, and excited to see what it could do.

What is CWS Lab?

You couldn't be faulted for asking: what's the big deal about this CWS Lab thingie anyway? CWSL (or CWS Lab) stands for "Climate & Weather Science Laboratory".  CWSL is explained in depth in other places, such as here or here, so I'll give you the quick & easy version: in short, at present, its a quick, easy way to access the CMIP5 suite of climate model output (used in the Fifth IPCC report), which is a treasure trove of potential analyses and juicy science just waiting to happen.  And like many researchers, I've had so many research ideas bouncing around in my head regarding this type of data - but I've always been held back by the learning curve required to study up on all the scripting needed to load, process and visualise data.  

Well, this is exactly what CWS Lab is for.  It makes it easy to generate visualisations/analyses of the model output data through workflow tools such as VisTrails and a custom set of tools/examples hosted on GitHub.  The tool was so simple and so fast that Tim was able to whip up some imagery of the mean incoming solar radiation fields for Australia for a given model year - and from that point I was hooked. I had to get access, get started and make some science happen.

NCI logo

Getting Access to the NCI

Getting started wasn't quite as easy as whipping up those solar radiation plots. There were several things I needed to do first.  For example: in order to use CWS Lab, you need to have an active NCI (National Computational Infrastructure) account (sign up here) AND you have to have an allocation to associate your account with.  This means an existing project, with a set amount of computational resources that you can use.  Now for everyone using CWS Lab so far, they've been able to join a project that someone else had already started (aka someone in your research group or university).  But not me - I had to do some extra steps.  However, that means its worth me writing about, so that all you other lonely people out there without any climate nerd friends to collaborate with, can learn from my experience.

Applying for an NCI Start-up Account

Thankfully, the NCI folks have planned ahead for people just like me (in between funding, no prior experience on NCI, no friends with allocations to piggy-back on) and created the NCI Start-up Grants.  They are very straightforward to apply for, and I experienced a very quick turn around of <24 hours between application and being granted an allocation!  After you have your NCI login setup (previous step), then navigate to "Propose a Project"

NCI propose

 Now you need to give the project a title and provide some justification for why you need your startup allocation. I wrote about 3 paragraphs, justified the need for the science, put in a basic research question and explained I'd use the allocation to provide some student projects.  Then you need to enter your FOR and SEO codes, and advance to the next few pages.  Leave all the allocation specific requests as the default - the startup scheme has a set allocation size.

Wait for it - Being Granted Your Project

While you're waiting for your project to be approved, its a great time to go install the pre-requisite software for logging into the CWS Lab virtual machine (that's a computer with operating system that runs in the cloud).  The instructions are hosted on Google Drive via link on the project github page.  Once you've installed all of this stuff (a VNC Client and Desktop Launcher, and followed the instructions carefully), there are two remaining things you need to happen:

1.  Your project has to be approved and assigned:

NCI Project approval


2.  You need to be granted access to CWS Lab 

I'm not exactly sure how I was granted access to the CSW Lab environment, because I emailed multiple different people.  You should send a request to access the service to and - that should cover it.

Getting Started!? Yeah...Not quite

So once I was given access to the CWS Lab tools, had my allocation and login information, and I installed all the needed software - I was able to login into the CWS Lab environment! Hooray! 

So now, surely, I could get started plotting all my amazing solar radiation maps, and all will be right in the world, with research papers flowing out of my ears and into the science-verse!  But alas, I had more to do - and some of it took even more help from the (thankfully wonderful) folks at the NCI. So that's what's up next, in my second blog post. Why wait? Because honestly, I'm still figuring it all out myself! So for now - I hope this first post helps you along a bit, getting started with CWS Lab! More soon!

Peace, Love and Solar,