Launching Our Grid-based Solar Forecasting API at #CIDER17

CIDER17

Today, I’m presenting at the #CIDER17 conference in Sydney.

This conference, is focused on DER integration with networks in the Asia-Pacific, and nowhere is this more relevant than here in Australia.  In my presentation, I am launching of our grid-based solar forecasting API, which has now gone live in beta with a total of 8 Australian distribution networks now available (3 more added soon...).

This API deployment is a part of our ARENA funded distributed solar forecasting and modelling project where we are tasked with the integration solar forecasts into LV network operations.  Here, we have two key outcomes. First, to make accessing solar forecasts easy, scalable and to directly integrate them with the existing software in use by the DNSPs.  And second, raise the allowable penetrations of solar PV in these networks through #orchestration and proactive (rather than reactive) LV network management.

Integration via an API Framework

With that first outcome in mind, we have chosen an API framework for the delivery of our solar forecasting service.  This API is straightforward and built to start the integration process. Data from the API can be previewed in an HTML page for ease of use or testing, but can also be downloaded in specific data formats including JSON, XML and CSV via your own scripts.  Over the next year, we’ll be focusing in on building integrations for your load flow & network modelling software of choice – so be sure to provide us your feedback on what development you’d like to see in this space.

Getting to the grid-based API specifics, lets start off with “the how” we create the forecasting aggregations exposed in this API service.  Using the installed DER metadata for small-scale solar PV sites, provided by DNSPs, we have been able to:

  1. Generate PV power output forecasts & estimated actuals by matching the radiation estimates from our satellite nowcasting system to the locations and characteristics of the installed PV sites
  2. We then aggregate the PV power output forecasts to the upstream distribution transformers and zone substations; providing the total PV power output forecast for those networks assets
  3. Finally, we expose this forecast data via an API framework, which currently makes these aggregations available the zone substation level

Once these aggregations are completed, participating DNSPs are able to retrieve forecasts (as well as estimated actuals, more on that soon) via a wide variety of methods and file formats (as discussed above).  DNSPs first must register for the Solcast API services, and then are granted access to these aggregations for their network via their unique API key. We have instructions available on how to retrieve these forecasts & use the API, which is available on request (just send us an email!).

At this time, we have a total of 3 API endpoints unique to each DNSP:

  1. The ability to retrieve the API’s list of zone substation names available for forecast retrieval. This makes it easy to access the next two endpoints…
  2. An endpoint which provides a 0-7 day forecast, at 30 minute intervals (updated behind the scenes every 10 minutes) for the specified zone substation
  3. An endpoint which allows the DNSP to retrieve the estimated actuals for the downstream PV sites at that asset.  The estimated actuals are a look back at the past 7 days, at 30 minute intervals using the valid satellite scan from that time [no forecasting involved] (Note: this will be expanded to reach back further soon!)

#CIDER17 Demo of the API:

In the interest of showing off the capabilities of this API system, I am attaching the following screenshots and images of the API in actions. Several of these images are from forecasts & estimated actuals retrieved just prior to this talk, the remainder are from other events of interest. 

(Follow-up to Solar Analytics talk yesterday - ANU solar PV Simulations from 2014....)

 

Help Us Improve: Our Priority Here is DNSP Testing & Feedback

At this stage, we are asking DNSPs to access, analyse and critically review this API offering.  We want to know how they will use it, what features they’d like to see added and where we can integrate with existing software and operational strategies.

Looking Ahead: LV Network Orchestration

Getting back to outcome #2, raising the allowable penetration levels of solar PV in our LV networks, we turn to what I believe becomes possible with this grid-based solar forecasting API: using this service in the #orchestration of DER services to LV networks.  By mapping solar variability to our LV networks, we can pair it up with solutions in demand management, energy storage and smart inverters.  With 10GW of small-scale solar, 4GW of large scale solar and 500k energy storage systems on the way by 2020 – I believe this our API can directly contribute to the successful integration of the incoming high penetrations of DER. 

Zooming out, orchestration is really the forefront of the renewables powered transition now underway across the Asia Pacific.  In this effort, we must look to turn intermittency into opportunity (on that topic, watch the video embedded here). Our ANU/Solcast team is working on precisely that! And we plan to make the API open to collaboration in that regard (have you seen our data sharing campaign for researchers?), wherever our DNSP partners determine that to be beneficial.

In Closing, Our Message: Let's Collaborate!

Thanks for taking the time to read this blog post and review the above information. If you are an Australian DNSP project partner, all of the API related services are open to use across your entire organisation. Please share it widely and freely – we can handle any amount of requests that you throw at the API, so have fun.

Himawari 8 domain/imagery

Himawari 8 domain/imagery

If you are interested in deploying a similar solution (bottom up DER mapping of solar, deployment of a solar forecasting API service for solar farms and small-scale solar sites) elsewhere across the Asia-Pacific, get in touch with Solcast directly.  We’re looking for opportunities to collaborate across the Himawari 8 domain!

It’s all part of teaming up for the solar powered future!

-Dr. Nick Engerer-

PVSC44: Himawari 8 enabled real-time distributed PV simulations for distribution networks

Just over a year ago, I announced an exciting new project focused on delivering real-time distributed PV simulations to distribution network service providers (DNSPs). This is an Australian Renewable Energy Agency (ARENA) funded research project, led by the Australian National University. As the Chief Investigator on this research project, I have spent the last year regularly meeting with and networking with DNSPs across the country. As a result, we now have 12 of the 15 DNSPs on board, which is a tremendous outcome.

Each of theses DNSPs is providing us with the metadata for the installed PV system within their networks, which we are using to produce simulations of their power output.  This work extends the efforts of my PhD, where I built the Regional PV Simulation System (RPSS) through collaboration with locality utility ActewAGL, but adds new capability through real-time PV monitoring and derived irradiance estimates from the Himawari 8 satellite.

This week, I am at the PVSC44 conference, co-chairing the session on solar forecasting for grid integration of PV, as well as presenting a paper which outlines our modelling methodology for this new tool. I wanted to take the chance to put up an accompanying blog post, to help make it easily accessible to my fellow researchers. Of course if you want the real details, you should check out the paper :-).  It is important to note that the following methods are the result of a close industry collaboration with industry partner Solcast, and represented the state-of-the-art of its satellite nowcasting technologies from early 2017

[Download the Paper]

Here's how it all works..

Flow diagram of our methodology

Flow diagram of our methodology

We take images from the Himawari-8 satellite, in particular the visible and infrared channels, and apply the general principles of the Heliosat-2 approach for turning these into cloud opacity estimates. What's very interesting about this process, is that the spatial and temporal scales of this data are quite advanced, as compared to other geostationary weather satellites (e.g. GOES 13/15 over the US). At 1-2km^2 resolution with 10 minute update cycles, the radiation modelling enabled by this satellite is quite an exciting advancement for our field.

We use the Himawari 8 data in combination with the GFS NWP model to decompose clouds into lower, middle and upper troposphere layers, and then aggregate these layers to estimate total cloudiness. Total cloud opacity (represented as an index between 0 and 1) is then derived using differences between the lowest visible return value of albedo. With this value in hand, we apply a linear reduction to the Esra clear sky radiation model to produce an estimate of global horizontal irradiance (Gh) for each pixel. Estimates of Gh are afterwards decomposed into direct normal irradiance (Bn) and diffuse horizontal irradiance (Dh) through the Engerer2 separation model.

Satellite imagery decomposition

Satellite imagery decomposition

Radiation Validation on "Estimated Actuals"

estimated actuals

For this conference paper, we have validated the Gh values produced by this system. These "Estimated Actuals" are produced using the satellite scan for each time stamp (no forecasting involved). It is necessary to compare these outputs to the observed radiation conditions at the surface, which we accomplish using data from the Australian Bureau of Meteorology.  Using 7 sites and 2 months of data from the BoM, we were able to determine these estimated actuals to have a Mean Bias Error (MBE) of -7 W m-2 and Root Mean Squared Error (RMSE) of 55 W m-2.  This accomplishes the industry standard of a 'good' radiation model, which is defined in the literature as having an relative MBE less than 5% (1.8% reported) and relative RMSE less than 15% (10.7% reported).

PV Power Estimated Actuals + Validation

The next step in the methodology is to compute PV power output calculations, based on the installed characteristics of the PV systems considered.  This requires information such as the azimuth & tilt of the system, as well as the total installed capacity. We apply the Reindl transposition model to estimate the available plane of array irradiance, followed by a quadratic PV power model to compute the estimated power output. By combining the Estimated Actuals from the satellite system with this model, we also can compute Estimated Actuals for PV system power output.

Screen Shot 2017-06-27 at 14.47.55 pm.png

In order to accomplish this step & validate it, we collected data from 78 PV systems across the Canberra, Australia region for a period of 6 months.  Using these time-series data, we then applied our QCPV methodology to both quality control the reported power outputs, but also to determine the actual azimuths & tilts for the POA radiation modelling step.  For this validation, we report an MBE of 0.04 W/Wp and RMSE of 0.15 W/Wp

Overall scatter in the validation data shows well behaved results, but with accuracy losses nearly double that of the radiation validation. We note that this validation reports uneven bias across the distribution of PV system measurement values, suggesting further refinement of the quadratic PV model coefficients could be required.

Scaling it up for DNSPs

Given the relatively good performance of the modelling system (much improvement needed still for operational purposes) we apply this system to a network wide simulation.  By utilising the metadata for 15,500+ embedded solar PV systems in the ActewAGL distribution network, we produce Estimated Actuals for the PV power systems in this network via this methodology. Additionally, we implement a simple cross-validation using 6 PV systems in a real-time scenario and provide the demo in video format:

In the provided image, we see the Estimated Actuals for each individual PV system in the ActewAGL network, as provided for the 23rd of June 2016. This day was selected based upon its complicated, multi-layered cloud conditions to best demonstrate the capability of this system in a challenging circumstance. A heat map colour profile shows the relative power output from each PV system based on its installed capacity.  This is accompanied by the data from the 6 PV cross-validation sites in the circles which are colour filled according to their observed power output.

At bottom right of this image, the Estimated Actuals for these systems are displayed in grey, with the red line showing the observed power output values. Overall, we observe a slightly positive bias in the Estimated Actuals of 0.038 W/Wp and a relatively lower RMSE value of 0.12 W/Wp.

A Great Start: Where Next?

Overall, our team is quite happy with this first set of results, but we've many improvements to make to this system moving forward. As a part of this validation, we've already found & corrected issues with positive cloud opacity bias during overcast conditions, and we have implemented a dynamic kernel which corrects the cloud shadow displacement at low sun angles. Up next for us, is improving the radiation modelling routines. For one, Engerer2 was not designed for this purpose and our linear reduction on the Esra model needs to be updated.

The good news is, we've filled up our team with postdocs, PhDs, honours students and software developers, and are working hard to make these updates.  Soon, we'll even highlight the forecasting capability we've added to the system in an upcoming issue of Progress in Solar Energy.  In the meantime, we're deploying this system for our participating DNSP partners right now, and will iterate forward based on their feedback. So check back soon and connect with us on Research Gate to stay up to date & to collaborate.

It's all a part of teaming up for that solar-powered future! And in that spirit be sure to check out our data sharing campaign, where we are making the output of this system available for your research purposes. Read more at this link!

Lastly, the ANU team would like thank & acknowledge our industry partner Solcast's contributions to this project. 


A Data Sharing Compaign to Enable Researchers

A Call to University Researchers to "Get Applied"

It's no big secret that Australia's electricity sector is undergoing a rapid transformation, one that continues to surprise even the most aggressive growth targets for solar and storage, and has our regulators and market operators in a very demanding position. High electricity wholesale and spot prices, mean the costs of electricity are going up. Meanwhile, the cost of renewables and storage are falling quickly.  All across Australia, innovative companies are popping up which are providing solutions in the space of demand management, energy storage and energy trading and driving this transformation from the ground up.

in this electricity sector transformation, what role can universities play?

In this mix, I wonder, what role can Universities play? That's what my Academic mindset wants to know: how can we as researchers contribute to this challenging transformation?  I've had the great privilege of traveling much of Australia lately to visit with many segments of the electricity industry, and in each of those instances, I am filled with ideas on where Universities could contribute to the challenges that this sector is facing. 

But in wondering how best to make these relationships, these BIG ideas happen, I find myself exceptionally time-constrained. I simply can't work with everyone directly! Yet I feel a strong drive to help my fellow University researchers reach the real-world - how best to reconcile these?

Solar Data Enables

In contemplating this, I came to an interesting conclusion: high-resolution solar data is actually an enabling resource.  Let me explain what I mean - want to decide how to charge/discharge a battery? You need to know when the sun will or will not shine. Want to build a demand management algorithm for a commercial building? You need to understand how the rooftop solar array's power output will fluctuate during peak demand periods.  Need to orchestrate the dispatch of hundreds of MW of storage technologies? When will the next solar ramp event arrive and drive 100+MW shortfall of supply over the energy market zone you're trading in?

In the future grid, solar data is everything.  In fact, in my recent talk to the Energy Networks Australia "Grid Edge" Event [watch it here], I proposed that solar intermittency is the next frontier of opportunity for future grid technologies.

If we, as Universities, want to contribute to the future grid, this incredible transformation, then the research we complete will inherently require access to good solar data. And by 'data', I more specifically mean solar radiation or solar power generation, at high spatial and temporal resolutions.

Data Access Can Be Hard

However, solar data access is not easy. Most solar data in Australia is based on Typical Meteorological Years (TMY) climatologies, comes from only a handful of solar radiation sites scattered about the continent, or comes bundled on a harddrive at hourly resolutions and full of bias error. Other options include paying significant cash to commercial companies, which are happy to make you pay thousands of dollars for small datasets - no thanks! Shouldn't there be a better solution?

Building a Solar Forecasting System is Not Easy!

Example forecasts for small-scale solar in South Australia from 22 March 2017. Probability bounds are provided in the shaded regions of each forecast.

Example forecasts for small-scale solar in South Australia from 22 March 2017. Probability bounds are provided in the shaded regions of each forecast.

In addition to historical solar radiation data, researchers equally need to be able to access solar forecasting information. If you want to build any of the future grid enabling technologies which seize the opportunity of intermittency, they must be built in a fashion which captures the uncertainty of solar forecasting. 

At ANU/Solcast, we are pretty good at predicting the near-term availability of solar radiation, but we're not perfect either.  Given the stochastic nature of cloud formation & dissipation, their fast-moving, fast-changing characteristics and their complex optical properties, we are never going to get a cloud/solar radiation forecast 100% correct, 100% of the time.  But what we can do, is provide researchers with access to a state-of-the-art solar forecasting technology with world-class accuracy, that includes probabilistic forecast information. This way, they can build tech with these uncertainties built in - a key part of orchestrating the future grid!

Solar Data + Solar Forecasts = Easy to Share

Our Himawari 8 derived solar irradiance datasets at right, compared with the Bureau's old satellite data at left.

Our Himawari 8 derived solar irradiance datasets at right, compared with the Bureau's old satellite data at left.

In my personal quest to enable other researchers, I've realised, that in my position, with my team at the ANU, and our work with our start-up company Solcast, we can fix this problem.  We can openly and freely share solar radiation data, PV system power output datasets and solar forecasts through our joint ARENA project.   As project Chief Investigator, I have decided to inject some ARENA project funds into a data sharing campaign as part of our knowledge sharing efforts, to support the computational requirements of servicing this data to external parties. 

To be clear, that means that YOU, the researcher, get to extract the data for free for R&D purposes

And with the #SolcastAPI it is actually that easy. Within minutes of registering, you can access historical "Estimated Actuals" derived from the Himawari 8 satellite (right now, reaching back 7 days, in the near future - much deeper!) and solar radiation forecasts for GHI, DNI and DHI (powered by my Engerer2 model).  You can also use the API to simulate solar PV system power output, either via the Estimated Actuals or through the forecasts (0-7 days our, 30 minute increments).

To get started, visit the data sharing page I placed on my webpage, to provide more information about this service.  There you'll find some links for getting started. In the near future, I'll add some demos which show how to access this data, and will provide some research ideas for using high-resolution solar data for applied university research.

It's all part of teaming-up for the solar powered future! I look forward to seeing what you come up with!


Find More Stuff:

#saveARENA - There Has Never Been a More Critical Time for Renewables Research in Australia

I Have an Urgent & Important Message for You about The Australian Renewable Energy Agency

For those of you who follow my page, or have adventured here from beyond, it should be readily apparent to you that I’m an academic working in solar energy related research, more specifically in the modelling & forecasting of distributed PV power output.

To date, the majority of my work has been funded by ARENA, the Australian Renewable Energy Agency.  A funding body that has been absolutely crucial in supporting Australia as a global leader in renewable energy research over the past several years.

But ARENA is Under Imminent Threat

But right now, ARENA is under threat. The Coalition Government is planning legislation that will strip it of more than $1B of funding, and Labor are saying they won’t oppose it.

Let me explain to you why these cuts to ARENA research funding are a bad idea.

ARENA is currently funding nine university led research projects across Australia with more than $17M in funding, which are focused on partnering industry and researchers together to solve the major challenges we face integrating renewables into our electricity grid. What’s more, they intend to produce commercial outcomes valuable to the Australian economy.  (Isn't this exactly what the Australian government wants universities to accomplish? i.e. the National Innovation Science Agenda)

As an example, look no further than my own project, which received $1M from ARENA and raised $300k of industry cash to build distributed PV modelling and forecasting software for six distribution networks: ActewAGL, Ergon, Essential Energy, Western Power Horizon Power & Power and Water.

Perhaps this isn't quite enough industry partnership activity for the Coalition government?

Perhaps this isn't quite enough industry partnership activity for the Coalition government?

Don't believe the lies about a lack of value: ARENA has shifted toward driving commercially-focused outcomes

At the ANU, we're not just developing this distributed PV modelling technology to write some fancy publications.  We are delivering our software into the operations of our distribution network partners and creating a technology that will enable higher penetrations of solar to be added to the grid.  Going one step further, this software is being commercialised by an ANU start-up company, which we have already formed and is starting to complete this work now.  In the future, we hope to take this technology beyond Australia, and into other countries around the world.

Without ARENA funded research like mine, fewer solar PV systems will be permitted on our distribution networks.  Without research projects like the other eight industry-researcher collaborations in ARENA's R&D funding round, what other innovative technologies will Australia miss out on

Some Awesome Examples of Current ARENA R&D:

ARENA & Industry Funded: Drones that survey solar farms

ARENA & Industry Funded: Integrating solar thermal into alumina processing

ARENA & Industry Funded: Batteries & Solar PV into Apartment Buildings

If my project, and others like it, aren’t part of the “Innovation” agenda that the government wants, part of its “Ideas Boom”, part of the technology driven future of the Australian economy where it produces new ideas of global relevance, I don’t know what is.

If these projects aren't part of the "Ideas Boom"... I don't know what is!

We simply have a government that is demanding innovation on one hand, but clearly saying “just not for renewables” on the other.

What do we do about it?

Across Australia, our renewables powered people groups are springing into action, and our universities aren't letting this go down without a fight:

Solar Citizens has made it easy for you to tell your local parliamentarian to stop this madness

GetUp has put together a #saveARENA campaign where you can share your story about working on an ARENA funded project to your social media accounts.

I want YOU to join with me to #saveARENA

So I want YOU, to join with me, to tell our Parliament to #saveARENA. The legislation to cut $1B from ARENA is set to be introduced to Parliament early next week as part of an “omnibus bill”, so we must act now.

Let me be clear: There has never been a more critical moment for the future of renewables in Australia, or for our potential to be a leader in renewable energy technologies throughout the Asia-Pacific, and more broadly, across the globe.

Renewably Yours,

Dr. Nick Engerer


Be sure to retweet my twitter video:

And Follow #saveARENA on Twitter:

 

 

/*