April 14

How to Generate an AerisWeather Forecast Accuracy Report

At AerisWeather, we understand the importance of having accurate forecasts – which is why we offer our unique, proprietary forecast data to our clients. The importance and specific degree of forecast accuracy will depend heavily on use case and industry needs. Errors within forecast accuracy can come from a number of sources, including data sources available at the location, geographic features of or around the location, time of year, lead time of interest, and the complexity of typical weather in the region.

In the event that you’d like to check the accuracy of our forecast data, this blog post will act as a guide to creating your own forecast accuracy report. Provided you already have your weather data available, we estimate the working time of this report to be about five hours. Note that as we walk through this, we’ll be using JavaScript, but it is possible to run forecast accuracy reports using other languages as well.

Gather the Following:

Before implementing a forecast accuracy querying schedule, you will need to collect the following data:

  • The locations you want to test: We recommend locations that are relevant to your business application.
  • The locations of the ICAO stations that will be used for observational data: These should be as close as possible to your forecast accuracy locations.
  • Weather data for a range of at least 3 to 6 months: This is the most important factor for an accurate report. A minimum of 2 months’ data may be sufficient if the area of focus hasn’t seen severe weather within that time range. However, severe or abnormal weather in your location of concern can skew the forecast accuracy, which is why we recommend a longer period of data gathering.

Start Querying:

After all the necessary information is gathered, you can start making two types of queries: Forecast and observational. Each will contain the same information, but the difference in the values returned will be the forecast accuracy report data needed.

Here’s an example of a forecast query response:

url = “https://{}/forecasts/{}?client_id={}&client_secret={}&fields={}&filter={}”.format(
place.name,
self.aeris_api_id,
self.aeris_api_secret,
“,”.join([
“periods[].dateTimeISO”,
“periods[].tempC”,
“periods[].humidity”,
“periods[].pressureMB”,
“periods[].precipMM”,
“periods[].sky”,
“periods[].pop”,
“periods[].windDirDEG”,
“periods[].windSpeedKPH”
]),
“1hr,precise”
)

First, a forecast query will be made for the chosen location (as seen above). We recommend a lead time of 1 to 3 hours. This means a forecast will be made for anywhere from 1 to 3 hours from now, based on your input. Then, when that time occurs, an observational query will be made (see below).

An example of an observations archive query response:

url = "https://{}/observations/archive/{}?client_id={}&client_secret={}&fields={}&from={}".format(
    self.host,
    place.name,
    self.aeris_api_id,
    self.aeris_api_secret,
    ",".join([
        "periods[].ob.dateTimeISO",
        "periods[].ob.tempC",
        "periods[].ob.humidity",
        "periods[].ob.pressureMB",
        "periods[].ob.precipMM",
        "periods[].ob.sky",
        "periods[].ob.windDirDEG",
        "periods[].ob.windSpeedKPH"
    ]),
    "-6hours",
)

Second, the two above queries will be consistently run for a period – preferably of 3 to 6 months – as detailed earlier. We recommend storing the data received during this timeframe in a database. The comparison between forecasted values and observed values can now be made, such as standard deviation or boxplots.

Note: For the purpose of forecast accuracy reports, observations is used as it reports the raw data from the closest weather station. This is the data most relevant for calculating forecast accuracy. For applications requiring current conditions outside of forecast validation, we strongly recommend using the conditions endpoint, as it interpolates observations with forecast data to provide spatial and temporal data.


Things to Consider:

The biggest factor in preventing forecasts from being used in the report is observational data that has unexpected gaps. This can occur for any number of reasons but is most commonly caused when a station stops reporting data. The below pseudocode will check for any data gaps that may have occurred.

# NOTE - all data is pulled from your compiled database at this point, not the Aeris API for each station
  for each forecast
    for each timestep
      ob = get_observation(station, timestep)
      for each dataset
  
        # if there are any null ob.dataset values, we throw the associated
        # timestep.dataset value out too.
        
        # convert sky cover to oktas and wind direction to 45-degree cardinal directions
                
        # assuming we have both a timestep.dataset and ob.dataset value
        # at this point, you can calculate our error and add it to the
        # super dictionary used to build the boxplots
        errors[timestep.lead_time][dataset].append(timestep.dataset - ob.dataset)
 
# and at the end we have this super dictionary that looks like this:
errors = {
  1: {                                      # the key is the lead time, "1 hour in the future" in this case
    tempC: [1.2, 1.4, 1.4, 1.6, 0.5, ...],  # the list contains every forecasts error at the 1 hour leadtime relative to the one observation
    ...
  },
  ...  # this goes out hourly to 336 (14 days * 24 hours per day) if doing a full validation, which never happens, usually we pick a handful of lead times
}

Lastly, we recommend converting a few forecast and observations values to make comparison easier:

  • Wind direction: Transform these values to the traditional 45-degree cardinal directions.
  • Sky cover: Sky values within our forecast are 0-100; most observation stations report okta values (0-8). Convert our value to the nearest okta to make data comparison easier.

Wrap-Up

Generating a forecast accuracy report may seem like a daunting task, but we hope this guide serves as a helpful starting point and gives you confidence in AerisWeather’s forecast data accuracy. Reach out to us if you have any questions about implementing additional systems to automatically query data.

Share this post:

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

AerisWeather
{{page_content}}