First day after calculation (UTC or Europe/London)

[peterdudfield]

Just want to check following on from #20 that the first day after calculation is made roughly at 10.30 UTC not Europe/London time? For some reason i thought it was local time

Small code to show its UTC

from pvlive_api import PVLive
from datetime import datetime
import pytz

pvl = PVLive()

start=datetime(2023, 4, 20, tzinfo=pytz.utc)
end=datetime(2023, 4, 21, tzinfo=pytz.utc)

data = pvl.between(start=start, end=end, entity_type="gsp", entity_id=0,
                            dataframe=True,extra_fields='updated_gmt')

[JamieTaylor-TUOS]

Hi @peterdudfield,
I think in the past the day+1 run has been triggered at 10:30 local time, but I confirm we now trigger it at 10:30 UTC. This is subject to change though (e.g. if one or more data suppliers changes their release cycle), so a more reliable way to distinguish between retrospective updates would be to consider the now-available updated_gmt field.
Best
Jamie

[JackKelly]

I've been pulling PV_Live data every 15 minutes since Tuesday evening (and saving every "pull" separately).

Below is a plot of the changes in the data:

All subplots share the same x-axis. The x-axis is the datetime (UTC) that I polled the PV_Live API.

In the top subplot ("Changes to PV_Live National estimate"):

• The y-axis is the datetime (UTC) that the estimate describes. (as in: "PV_Live estimates that national PV generation is x MW at this datetime")
• Purple indicates a change relative to the previous pull.
• The "south west" edge of the diagonal line (from the left to the bottom right) shows the initial PV_Live estimates.
• The "dotted" sections of the diagonal line shows nighttime (where the initial estimate is 0 MW, and that estimate doesn't change with subsequent updates).

The three lower subplots show how the PV_Live national estimate for 9:00, 12:00, and 18:00 change over subsequent pulls.

If I've understood correctly, some conclusions are:

• Updated estimates are ready to download at 23:00 UTC and 11:00 UTC (even though the updated_gmt might say a time shortly before 23:00 and 11:00 UTC)
• The update at 23:00 UTC is tiny
• Daytime values are updated multiple times within the first 3 hours after the initial estimate

Does that sound about right?

[JamieTaylor-TUOS]

@JackKelly That's exactly what I would expect to see based on our scheduling and matches our recommended polling cycle (now included in the README)

Great plots - excellent way to visualise the impact of retrospective revision. It'd be interested to see something similar aggregated over more days, correcting for capacity and considering seasonality. I know I still have a similarly-themed email from you to follow up on - hoping to get to this next week!

[JackKelly]

It'd be interested to see something similar aggregated over more days

Here's the same plot, but this time with a week of data collected from PV_Live (this latest plot seems to show exactly the same behavior as observed above):

I'll keep collecting data for a little longer...

[JamieTaylor-TUOS]

@JackKelly @peterdudfield

Picking up from email thread about simulating historical PV_Live outturns using only the intraday/near-real-time sample...

I've uploaded some historical simulated intraday PV_Live outturn estimates here:
https://drive.google.com/drive/folders/1IRt4NXV3SKq-z-7r83w2kcWv4zf0U1Bt

And done some brief sense-checking/analysis here:
https://github.com/SheffieldSolar/PV_Live-API/tree/retrospective_revision_analysis_202306/analysis/retrospective_revision

Preliminary conclusions:

• The analysis suggests that, post inclusion of PVOutput.org sample data, the intraday PV_Live outturn estimates at midday are subsequently revised up by on average 125 MW.
• If we consider all half-hours, the intraday estimates are subsequently revised up by on average 30 MW
• The largest retrospective upwards revision was 1490 MW at ~midday on 2021-06-04
  • Not clear why, warrants further investigation
• The largest retrospective downwards revision was -1414 MW at ~midday on 2022-08-25
  • Not clear why, warrants further investigation
• The retrospective revision is greater and more biased in Spring and Autumn

[JackKelly]

Awesome, thank you so much @JamieTaylor-TUOS!