Manufacturing Hindsight: Six Years of Weather, Sixteen Years of Sunshine, Five Years of Prices

• 6 yearsof local weather records
• 16 yearsreconstructed sunshine record
• 5 yearsof reconstructed wholesale prices

You can’t buy hindsight, but you can manufacture a decent imitation from public weather records. Along the way you may also discover that your own house has been hiding dinner from you for months. Mine had.

The last post ended with a problem. My billing history, accurate to the last cent, was the wrong evidence for choosing my next plan. It recorded how this house behaved under the old plan, including the roughly 40 per cent of potential summer generation it deliberately declined to produce. That was curtailment: not harvesting your own sunshine on afternoons when exporting it would cost you money. The missing evidence, an honest picture of what this house would generate and use with no plan’s fingerprints on it, had to be built from public data and checked against my own meters.

This post is the build: three layers of evidence, each with one job.

Layer one: six years of weather

The first job is generation: what could the panels have produced, day by day, if nothing had held them back?

You can’t measure that directly, because the meter records what the panels actually produced. But you can estimate it. Solar generation is weather arithmetic: how much sunshine fell, how warm it was (panels lose efficiency in heat), and what your panels are rated to do. It’s the same logic as working out how much rain your roof could have caught from the rainfall records and the size of your roof. You don’t need to have caught the water to know it fell.

So the first layer is six years of public weather records for my corner of Melbourne, converted day by day into the generation of a 19.4 kW array that never holds back. Six years rather than one, because one summer is a sample of one. A freakishly cloudy February could swing a plan comparison single-handed. Spread across six years, no individual season shouts the others down.

Layer two: sixteen years of sunshine

What if the six years are themselves the fluke? Weather varies on scales longer than any household notices, and a six-year window could sit on the sunny or gloomy side of normal. Every number in the model would inherit the tilt.

So the second layer is a reconstructed sunshine record, sixteen years long, with one job: to say whether the six modelled years are ordinary. It’s the difference between judging a batsman on six innings and checking those innings against the career average. The six years sit within the ordinary range. Not a golden run, not a dismal one, just years. That doesn’t make the model right, but it does mean it isn’t built on a weather fluke.

The case of the dinner-time gap

Generation is only half a ledger. To price a plan you also need to know when the house uses power, because modern plans care enormously about timing. This is where the build turned into detective work.

Two independent witnesses watch this house: the meter the retailer bills from, which sees everything crossing between house and grid, and the monitoring built into the solar and battery system, which watches from the inside. Cross-checking them was meant to be a formality.

They disagreed. Not dramatically, and not randomly: a modest, stubborn gap between what the meter said the house drew and what the internal monitoring could account for, opening at about the same time every evening and closing a little later. The gap kept dinner hours.

The culprit took an embarrassing amount of investigation for something sitting in plain sight: the cooker. Its circuit runs from the switchboard in a way the internal monitoring never sees, so every meal cooked here was invisible to my fine-grained data. The meter saw it. The record I was about to build a model on had a hole exactly the shape of dinner.

The repair is imputation, a respectable statistical word for careful filling-in. Where two honest witnesses disagree, the difference is the testimony of whatever only one of them saw. Subtract the internal monitoring’s account from the meter’s, evening by evening, and the cooking load emerges from the gap.

Chart to come

This matters because early evening is when this house, like most, draws hardest from the grid, and when prices on most plans are at their daily highest. A model blind to cooking would have flattered every candidate plan at the worst possible hour.

The lesson travels beyond cookers. Household data has holes. They sit wherever wiring history put them, not where your analysis would prefer, and you find them by making your data sources argue.

Layer three: five years of prices

The third layer answers the money question: what would each candidate plan have charged?

The first two layers supply the pattern of use. The prices come from published wholesale market records, the half-hourly prices at which power is traded, which are public record. From those the analysis reconstructs five separate years of market conditions, so each plan is priced under five years rather than the one season I happened to live through.

The job is the same as the weather layers’: don’t let a single year pick the winner. A plan that shines in a jumpy year may look ordinary in a calm one. Why prices did what they did in any year is a question this series leaves alone. The comparison only needs five different years to get a vote.

Where the model earns its keep

An untested model is an opinion with charts.

This one can be tested, because not every day is contaminated. On plenty of days nothing held the panels back, so the meter recorded them running unconstrained. On those days the model’s would-have-been figure ought to match the meter’s did-happen figure.

It does. Individual days scatter, but the modelled figures track the metered ones closely, with no systematic lean towards optimism or pessimism. That last part is what matters.

Chart to come

The goal was never a perfect model, which public weather records can’t deliver. The goal is an unbiased one, whose errors don’t pile up on one side. My working test was blunt: the model is good enough when the plan ranking stops flipping as the weather changes. When plan A beats plan B across cloudy years and clear ones, calm price-years and jumpy ones, more polish won’t change the decision, and the decision is the point.

The simpler version

None of this needed special resources: public weather records, published prices, my own meter data and a home automation system doing the bookkeeping. The expensive ingredient was stubbornness. The method shrinks gracefully. If your version is a spreadsheet and a free weekend, the same three habits transfer whole.

Judge your solar system by what it could have produced, not just what it did. Even one year of public weather observations laid against your meter history will show whether your own behaviour has been hiding capacity.

Make your data sources argue. Your retailer’s usage data against your monitoring app is the household version of my quarrelling meter and monitoring, and any stubborn daily gap is telling you something.

Never let one year decide. A comparison tested against a single season is one freak summer from the wrong answer.

The bills were a biased witness. This is the fairer one: generation the house was always capable of, consumption with the dinner-shaped hole repaired, and five years of prices so no single season does the choosing. Manufactured hindsight isn’t the real thing. In one respect it’s better, because real hindsight only ever tells you what happened under one plan, one season, one set of habits.

What it can’t yet do is settle anything, because knowing what you could sell means nothing until you know what it cost you to make. That’s next: same wire, same electrons, three very different receipts.

The spreadsheet doesn’t care what you hoped the answer was.