squeezing into an ever shrinking maintenance window

There once was a down day in store
But the mill wanted just to roll more
For early on Monday
They'd lost nearly one day
Turned up bar had destroyed feed roll door

Rediscovered this comment from June 2009 on the frustration of seeing yet another planned maintenance day cancelled so the plant could stay operational.

This is one of the challenges of supporting computer systems in a market where plant throughput is valued higher than essential maintenance. After all, that's what weekends are for and your systems support personnel don't have a real life anyway, just a second life at work. In the words of one of our managers, "I know weekends are obnoxious but the work has to be done sometime (nudge, nudge, wink, wink)". Another subtle hint.

My wife refers to my periods away from home during the week as meeting up with my other wife/mistress. The server might be a lot like a nagging wife and the fans do whine a bit. She calls in the middle of the night when she hasn't received enough attention. A bit of remote fiddling seems to quieten her. She's getting on in years now. Perhaps her hardware has a seven year itch, coinciding with the time the manufacturers withdraw extended support for her.

There'll be other suitors knocking on the door soon, a fancy blade server might want my attention, or some high end virtualisation rig. Who knows, I might take on their advances. They promise increasing flexibility for essential maintenance tasks and that could mean more time to dedicate to my real wife and family. Just have to make the manager believe that not doing something will lead to a far more obnoxious outcome.

Maintenance periods shrink ever smaller.
The amount of maintenance grows ever bigger.
The only solution is to perform the maintenance continually as the window shrinks to nothing.

the hazards of cumulative programmers

Ever wonder how long an error or omission in a piece of code can go undetected?

I found one in a piece of FORTRAN code last week that had remained undetected for around 23 years. It was based around the assumption that a particular method of processing for a slab of steel was the same every time. But conditions do change and the result of a small change in incoming dimensions coupled with the use of a particular processing selection produced an unfavourable result in the final product. As usual, while it would be justifiable for me to assume that this was one programmer's shortcoming, the real story is more mundane.

Programmer A assumes that the product will always be processed in the same way and determines critical product parameters based on where he thinks the process starts and ends.

Programmers B, C, D, E, F and G work on the same 25000 lines of code over the next 23 years, making unrelated changes that all slightly alter the conditions that programmer A's coding solution held valid under, until such time that, in combination, they result in a truly unfavourable, fuel tanker, freight train collision cum explosion of a bad product outcome.

Scratch many tonnes of finished product and suddenly, programmer H, who has been left to cook with such spaghetti, is caught in the headlights of a large truck being driven by the angry production manager.

So programmer H has to sort it all out over the following week until the "aha" moment arrives. One line of missing code is all it took to wreak havoc. Inserting it is simple but satisfying and all but guarantees some future proofing against a repeat of this carnage.

Imagine an error like that bringing down a bird at the end of NASA's shuttle program. Unthinkable but possible. How many probes have now been lost due to such errors and it aint a short trip to Mars to reload either.