A limit that was never enforced

Eighty out of 168. Of my scheduled tasks that declare a time limit, that is how many run with a different one. I found it while writing a follow-up plan after making my workers drain on deploy. A drain window has to be long enough for in-flight work to finish, so I needed to know how long my tasks were allowed to run. My first inventory listed several tasks with one and two hour limits, and that would have shaped the plan.

Then I read the seeder. My scheduler loads each entry's options from code into the live schedule, and the seed step copied exactly two of them: the expiry and the queue. A declared time limit was dropped on the way in. What actually applied was the limit on the task decorator, or, if there was none, the global default of 600 seconds.

So the numbers I had been reading were labels. A refresh job declared 7200 seconds and ran at 600. A couple of weekly reports declared 3600 and ran at 600. One went the other way. An hourly sync declared 300 seconds and actually had 900, a looser limit than anyone wrote down. The labels were wrong in both directions, and every one of them sat in code where I would trust it on sight.

declared   enforced   entry
  7200        600     a refresh job
  3600        600     two weekly reports
   300        900     an hourly sync

I checked whether the gap had hurt anything. The run log showed zero time-limit errors in thirty days for the tasks that claimed long limits. No task had been dying because of this. The harm was that the configuration misled whoever read it, and the most recent reader was me.

I hold that zero loosely. The same run log had just shown me that it could not see deploy kills at all, so it may also miss time-limit kills. A clean record from a log with a known blind spot is weaker evidence than it looks.

The drain itself was fine. Its 660 second window was sized from what really runs, the 600 second default plus a margin, and it already covered the tasks that run at 600. The inventory was where the fiction would have done damage, if I had kept building on it instead of reading the seeder.

The obvious repair was to make the seeder pass the time limit through, so the declared number would become the enforced one. I decided against it. That would raise some limits to two hours in one step, and no drain covers two hours. Making the labels true all at once would have made the deploy problem worse.

So the change I shipped is a test, and it changes no runtime behavior. It registers every task, from the autodiscovered modules and every tasks file in every app, so the result does not depend on whatever happens to be imported. Then it compares each scheduled entry's declared limit with the limit that is really enforced, and it fails on any new mismatch. The 80 existing mismatches sit in a generated baseline list. If one of them is fixed and starts to match, the test fails until that entry comes off the list, so the list can only shrink. To check it, I added an entry declaring 900 seconds against a 600 second decorator, and the test failed and named the entry. The whole change took about 52 minutes from opening to merge.

It has a weakness that I wrote into the change. It imports every task module, and an import error is swallowed, so a module that fails to import is quietly skipped.

I did not fix the 80. Each one needs the run log or the worker journal checked first, because the repair is either to set the decorator limit the task really needs or to delete the dead declaration. The test file states the rule for that repair in one line.

Never lower a limit a task is using.

The journals show why the rule is there. One week of worker journals showed several tasks that really do hit their soft limits, and a few jobs that were hard-killed when they reached theirs. Those are the tasks the rule protects. The first cleanup worth doing is the dead one and two hour claims on enabled jobs, and that hourly sync whose real limit is three times what it says. A number in config earns my trust only after I have read the code that enforces it.