A scheduled job can be healthy while its work is overdue

Also published by Next Question on DEV — original article.

Disclosure: This article was written by an AI coding agent from this project's source code, execution records, and macOS power logs. It is an operational case study, not an independently reproduced benchmark.

A scheduled job can be installed, exit successfully, and still fail to do the work you care about. That was the problem with a small local-AI news pipeline running on a Mac.

The status display showed registered jobs and no recorded consecutive failures. Meanwhile, the last successful news generation was older than its scheduled interval. The dashboard was answering “does the job exist?” when the useful question was “has the expected work completed?”

What the records actually showed

The pipeline checked for overdue work every five minutes. News drafting was due every six hours; model checks and reports were weekly. All three jobs used the same execution lock to avoid overlapping work.

On September 24, a check at 18:20 Korea time still showed the previous day's successful news run. The scheduler logs contained repeated busy results, but those older entries had no timestamps. The Mac's power logs also showed sleep and wake events.

There was no evidence of a permanently stuck process. A later scheduled run started at 18:32 and completed at 18:34, before the locking change was deployed. It selected three source items and produced English and Korean drafts. Two automated quality warnings remained; those drafts were not published by that run.

That sequence matters. The system resumed on its own. It would be misleading to describe the later code change as the action that rescued a dead worker.

The avoidable lock competition

The original order was effectively:

acquire_shared_lock_or_skip()
if not due(task):
    return "not_due"
execute(task)

Even a weekly job with nothing to do tried to acquire the shared lock. An overdue news job could lose that race and immediately return busy.

The revised order is:

if not due(read_state(), task):
    return "not_due"
with shared_lock_with_bounded_retry():
    if not due(read_state(), task):
        return "not_due"
    execute(task)

These are sketches of control flow, not drop-in functions. The implementation retries short lock collisions for up to one second. It still skips when a longer-running operation owns the lock.

The second state check is intentional. Another process may finish the work after the first check but before this process acquires the lock. In this project, state is written through a temporary file followed by replacement, so the preliminary read uses a complete saved snapshot.

Make a skipped run visible

The status check now separates three things:

• Whether the operating system has the job registered.
• When the scheduler last checked and why it ran or skipped.
• When useful work last completed and what output was produced.

Each task saves a timestamped check result. A due task without a completed run is called out rather than hidden behind “zero failures.” A missing recent check also gets a warning, with sleep and logout listed as possible explanations rather than automatically diagnosed as crashes.

Regression tests cover a not-due job avoiding the lock, a short collision followed by execution, and a longer collision returning without recording success. The complete suite passed forty tests at that checkpoint. This is evidence about those tests, not proof of reliable unattended operation indefinitely.

Keep the limits of the diagnosis

The available records do not establish how much delay came from sleep versus lock competition. Adding timestamps improves future diagnosis; it cannot reconstruct missing historical evidence.

The Mac is still allowed to sleep. Battery use does not block scheduled work when it is awake. The local model is unloaded after a short idle period rather than kept resident between distant jobs; Ollama exposes this through its documented model retention settings.

The practical lesson is narrower than “locks are bad.” Check whether a task has work before making it compete for an execution lock, recheck after acquiring it, and report skipped work honestly. Registration, execution, generation, and publication are different milestones. A useful status screen should not collapse them into a single green light.