Skip to content

Watches

A watch runs a handler whenever a source has eligible work, one item at a time, and picks up from the last confirmed effect after a restart. A Linear board, open pull requests or a local queue can be a source; a June workflow, the Ship Agent or the Code Review Agent can be the handler.

import {
  completed,
  defineWatch,
  deferred,
  type Reconciliation,
  type WatchSource,
} from "june/watch";

interface Ticket {
  needsHardware: boolean;
}
declare const tickets: WatchSource<Ticket>;
declare function ship(ticket: Ticket): Promise<{ commit: string }>;
declare function findShipped(
  key: string,
): Promise<Reconciliation<{ commit: string }>>;

export default defineWatch({
  name: "tickets",
  source: tickets,
  idleFor: "1h",
  async handle(item, { effect }) {
    if (item.payload.needsHardware) return deferred("needs a signed device");
    const receipt = await effect("ship", {
      perform: () => ship(item.payload),
      reconcile: () => findShipped(item.key),
    });
    return completed(receipt);
  },
});

examples/watch/in-memory.ts is a complete watch over an in-memory queue.

Items and identity

A source returns pages of WatchItems: { scope, key, version?, payload, title?, url? }, and an opaque cursor for the next page.

  • scope and key are the item's identity. SNO-12 in two Linear workspaces, or PR 12 in two repositories, are different items because their scopes differ. itemId() derives the id the watch stores them under.
  • version is the source's revision of the item. The same version is the same work: an item completed at a version isn't handled again until the version changes. The source decides what goes into it, so a PR source can version by head SHA, and an issue source by a hash of the requirements that ignores comments and status churn.
  • payload is persisted with the item, so it must be JSON, and must hold no secret.

Sources

WatchSource<P>
  namespace     string
  discover      ({ cursor?, signal }) → SourcePage<P>: { items, next? }
  claim?        (item, context) → ClaimResult<P>
  acknowledge?  (item, outcome, context) → void
  check?        (signal) → CheckFinding[]
  • discover reads one page. It must change nothing remotely: discovery runs on every poll, including the empty ones.
  • claim takes ownership just before the item is handled, re-reading what makes it eligible. It can return the item as re-read (claimed), say it left the queue (ineligible), or defer, retry or stop it for a person. Read-only sources leave it out.
  • acknowledge reports the outcome back, such as moving an issue to review.
  • check makes read-only checks for june watch --check.

namespace is saved with the watch. Pointing a saved watch at a source with another namespace is an incompatible change.

Handlers and outcomes

A handler gets the item and a context: an AbortSignal, the durable attempt identity, effect() and link(). It returns one of:

OutcomeMeans
completed(receipt?)Done at this version.
deferred(reason, { retryAt? })Blocked by something outside the watch. Other items go ahead; it is looked at again when its version changes or at retryAt.
retryable(reason, { retryAt? })Worth trying again, after retryAt or the watch's backoff, up to retry.maxAttempts.
terminal(reason)Will never succeed at this version.
needsAttention(reason)Only a person can move it on.

A handler can be called again for the same attempt after a restart, so anything it does outside the process goes through effect().

Effects

import type { EffectContext, Reconciliation } from "june/watch";

declare const context: EffectContext;
declare function postComment(signal: AbortSignal): Promise<{ id: string }>;
declare function findComment(
  signal: AbortSignal,
): Promise<Reconciliation<{ id: string }>>;

// perform returns the receipt; reconcile answers { status: "applied", receipt },
// { status: "missing" } or { status: "unknown", reason }.
const receipt = await context.effect("comment", {
  perform: (signal) => postComment(signal),
  reconcile: (signal) => findComment(signal),
});

Effects are journaled by key for each item version. The intent is recorded before perform and the receipt after it, so once a key has a receipt it is returned without performing again.

When perform throws, the effect may still have happened: a timeout, a dropped connection or an abort can come after the service applied the request. So the key stays unknown, and the next call reconciles first: an applied effect is adopted, a missing one is performed, and an unknown one stops the item for a person. The same happens when the process dies part-way. Throw EffectNotAppliedError only when the effect certainly didn't happen, such as a request the service refused: that frees the key to be performed again without reconciling.

Keys are scoped by phase, so a source's claim, the handler and acknowledge never share one, whatever names they pick. Claim and handler keys last for the item version, across attempts: a PR a failed attempt opened is adopted by the next. Acknowledgement keys belong to one report: the attempt whose outcome is being reported. The host records the outcome before reporting it, and a report that a crash or a failed acknowledgement interrupted is finished, with the same outcome and the same keys, before the item is handled again.

memoryEffects() is an in-memory effect context with the same rules, for calling a handler or source directly in a test or a script.

link({ kind, id, url? }) records a reference to downstream work, such as a Ship task, a session or a pull request.

Running a watch

The june watch command runs a definition file. From TypeScript, createWatchHost does the same:

import { createWatchHost, type WatchDefinition } from "june/watch";

declare const watch: WatchDefinition;

const host = await createWatchHost(watch, { stateRoot: ".june/watch" });
await host.drain({ schedule: false }); // work until the queue is idle, then return
const controller = new AbortController();
await host.run({ signal: controller.signal }); // or keep going until aborted
await host.close();

A host works one item at a time:

  1. A pass pages through the source, enrolling each page before asking for the next.
  2. The next item is the first eligible one in the source's order. An attempt a stopped host left under way comes first, even if the source no longer lists it.
  3. An attempt claims the item, handles it and acknowledges the outcome.
  4. After every attempt the source is read again at once, so work that arrived while draining is found straight away.
  5. A pass with nothing actionable stores an absolute next wake: the end of the pass plus idleFor, or sooner if a retry falls due first. An empty queue and a queue of blocked items both wait the same idleFor. A host restarted before that wake waits for it; one restarted after it polls at once.

Discovery and inspection call no model, so an empty poll costs one pass over the source and nothing else.

OutcomeThe item is
completeddone at this version
deferredskipped until its version changes, or until retryAt; other items go ahead
retryableretried after retryAt or an exponential backoff; after retry.maxAttempts, held as exhausted, and the source's acknowledge is told it needs a person
terminalheld, and not retried at this version
needs_attentionheld until a person acts or its version changes

A handler or source that throws counts as retryable, and an effect whose outcome can't be established as needs_attention. A claim that finds the item has left the queue defers it until the next idle wake without counting an attempt. A source that can't be reached throws SourceUnavailableError, optionally with the provider's retryAt, from discover, claim or acknowledge. That is a wait for the whole watch, not a failure of the item: the drain stops, no attempt is charged, an unfinished claim or report resumes later, and the host waits for the hint or its own backoff. Every retry time is at least retry.initialBackoff away, whatever the hint says.

Pass clock: manualClock() to run a watch on a clock that only moves when told to, so an hour of idle time takes no time in a test.

Operating a watch

watchStatus(stateRoot, name) and watchHistory(stateRoot, name) read a watch from any process, without taking it or calling anything remote. pauseWatch, resumeWatch and stopWatch write the watch's control.json, which the running host reads between attempts and while it sleeps (every controlPollMs, one second by default): a pause lets the attempt under way finish and starts no other, a resume polls at once, and a stop exits after the attempt under way. The CLI has the same commands.

Recovery

A watch can be killed at any moment: before a claim, after one, while a handler runs, after a service accepted an effect but before its receipt was written, or after everything was acknowledged. The next host to start picks up from what is on disk:

  1. The interrupted attempt comes first, before any other item is dispatched and whether or not the source still lists the item, since its own claim may have taken it out of the queue. It resumes as the same attempt, with the same identity, so its journaled effects are adopted.
  2. Each effect with no receipt is reconciled through the adapter's reconcile, which looks at the service: an effect that happened is adopted, one that certainly didn't is performed, and one the service can't vouch for either way is recorded as unknown. A lookup that fails (a timeout, a 500) is no answer at all: the effect stays unrecorded and the item is retried to look again, or the whole watch waits if the provider threw SourceUnavailableError.
  3. An unknown outcome stops only its item. The item is held as needing attention, with the effect named in its reason, and the other items go on. It isn't retried until a person acts or its version changes.
  4. A report that was under way is finished with the outcome recorded before it started, and the handler isn't run again.

history.jsonl records each step (attempt.started, attempt.resumed, effect.intended, effect.applied, effect.adopted, effect.unknown, effect.reconcile-failed, report.failed, attempt.ended), so the decisions a restarted host made can be read afterwards. Recovery never replays a June run's snapshot to repeat an effect: the journal and the service are the only evidence.

What this guarantees is at-least-once attempts with effects that happen once, as far as the service can be asked about them. That rests on the adapter:

  • reconcile has to find an effect by something the effect carried, such as a marker in a comment, a branch name or a claim reference. Claim and handler effects last across attempts, so name their evidence after the item and its version (attempt.itemId and attempt.version), not attempt.id, which changes with each attempt. Answer unknown only when the service can't say; a lookup that fails should throw, and the item is retried to look again.
  • A host only coordinates with processes sharing its state directory. When another host, or a person, can write to the same service, the source's claim has to re-read ownership and stand down on evidence that someone else holds the item.

State and ownership

Each watch keeps its state under its own directory in a state root:

<state root>/<name>/
  state.json       the watch: its identity, discovery pass and cursor, next wake, and every item
  state.json.bak   the state before the last write
  history.jsonl    what happened, one event per line
  owner.lock       which process runs the watch
  control.json     what an operator asked: pause, resume, stop
  • Writes are atomic. The state is replaced by renaming a complete file over it, so a process killed at any point leaves one write or the next. A discovery page and its cursor are written together, so the cursor never moves past items that aren't on disk.
  • Nothing is overwritten that can't be read. A state that isn't valid JSON, was written by a newer June, or belongs to a definition whose name, source namespace or config has changed is refused with a message naming the file, and left as it is.
  • One process runs a watch. owner.lock names the process holding it. A second process is refused while that one is alive, and takes the lease over only from a process on the same machine that has died. An owner on another machine is never presumed dead: the lease coordinates processes sharing one state directory on one host, not hosts. When several hosts could act on the same source, the source's own claim has to keep them apart.
  • No secrets and no transcripts. Items hold their source payload and effects hold their receipts, which are capped at 64 KB. Keep credentials in the environment and refer to downstream work with link().

Two watches with different names keep separate state and leases, and run independently.

Definitions

defineWatch({ name, source, handle, idleFor, retry?, config? }) validates the definition and returns it.

  • name is the watch's stable identity among the watches sharing a state directory: lowercase letters, digits and dashes.
  • idleFor is how long to wait after a pass that found nothing actionable: milliseconds or "90s", "15m", "1h", "1d".
  • retry bounds retries per item version: maxAttempts (3), initialBackoff (one minute) and maxBackoff (one hour).
  • config is JSON whose change makes saved state incompatible. Together with the name and the source's namespace it makes up the definition's fingerprint. Keep secrets out of it.