# Installing the worker and the nightly job

Everything else in Crystal 2 runs without root. These two steps do not, because
a systemd unit lives in `/etc/systemd/system` and a system cron entry lives in
`/etc/cron.d`.

Until this is done the worker survives a session but **not a reboot** — it is a
hand-started process, and nothing brings it back.

Run every command from `/var/www/html/crystal/crystal2`.

---

## Step 1 — stop the hand-started worker

**Do this first.** If the service starts while the manual process is still
running, two workers poll the same table and both will claim the same PENDING
run.

    pgrep -af "worker/worker.py"

If that prints a process, kill it:

    pkill -f "python3 worker/worker.py"

Confirm nothing is left:

    pgrep -af "worker/worker.py" || echo "clear"

---

## Step 2 — install the service unit

    sudo cp worker/crystal-worker.service /etc/systemd/system/
    sudo chown root:root /etc/systemd/system/crystal-worker.service
    sudo chmod 644 /etc/systemd/system/crystal-worker.service

## Step 3 — start it, and have it start at boot

    sudo systemctl daemon-reload
    sudo systemctl enable --now crystal-worker

`enable` is what survives the reboot; `--now` starts it immediately.

## Step 4 — verify it is actually running

    systemctl is-enabled crystal-worker     # expect: enabled
    systemctl is-active  crystal-worker     # expect: active
    systemctl status crystal-worker --no-pager

Then watch it think:

    journalctl -u crystal-worker -f

Leave that open, upload an EPG through `new.php`, and you should see it claim
the run. `Ctrl-C` to stop watching — it does not stop the service.

**If it says `active` for a few seconds and then `activating (auto-restart)`,**
it is crash-looping. `Restart=always` hides the failure behind a restart, so
read the actual error:

    journalctl -u crystal-worker -n 50 --no-pager

The two likely causes:

- `Permission denied` on `worker/config.json` — it must be group `www-data`,
  mode 640. Check with `ls -l worker/config.json`.
- `ModuleNotFoundError` — `vendor/` must be readable. The service runs as
  `www-data`, which has no `~/.local` site-packages, so every third-party
  import comes from there.

---

## Step 5 — install the nightly refresh

    sudo cp worker/crystal-nightly.cron /etc/cron.d/crystal
    sudo chown root:root /etc/cron.d/crystal
    sudo chmod 644 /etc/cron.d/crystal

`/etc/cron.d` files are ignored if they are group- or world-writable, which is
why the `chmod` is not optional.

## Step 6 — verify cron accepted it

    sudo systemctl restart cron
    grep crystal /var/log/syslog | tail -5

## Step 7 — test the nightly job now, rather than finding out at 4am

It takes a few minutes; it pulls ~6M rows.

    sudo -u www-data python3 worker/build_cache.py
    sudo -u www-data python3 worker/fit_model.py

Both must be run **as `www-data`**, the same user cron uses. `build_cache.py`
creates a new snapshot directory and swaps it into place, so whoever runs it
owns the result — run it as yourself and the worker may lose write access to
`reference_cache/` and silently pay a 14s index rebuild on every single run.

Watch for these lines:

- `WARNING: report frames empty or absent` — the OTT, OOH and pan sheets will
  read zero. See `HANDOFF.md` §7 item 7.
- `carried calibration.json across, marked stale` — expected. `fit_model.py`
  clears the stale flag by re-measuring.

Afterwards, confirm the worker can still write into the snapshot:

    ls -ld reference_cache          # expect: drwxrwsr-x ... www-data

The `s` is the setgid bit and the group must be `www-data`.

---

## Rolling it back

    sudo systemctl disable --now crystal-worker
    sudo rm /etc/systemd/system/crystal-worker.service
    sudo rm /etc/cron.d/crystal
    sudo systemctl daemon-reload

Then restart the worker by hand if you still need it running:

    cd /var/www/html/crystal/crystal2 && nohup python3 worker/worker.py &

---

## Day-to-day afterwards

    sudo systemctl restart crystal-worker    # after changing engine code
    journalctl -u crystal-worker -f          # live log
    journalctl -t crystal-nightly --since yesterday    # last nightly run

The restart is belt-and-braces: `run_estimation()` already drops `engine.*`
from `sys.modules` each run, so the worker picks up engine changes without one.
A change to `worker.py` itself still needs the restart.