# Webresponder + Click-to-Call — Project Runbook

> Last updated: 2026-06-12. This is the single source of truth for operating and
> extending the NS webresponder / click-to-call widget service. Read this first
> if you're returning to the project after time away.

---

## 1. What this project is

A small Flask app that powers two customer-facing features for PressOne, both
driven by a per-customer config file (`domains.json`):

1. **SMS Directions (webresponder)** — NetSapiens calls a webhook when a caller
   reaches a treatment; the app texts the caller back hours/address info.
2. **Click-to-Call widget** — an embeddable `<script>` that drops a "Call Us"
   button on any website. A visitor enters their phone number, and NetSapiens
   originates a callback that bridges them into a call queue.

A Zoho Form lets CSRs add/update customer configs without server access (posts
to `/update-domain`).

---

## 2. Key concept: the `id` config key

Each entry in `domains.json` is keyed by a free-form **`id`** of the form
`<api_domain>-<suffix>`, e.g. `pressone.net-sales`, `pressone.net-sms`. This
decouples the config from the NS domain so one NS domain can have multiple
widgets/SMS numbers/queues.

- `api_domain` is an explicit field inside each entry (the real NS domain).
- All routes look up config by `id`, never by the NS `AccountDomain`.
- `_DOMAIN_RE` in `app.py` allows letters/digits/dots/hyphens, so the
  `domain-suffix` format validates without changes.

> Historical note: configs used to be keyed by the raw NS domain. The
> 2026-06-11 "config ID refactor" (commit `19cb0e3` lineage) replaced `domain`
> with `id` everywhere. Design + plan docs are in `docs/plans/`.

---

## 3. Architecture / infrastructure

| Thing | Value |
|---|---|
| App | Flask, served by gunicorn (`-w 2 -b 127.0.0.1:5100`) |
| Server | `root@ns-scripts.pressone.net` (hostname `ponjsecdev1`) |
| App path on server | `/opt/ns-scripts/webresponder/` |
| venv on server | `/opt/ns-scripts/webresponder/venv/` (gitignored) |
| systemd unit | `ns-webresponder.service` (`/etc/systemd/system/`) |
| Reverse proxy | **Caddy** (NOT nginx), config in git at `server-config/Caddyfile`, symlinked to `/etc/caddy/Caddyfile` |
| Static file server | nginx Docker container (`ns-scripts`), serves `/opt/ns-scripts` read-only at :8081 |
| Public host | `https://ns-scripts.pressone.net` |
| Git repo | `https://github.com/shripald/netsapiens-scripts` |
| Local (source of truth) | `~/ClaudProjects/Netsapiens-Scripts/` on the Mac |

Caddy routes `/webresponder*`, `/widget.js*`, `/click-to-call*`,
`/update-domain*`, `/update-sms*` → `localhost:5100`; everything else →
`localhost:8081` (static). **New app routes must be added to
`server-config/Caddyfile`** or they'll 404 (fall through to the static server).

---

## 4. NetSapiens API details

- **Outbound callback (click-to-call):** legacy API
  `POST https://ucportal.pressone.net/ns-api/?object=queuedcall&action=create`
  multipart form-data: `format=json`, `uid=<queue>`, `queue=<queue>`,
  `destination=+1XXXXXXXXXX`. Returns **202** on success.
  - **Queue MUST be `EXT@DOMAIN` format** (e.g. `9002@pressone.net`). A bare
    extension (`9002`) silently fails with "Call Queue not found" in NS logs.
- **SMS (webresponder):** v2 API
  `POST .../v2/domains/{api_domain}/users/{user}/messagesessions/{sid}/messages`
  with `type=sms`, `message`, `destination` (digits, no `+`), `from-number`.
- **Auth:** Bearer token in `NS_TOKEN` env var, used for all APIs.
- **SMS auto-provisioning (v2):** when `/update-sms` runs it builds three NS
  objects in the api-domain so an inbound call routes to the webhook. All are
  **idempotent** — a 409 / "already exists" is treated as success, so
  re-submitting a form to edit the message never errors:
  1. **Create user** `POST .../v2/domains/{api_domain}/users` — Simple User
     `{ns_system_user}`, `service-code=system-webresponder`,
     `email-address=operations@pressone.net` (hard-coded).
  2. **Create dial rule** `POST .../v2/domains/{api_domain}/dialplans/{api_domain}/dialrules`
     — note the **separate `{dialplan}` segment** (dial plan is named after the
     api-domain). Matches `webresponder{ns_system_user}`, application `To Web`, →
     `dial-rule-parameter=https://ns-scripts.pressone.net/webresponder?id={id}`
     (the `To Web` application is what makes the rule call the webhook URL).
  3. **Set answer rule** `PUT .../v2/domains/{api_domain}/users/{ns_system_user}/answerrules/*`
     (catch-all timeframe) forward-always → `webresponder{ns_system_user}`. A
     brand-new user may have no `*` rule, so the code falls back to
     `POST .../answerrules` on a 404.
  - Routing chain: `call → {ns_system_user}@{api_domain} → answer rule
    forward-always → webresponder{ns_system_user} → dial rule match → webhook
    GET ?id={id} → SMS sent`.

---

## 5. Routes (all in `webresponder/app.py`)

| Route | Method | Looks up by | Notes |
|---|---|---|---|
| `/webresponder` | GET/POST | `?id=` query param | NS calls it via Dial Translation URL; returns TwiML-like XML |
| `/click-to-call` | POST | `id` in JSON body | rate-limited per-IP (3/min) and per-id (30/hr); validates US phone |
| `/widget.js` | GET | `?id=` query param | returns branded JS embedding the button |
| `/update-domain` | POST | `id` form field | **Click-to-call** Zoho form; auth via `UPDATE_SECRET`; upserts domains.json |
| `/update-sms` | POST | `id` form field | **SMS** Zoho form; auth via `UPDATE_SECRET`; upserts domains.json **and** auto-provisions NS (user + dial rule + answer rule). Returns a composite `provisioning` status per step. Config is always saved; partial NS failures still return 200. |

Widget embed example:
```html
<script src="https://ns-scripts.pressone.net/widget.js?id=pressone.net-sales"></script>
```

NS Dial Translation webresponder URL:
```
https://ns-scripts.pressone.net/webresponder?id=pressone.net-sms
```

---

## 6. Config: `domains.json`

**Gitignored** (holds live values). Template is `webresponder/domains.example.json`.
The app re-reads it on every request — no restart needed after edits.

Current production shape:
```json
{
  "pressone.net-sms": {
    "api_domain": "pressone.net",
    "user": "2000",
    "ns_system_user": "8001",
    "from_number": "19294873999",
    "message": "Our hours are Mon-Fri 9am-5pm. We are located at 123 Main St, Tampa FL 33601."
  },
  "pressone.net-sales": {
    "api_domain": "pressone.net",
    "queue": "9002@pressone.net",
    "branding": {
      "button_text": "Call Us Now Test!",
      "button_color": "#ff9900",
      "company_name": "PressONE",
      "logo_url": ""
    }
  }
}
```

Two distinct user fields on SMS entries:
- **`user`** — the owner of the SMS number; used by `send_sms()` to POST the
  outbound text. Unchanged from before.
- **`ns_system_user`** — the Simple User created/used by auto-provisioning that
  the inbound call routes through (matches `webresponder{ns_system_user}`). New
  in the SMS-provisioning work; comes from the "NS System User" Zoho field.

`/update-domain` (click-to-call) preserves existing SMS fields (`user`,
`ns_system_user`, `from_number`, `message`) if a submission omits them.
`/update-sms` conversely preserves existing click-to-call fields (`queue`,
`branding`), so the two forms can target the same `id` without clobbering each
other.

---

## 7. Environment variables

Set in `webresponder/.env` on the server (gitignored). Template: `.env.example`.

| Var | Purpose |
|---|---|
| `NS_TOKEN` | NetSapiens bearer token (both SMS + queuedcall APIs). App refuses to start without it. |
| `UPDATE_SECRET` | Shared secret the Zoho webhook must send to `/update-domain`. App refuses to start without it. |

---

## 8. Deploy process (FOLLOW THIS — do not hand-edit the server)

Git discipline: **edit local → commit → push → pull on server → restart.**
Never edit files directly on the server; it caused multiple outages.

On the Mac:
```bash
cd ~/ClaudProjects/Netsapiens-Scripts
# make changes, run tests:
cd webresponder && venv/bin/pytest -q && cd ..
git add -A && git commit -m "..." && git push
```

On the server:
```bash
cd /opt/ns-scripts && git pull
systemctl restart ns-webresponder
journalctl -u ns-webresponder -n 20 --no-pager   # verify
```

`domains.json` is gitignored, so pulls no longer clobber live config.

---

## 9. Tests

`webresponder/tests/` — pytest. `conftest.py` sets dummy `NS_TOKEN` /
`UPDATE_SECRET` so tests import the app. Run from `webresponder/`:
```bash
venv/bin/pytest -v
```
Coverage: SMS route, click-to-call (happy + error + rate limit), widget.js,
update-domain (auth, validation, upsert). ~47 tests.

---

## 10. Gotchas & lessons learned (read before debugging)

- **"Call Queue not found" / no ring:** queue value is missing the `@domain`
  suffix. Must be `9002@pressone.net`, not `9002`.
- **Service runs old code after a pull:** check the systemd unit's
  `WorkingDirectory`/`ExecStart` actually point at `/opt/ns-scripts/webresponder`.
  There was once a duplicate nested tree at
  `/opt/ns-scripts/ClaudProjects/Netsapiens-Scripts/webresponder` and the
  service ran from the wrong one. That nested copy has been removed and
  `ClaudProjects/` is gitignored — don't let it come back.
- **Widget shows "Something went wrong":** the widget posts to an absolute URL
  (`https://ns-scripts.pressone.net/click-to-call`). A relative URL breaks when
  the page is opened as `file://`.
- **Browser shows stale test page:** hard-refresh (Cmd+Shift+R); the static
  container caches.
- **`reset --hard` won't delete gitignored folders:** if you ever need to purge
  `ClaudProjects/` again, `rm -rf` it manually.
- **Auth on git push/pull:** GitHub needs the **token as the password** (account
  password fails). Username can be `shripald` or the email. Use a fine-grained
  token scoped to `Contents: read/write` on `netsapiens-scripts` only.

---

## 11. Security notes

- Never paste tokens/passwords into chat or commit them. Two PATs and the root
  password were leaked into a working session and had to be rotated.
- Remotes must NOT embed the token in the URL
  (`https://user:TOKEN@github.com/...`). Use:
  `git remote set-url origin https://github.com/shripald/netsapiens-scripts.git`
  and a credential helper (macOS keychain on the Mac; `store` on the server
  writes plaintext to `/root/.git-credentials` — acceptable only for a
  read-only single-repo token).
- `.env` and `domains.json` are gitignored and must stay that way.

---

## 12. Outstanding / future work

- Static portal files (`pressone-portal-icons.js`, `pressone-portal-modern.css`,
  `test_widget.html`) live untracked on the server at `/opt/ns-scripts/`. Move
  them into the repo if they should be version-controlled.
- `domains.example.json` is a placeholder template; keep it in sync as the
  schema evolves.
- Test page: `https://ns-scripts.pressone.net/test_widget.html`.

---

## 13. Related docs

- `docs/webresponder-how-to.md` / `.html` — end-user/CSR how-to
- `docs/webresponder-reference.md` / `.html` — reference
- `docs/plans/2026-06-11-config-id-refactor-design.md` — id refactor design
- `docs/plans/2026-06-11-config-id-refactor.md` — id refactor implementation plan
- `docs/plans/2026-06-08-click-to-call-widget*.md` — widget design/plan
- `docs/plans/2026-06-02-webresponder-sms-directions*.md` — SMS design/plan
- `docs/plans/2026-06-09-zoho-form-domain-config*.md` — Zoho form design/plan