#3420 [GUI] Web search and page reading for the web app's Chat (DuckDuckGo default, SearXNG, Brave, Tavily, Serper, Exa, Perplexity)

Description

## Problem and result

The Chat tab of the lean web app (the serve/Terminal frontend, not the App) has no web access at all: answers are limited to the model's weights, and agent-style use on a headless server had no source-backed option. Upstream shipped web search for the App in 0.2.0; the serve web chat route is still without it.

Before: no search in the web app's Chat, no sources. After: a built-in web search, modeled after how Open WebUI and similar chat tools do it: the model can look the question up before answering, and the answer shows its sources. **Web search is completely off by default**: until the user turns it on in Chat Settings, nothing about this feature sends anything out of the PC. Strata's Chat is deliberately lean, and this feature is built the same way: it adds no tokens until a chat turns search on, and results are budget-capped to a small slice of the model's context window (defaults scale with the context window), so the Chat gains recent information without losing what makes it comfortable for long sessions.

## Related discussion

Replaces #2826: that PR was closed in the pre-0.2.0-beta tracker reset ("closed here does not mean rejected"); after the rebase its stored head commit became unreachable, so GitHub refuses to reopen it and this PR carries the same work on a live base. Upstream's App search (`apps/host`, DuckDuckGo via MCP) does not overlap with this work: this targets the serve web chat, with self-hostable engines (SearXNG) and server-side page reading.

## Approach

Search runs server-side in the serve backend (`serve/websearch.py`, stdlib-only); the browser frontend only configures and renders it. All providers are fetched through the Strata server (a browser could not call them directly because of CORS). API keys stay browser-only (localStorage), like the existing API-key field; **nothing search-related is stored on the server**.

### What it looks like in the UI
- A globe button in the composer (first icon, before the paperclip) toggles search for the next message.
- **Chat Settings** (gear) has a *Web search* section: enable/disable, engine picker, SearXNG URL, API key field, max results, page reading and a page reader mode (see below). All web-search controls appear only when the search switch is on; switching it on expands the whole section and fires the privacy toast.
- When search is on, the results are injected as context before the model's turn, with an instruction to answer from them and cite by `[n]`.
- Under each answer the sources appear as `Sources (DuckDuckGo): [1] … | [2] …` (clickable, engine named, read pages get a `read` badge), and they are included in the "Save this chat as Markdown" export.
- While a search runs the status pill shows the truth instead of "Idle": a globe (instead of the state dot) with `Framing query` → `Searching web + reading pages · DuckDuckGo · 12.4 s`; a failed search reads `Search failed` for 2.5 s. The same status line appears in the answer bubble exactly where "Thinking…" would sit.
- If the engine is busy with someone else's turn (another tab or agent) while this chat searches, the pill shows it as a hint (`… · 1 queued`) instead of hiding it or hijacking the label: one priority function owns the pill (search phase > engine state > queue count > idle), so the 1 s metrics poll and the 250 ms search timer can no longer overwrite each other.

### Query optimizer
Conversational wording ("can you look up what the newest Qwen release from Alibaba is") makes weak search queries, so there is an **"Optimize search query"** toggle (on by default): one fast call to the user's own local model (temperature 0, reasoning off, max 100 tokens) extracts the core search question before the search runs.
- The query stays **in the language the user wrote in**: English stays English, German stays German etc. Obvious typos are fixed.
- If the call fails, is aborted, or returns something empty or rambling, the **raw message is searched instead** - the optimizer can never break a search.
- **Transparency**: whenever what was searched differs from what was typed, the sources line shows `Sources (DuckDuckGo) · searched: "…"`, and so does the Markdown export. An optimized query is never hidden behind a plain-looking result.

### Reading result pages in full
Snippets alone often miss the details, so search can also fetch the top result pages and inject their extracted text (Open WebUI's "Web Loader" idea, stdlib `html.parser` based, which skips nav/footer/scripts, prefers `main`/`article`):
- **"Read result pages"** toggle + **"Pages to read"** (1-5).
- **"Limit page text"** toggle (on by default) with a **"Max characters per page"** field (default 5000). Turning it off sends whole pages; see the budget below for why that is still safe.

### Page reader: direct fetch, or Jina Reader for JS-heavy pages
Plain `urllib` cannot render JavaScript-heavy pages, so there is a 3-tier **Page reader** setting:
- **Direct only (default)**: only the Strata server fetches pages; nothing goes to any third party.
- **Jina as fallback**: direct first; pages that fail (JS walls, blocks) are retried through [r.jina.ai](https://r.jina.ai) (free hosted headless browser, no key needed, Apache-2.0).
- **Jina for all pages**: every page goes through r.jina.ai.
- The base URL is overridable via `STRATA_JINA_URL` (e.g. a self-hosted instance), and failures of either path degrade to a readable combined reason.
- **Privacy is surfaced, not buried**: the dropdown explains per selection, a warning line appears whenever a third party could see the URLs, and turning search on fires a toast that names the engine *and* the reader mode ("… go through r.jina.ai - a third party sees those URLs").

### Context-aware reading budget
How much search text goes into the prompt scales with the model's own context window (`reported_ctx()`), so small-context instances are never overloaded:
- The reading budget is a slice of the window (~3 chars/token): **10 %** normally, **25 %** when the user explicitly turns the per-page limit off; the per-page cap is then derived from the budget instead of a fixed number.
- First-use defaults scale the same way from `/health`'s `max_context` (e.g. 16K ctx → 4 results / 1 page × 3000 chars; 128K → 6 / 2 × 5000; 262K → 8 / 3 × 5000).
- Unknown context falls back to the plain 5000/page default.

### Distinct, persistent search warnings
A failed search no longer fails silently: the reason stays on the message (in the sources line, warning-colored) and in the Markdown export, with a different message per cause:
- **no API key set** → `Brave Search needs its API key (Chat Settings > Web search)` (same for Tavily/Serper/Exa/Perplexity; a *rejected* key gets its own message, incl. Brave's 422)
- **engine unreachable** (e.g. SearXNG not running) → `SearXNG at http://127.0.0.1:8888 is not reachable (…); start it or pick another engine`
- **DuckDuckGo bot check** → DDG's lite endpoint answers Chrome-impersonating UAs with its `anomaly.js` page; the fetcher uses an honest custom UA and, if the bot page still shows up, says so instead of reporting "no results"
- **engine reachable but nothing found** → `no results`
- **unreadable page** → per-result `read_error` (shown as a warning line), never a silent empty string

### Security hardening
- **`/search` sits behind the web app's own-page guard** (`_own_page`): a JSON body is required (415 for form posts) and a foreign `Origin` is rejected (403; `trusted_origins` still allowed), so a random web page cannot make this PC run searches or fetch URLs through it.
- **http(s) only**: the SearXNG base URL and every result URL that gets page-read pass a scheme check (`_http_url`); `file://`, `data:`, `javascript:` and friends are refused with an explicit reason, so page reading can never become a local file reader.
- **Source links render defensively**: only `http(s)` URLs become clickable `<a href>`; anything else shows as plain text, so no `javascript:`/`data:` links from a poisoned result.
- **Prompt-injection hardening**: the injected search context states that titles, snippets and page text are untrusted web data whose embedded instructions must not be followed.
- **LAN-bound warning**: `/health` reports `lan_bound` (server bound beyond loopback, e.g. `0.0.0.0`); turning search on fires an extra toast line warning that queries and engine keys then cross the network in plain HTTP and that devices on it can use the search without an api_key. On the default `127.0.0.1` bind this warning never appears.
- API keys stay browser-only (localStorage); nothing search-related is stored server-side.

### Settings drawer updates
- The drawer was titled **"Sampling"**; since it now also holds web search settings it is renamed to **"Chat Settings"** (gear tooltip: "Chat Settings: Thinking & Web Search"), the close button got a proper "Close" tooltip, and the save toast says "Settings saved".
- The **Reset** button turns web search completely off and restores the first-use defaults: search disabled, engine back to DuckDuckGo, no API key, instead of leaving a stale engine or an active search behind.
- The empty-state privacy note is qualified and split into two fixed lines: "Nothing leaves it. / Web search is off by default - when on, only the search query goes out."
- Reset and Apply now sit fixed under the "Chat Settings" title while the options scroll below; the sampling options are grouped under a restored "Sampling" heading.

### Scroll fix (drive-by)
Auto-follow-to-bottom was broken for turns where content grows past the stick threshold mid-stream (the thinking panel expanding could pin the view at the user's own message). Stickiness is now captured *before* the content grows, so the view keeps following the answer.

### Engines
| engine | needs |
|---|---|
| DuckDuckGo (default, labeled "anonymous - no key needed") | nothing (lite endpoint) |
| SearXNG | a running instance (default `http://127.0.0.1:8888`) |
| Brave Search | API key |
| Tavily | API key |
| Serper (Google) | API key |
| Exa | API key |
| Perplexity (model `sonar`) | API key |

Exa and Perplexity call the same endpoints upstream's App calls (`api.exa.ai/search` with Bearer key, `api.perplexity.ai/chat/completions` model `sonar`, sources taken from `search_results`, falling back to bare `citations` when the plan omits them), so a key someone already has for the App works here unchanged.

### Design notes
- `serve/websearch.py` is **stdlib-only** (`urllib`, `json`, `re`, `html.parser`, `os`); no new dependencies, cross-platform (Windows/Linux/macOS).
- `POST /search` carries its own engine config per request; nothing is stored on the server.
- Page fetches are capped (10 s timeout, 2 MB download); read results use an explicit contract (`content` on success, `read_error` on failure, `read_via` for Jina-read pages) instead of heuristics.
- `/search` has a catch-all so an unexpected error returns a clean 502 with a server-side log line instead of a broken connection.

## Validation

Re-validated today (20.10.) on the rebased branch `web-search` @ `3d05a166`, which contains upstream `main` including `894d1be1` (0.2.1) and `8aaf188a` (user-args/VRAM fix). Environment: Windows 11, RTX 4080 SUPER 16 GB, Ryzen 9 7950X3D, engine 0.1.45 (`MIN_ENGINE` (0,1,45) met), model qwen3.8-flash-next-iq3_xxs, 128K ctx.

**Passed:**
- `python -m unittest serve.test_websearch -v` (the suite now committed in this branch): **18 tests, all green** (scheme guard rejects 15 poison URL types `file:// gopher:// javascript: data: ftp:// dict:// ldap:// sftp:// ssh:// mailto: telnet: jar: chrome-extension: vbscript:` and accepts http(s); reader tier matrix direct/fallback/always incl. Jina rescue on a failing direct fetch and the combined both-paths-fail reason; bad reader values degrade to direct; non-HTML content refused; reading-budget clamps at 10 %/25 % of context with per-page caps derived at 4K/16K/128K; `max_results` clamp; keyed engines: Exa/Perplexity deman