# Remote MCP `fetch` wire shape

Endpoint: `https://www.web2md.net/api/mcp` (Streamable HTTP, no API key).

## No AI in conversion

The `fetch` tool does **not** use an LLM, embeddings, or a hosted “reader” model. Conversion is deterministic:

1. HTTP GET of HTML (no headless browser on the default remote path).
2. Optional main-content extraction (scoring + semantic tags).
3. HTML→Markdown via Turndown + GFM plugins, plus fixed post-processing (link absolutization, dedupe, noise cleanup).

The playground at `/#playground` streams `POST /api/fetch-stream`, which runs the **same conversion core** as the `fetch` tool (`startFetchRun` / `finishFetchRun`) with the same arguments documented below — the Markdown is identical to `tools/call` output. The NDJSON event sequence is `phase` (`fetch`, `convert`) → zero or more `chunk` events (`{"type":"chunk","markdown":"…"}` — contiguous slices whose concatenation equals `done.response.markdown` exactly) → `done` (full `McpResponse`) or `error`.

Clients call JSON-RPC `tools/call` with `name: "fetch"`. The tool returns a single text content item whose body is a JSON string of the shapes below.

## Request arguments

| Field | Default | Notes |
|---|---|---|
| `url` | required | Target page |
| `main_content` | `false` | Also auto-enabled for article/product pages |
| `include_images` | `false` | Auto-enabled for product pages |
| `include_links` | `true` | |
| `include_tables` | `true` | |
| `include_comments` | `true` | Forum comment appendix on forum-classified pages |
| `keep_header` | `false` | |
| `favor_precision` / `favor_recall` | `false` | Main-content threshold knobs |
| `max_length` | omit | Truncate Markdown; appends `[truncated]` |
| `only_with_metadata` | `false` | Fail unless title + published_date |
| `respect_robots` | `false` | Honor origin `robots.txt` Disallow |
| `excerpt_only` | `false` | Short main-content preview (ignored when `format` set) |
| `if_none_match` | omit | Prior fingerprint/ETag → `unchanged: true` |
| `format` | omit | Structured output (see below) |
| `attr` | `[]` | Required for `format=attributes` (`selector:attr`) |

Example tools/call:

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "fetch",
    "arguments": {
      "url": "https://example.com/article",
      "main_content": true
    }
  }
}
```

## Default response (Markdown path)

`format` omitted. `markdown` is the cleaned page; metadata fields are flattened onto the same object.

```json
{
  "url": "https://example.com/article",
  "markdown": "Test Article\n============\n\nThis is the article body…",
  "title": "Test Article",
  "description": "A description.",
  "published_date": "2024-01-01T00:00:00Z",
  "language": "en",
  "excerpt": "This is the article body with enough text…",
  "page_type": "article",
  "extraction_quality": 0.82,
  "fingerprint": "a1b2c3d4e5f60718",
  "word_count": 42,
  "char_count": 210,
  "raw_char_count": 1840
}
```

Optional fields (`author`, `image`, `headline`, `site_name`, `keywords`, `categories`, `canonical_url`) appear only when present.

### Unchanged short-circuit

When `if_none_match` matches `fingerprint` (bare hex, `"hex"`, or `W/"hex"`):

```json
{
  "url": "https://example.com/article",
  "markdown": "",
  "unchanged": true,
  "fingerprint": "a1b2c3d4e5f60718",
  "title": "Test Article",
  "raw_char_count": 1840
}
```

HTTP clients may also send `If-None-Match` and receive **304** with an `ETag` header (same fingerprint).

### Excerpt-only preview

`excerpt_only: true` returns a short preview (metadata `excerpt` or ≤500 chars of Markdown), skips iframe inlining and comment/product appendices.

## Structured `format` responses

`markdown` is always `""`. Payload lives in `result`. Metadata/signals are still attached for quality/fingerprint.

| `format` | `result` shape |
|---|---|
| `links` | `{ url, text }[]` |
| `images` | `{ src, alt? }[]` (img, lazy, srcset, CSS bg, OG/Twitter/JSON-LD) |
| `attributes` | `string[]` from `attr` specs |
| `video` | `{ url, source?, title?, thumbnail?, duration? }[]` |
| `audio` | `{ url, source?, title?, duration? }[]` |
| `product` | ProductEntry (JSON-LD Product → variants) |
| `menu` | MenuEntry (JSON-LD Menu → sections/items) |
| `branding` | BrandingProfile (inline `<style>` colors/fonts/headings) |

### `format=product`

```json
{
  "url": "https://shop.example/widget",
  "markdown": "",
  "result": {
    "name": "Acme Widget",
    "brand": "Acme",
    "sku": "W-100",
    "variants": [{ "price": "19.99", "currency": "USD" }]
  },
  "page_type": "product",
  "fingerprint": "…"
}
```

### `format=menu`

```json
{
  "url": "https://cafe.example/menu",
  "markdown": "",
  "result": {
    "name": "Lunch",
    "currency": "USD",
    "sections": [
      {
        "name": "Mains",
        "items": [{ "name": "Burger", "price": "12", "currency": "USD" }]
      }
    ]
  }
}
```

### `format=branding`

```json
{
  "url": "https://brand.example/",
  "markdown": "",
  "result": {
    "color_scheme": "light",
    "colors": [{ "value": "#111111", "count": 3 }],
    "fonts": ["Inter", "sans-serif"],
    "background_colors": ["#ffffff"],
    "heading_sizes": [{ "selector": "h1", "size": "32px" }]
  }
}
```

## Remote limits

The public endpoint is **agent-context first**: Markdown + metadata, not a full replacement for the local MCP.

- **No iframe inlining** on hosted (set `WEB2MD_REMOTE_IFRAMES=1` on your own deploy to opt in).
- **Blocked structured formats** on hosted: `video`, `audio`, `branding` — use `links`, `images`, `product`, `menu`, `attributes`, or default Markdown.
- **No default output cap** — long pages convert completely; set `max_length` to truncate (`[truncated]` marker appended).
- Origin HTML above **4M characters** is rejected (use the [local MCP](https://github.com/yingkitw/web2md) for huge pages).
- Per-IP rate limit: **60** MCP requests/min (see response `x-ratelimit-*` headers).
- Serverless wall clock: **~10s** on Hobby, up to **60s** on Pro when `WEB2MD_PRO_BUDGET=1` is set.
- Timeouts and budget errors include a link to install the **local web2md MCP** for slow origins and heavy jobs.

## Errors

Tool failures surface as MCP `isError` / thrown messages, for example:

- `Missing required field: url`
- `HTML too large (…)`
- `HTTP error: 404 …`
- `Blocked by robots.txt: …` (when `respect_robots=true`)
- `no JSON-LD Product found on this page`
- `unsupported MCP format on the remote endpoint: "crawl"`

## Not on the remote endpoint

Crawl, TUI, corpus search, and other CLI-only formats require the [Rust binary](https://github.com/yingkitw/web2md).
