# OCR HTTP API (`/ocr`)

All endpoints are defined in [app/ocr/router.py](../../app/ocr/router.py) on
`APIRouter(prefix="/ocr")`. Interactive docs are at `/docs` and `/redoc`, except when
`ENV=production`. Pipeline details are in [PIPELINE.md](PIPELINE.md). Table definitions are in
[DATA_MODEL.md](DATA_MODEL.md).

## Conventions

- **Auth** (`require_auth` in [app/security.py](../../app/security.py)). When the server has
  `API_KEY` set, every `/ocr` route except `GET /ocr/health` requires the header
  `X-API-Key: <API_KEY>`. That is the only accepted form: `Authorization: Bearer ...` and
  `?token=` query parameters are ignored, and the backend does no user login or session check.
  User login belongs to the frontend, whose server adds the key when it calls this API; browsers
  should not hold the key. This also applies to the file, PDF and CSV-download routes, so a
  browser cannot load them with a plain `<img src>` or link: they go through the frontend server
  (or are fetched with the header and shown as a blob URL).

  A missing or wrong key returns `401 {"detail": "Not authenticated"}`. If `API_KEY` itself is
  unset, no auth is required and a warning is logged at startup.
- **Username** (`require_username` in [app/ocr/events.py](../../app/ocr/events.py)). Every route
  that changes data also needs the header `X-Username: <name of the logged-in user>` (1 to 64
  printable characters). Without it, or with a blank or longer value, the route returns
  `400 {"detail": "X-Username header is required (1-64 printable characters)"}` and changes nothing.
  The key is checked first, so a missing key is still `401`. The backend trusts the value the
  frontend sends (it is not authentication; the API key stays the gate) and writes it to the
  [event log](#event-log). Routes that need it: `POST /ocr/upload`, `/ocr/register-document`, `POST .../year/{fy}/verify`,
  `/ocr/pipeline/ocr`, `/ocr/pipeline/ocr-client-pages`, `/ocr/pipeline/extract`,
  `/ocr/pipeline/merge`, `/ocr/pipeline/merge/stream`, the four `POST /ocr/jobs/*` submit routes, `DELETE /ocr/jobs/{job_id}`,
  `POST` and `DELETE` of `/ocr/company/{code}/line-overrides`, `POST .../merge/review/{id}/resolve`,
  `POST` and `DELETE` of `.../merge/override`, `POST .../merge/override/{id}/decide`, and `GET` or `POST /ocr/sync-ocr-md`. Every other
  `GET` (reads, downloads, `GET /ocr/events`) does not need it. Browsers calling the API directly
  must be allowed to send it: CORS allows `X-Username` (and `DELETE`).
- **Instrument codes** (`{code}`, `instrument_code`) must match `^[A-Za-z0-9_&()\-]{1,64}$`, for
  example `CITYBANK`, `AMCL(PRAN)` or `KAY&QUE`. Dots and slashes are rejected. The server
  upper-cases and trims the code.
- **Fiscal years** (`{fy}`, `fiscal_year`, the values in `fiscal_years_json`) must match
  `^\d{4}(-\d{2})?$`: either a split year such as `2024-25` or a single year such as `2024`. Both
  work on every endpoint, and a company may have reports of both kinds. Send the value without the
  `FY ` prefix (that is only for display). Anything else (`2024-2025`, `24-25`, `FY 2024`) returns `422`.
  Reports are ordered by their end year (`2024-25` ends 2025, `2024` ends 2024).
- **Invalid input** returns `422`. That includes a bad code or year, a page number below 1, more
  than 2000 page numbers, or malformed `fiscal_years_json` / `page_numbers_json`.
- **Uploads** are limited to `MAX_UPLOAD_MB` per file (default 50). A larger file returns
  `413 {"detail": "Uploaded file is too large (limit 50 MB per file)"}`. The content must really be
  a PDF (`%PDF-`) or, for page images, a PNG or JPEG; anything else returns `415`. The whole request
  is validated before anything is written to disk.
- **Error shape** (handlers in [app/main.py](../../app/main.py)):
  - Every error response is JSON with a human-readable string `detail`. This covers 401, 403, 404,
    409, 413, 415, 422 and 500.
  - `422` validation errors also carry the original FastAPI list in `errors`, for example
    `{"detail": "fiscal_year: Field required", "errors": [{"loc": [...], "msg": ..., "type": ...}]}`.
  - Unhandled exceptions return `500 {"detail": "Internal server error"}`; the traceback is logged
    on the server.
  - In the OCR Markdown, failures appear as
    `*Error during live OCR inference (see server logs).*`.
- **Timestamps** in responses (`created_at`, `updated_at`, `last_updated`) are ISO-8601 UTC with
  a `Z` suffix, for example `2026-10-18T09:14:03.123456Z`.
- **Statuses:**
  - `pages[].verification_status` is one of `PENDING`, `VERIFIED`, `FAILED`, `NO_DATA_PAGE` or
    `DEPRECATED`.
  - `extraction_status` is one of `PENDING`, `COMPLETED`, `FAILED` or `DEPRECATED`. When it is
    `FAILED`, `extraction_message` says `Missing N cells`.
  - The constants are `VERIFICATION_STATUSES` and `EXTRACTION_STATUSES` in
    [config.py](../../app/ocr/config.py).
- **Long requests:** OCR and extraction call Gemini page by page and can take minutes. Either
  allow at least 300 s on the reverse proxy, or use the [job queue](#job-queue).

## Files

| Method | Path | Handler | Notes |
|---|---|---|---|
| GET | `/ocr/files/{path}` | `get_storage_file` | Serves a file from the OCR storage directory (`uploads/`, `pages/`, `ocr-md-files/`). Page images are at `/ocr/files/pages/{CODE}/{FY}/page_N.png`; this is the `image_url` returned by the page endpoints. A path that escapes the directory, or a missing file, returns `404`. Needs `X-API-Key`. |

## System and catalog

| Method | Path | Handler | Returns |
|---|---|---|---|
| GET | `/ocr/health` | `health_check` | **Public (no API key).** Minimal `application/health+json` body `{"status": ...}`: `pass` (200) when healthy; `warn` (200) when one of `OCR_API_KEY`, `EXTRACT_API_KEY`, `MERGE_API_KEY` is not set (reads work, that step does not); `fail` (503) when the OCR database cannot be opened. No other fields. |
| GET | `/ocr/companies` | `list_companies` | Companies in SQLite with their status (`database.get_all_companies`). With `?fields=basic`: a light list `[{security_code, company_name, sector}]` without per-company status lookups (`database.get_companies_basic`). It contains the catalog, which is loaded into SQLite at startup, plus any company registered later, so it can replace the frontend's bundled `companies.json`. |
| GET | `/ocr/companies/catalog` | `get_companies_catalog` | Raw contents of `app/ocr/data/companies.json` |
| GET | `/ocr/coverage` | `get_coverage` | The sector-wise coverage matrix (companies x years). See [Coverage](#coverage). |
| GET | `/ocr/company/{code}/status` | `get_company_status` | Analyzed years, documents, page progress |

## Coverage

`GET /ocr/coverage` answers "which companies are done, which are pending, how many years does each have" in one
call (a few aggregate queries; it does not loop over the companies). Query (all optional): `sector` (exact name,
case-insensitive), `search` (part of the code or name), `state` (`complete`, `pending` = not complete, `in_progress`,
`not_started`), `from_year`, `to_year` (1990-2100; they narrow the columns and the cells, not the company `state`).
Needs only `X-API-Key`.

```json
{"years": [2022, 2023, 2024],
 "totals": {"companies": 401, "complete": 1, "in_progress": 3, "not_started": 397},
 "sectors": [{"sector": "Bank", "companies": 36, "complete": 0, "in_progress": 1, "not_started": 35}, ...],
 "companies": [{
   "instrument_code": "DBH", "name": "DBH Finance PLC.", "sector": "Financial Institutions",
   "state": "IN_PROGRESS", "reports": 2, "missing_years": [2022],
   "merged": true, "merged_stale": false, "jobs": {"queued": 3, "running": 1},
   "years": {"2023": {"fiscal_year": "2023", "status": "EXTRACTED", "total_pages": 260, "pages_selected": 3,
                      "pages_ocr": 3, "pages_extracted": 3, "verified": true, "verified_by": "Fin001",
                      "verified_at": "2026-10-27T09:14:03.123456Z", "verify_note": null,
                      "jobs": {"queued": 0, "running": 0}},
             "2024": {"fiscal_year": "2024", "status": "OCR_DONE", "...": "..."}}}]}
```
- **`years`** are the columns: the **end year** of each report (`2022-23` and `2023` both fall under 2023), from the
  earliest to the latest year any company has, without gaps. A company's `years` has a key only for years it has a
  report for; a missing key is a missing year, and `missing_years` lists them.
- **Cell `status`:** `UPLOADED` (the report is registered), `OCR_DONE` (at least one page OCR'd) or `EXTRACTED` (at least
  one page extracted). `verified*` is the [checked mark](#report-verification). When two reports of one company end in
  the same year, the cell shows the one that got further and lists the other in `other_fiscal_years`.
- **Company `state`:** `NOT_STARTED` (no report), `COMPLETE` (every report is `EXTRACTED` **and** checked, and the
  company has a merge that is not stale) or `IN_PROGRESS`. `merged` / `merged_stale` tell whether the company has a
  merged result and whether it is outdated.
- **`jobs`**: queued and running [jobs](#job-queue) of the company (all its reports and its merge), and of each report,
  so the dashboard can show "processing" without calling `GET /jobs` per company. Jobs are in memory (a restart resets them).
- **`sectors`** counts all companies per sector and state whatever the filters, so the sector tabs do not change while
  filtering; a company without a sector is under `Unclassified`. Companies are sorted by sector, then code.

## Documents and pages

| Method | Path | Handler | Body / params |
|---|---|---|---|
| POST | `/ocr/upload` | `upload_documents` | multipart: `instrument_code`, `fiscal_years_json` (JSON object `{"<filename>": "2023-24"}`; `"2024"` is also valid), `files[]` (PDF). A file missing from the map defaults to `2024-25`. Renders every page. |
| POST | `/ocr/register-document` | `register_document_fast` | multipart: `instrument_code`, `fiscal_year`, `filename` (≤255 chars), `total_pages` (0–10000), optional `file` (PDF). Renders nothing. See [Registration](#registration). |
| POST | `/ocr/company/{code}/year/{fy}/verify` | `verify_report` | `{verified: bool, note?: string (≤500)}`. Marks the report's fiscal year as checked by a person, or clears the mark. See [Report verification](#report-verification). |
| GET | `/ocr/company/{code}/year/{fy}/pages` | `get_document_pages` | Page list. Registers the document automatically if its PDF is on disk. Each page has `fiscal_year`, `ocr_status`, `has_markdown`, `ocr_markdown`, `image_url` (`null` until the page is rendered), and the verification fields `extraction_status`, `extraction_message`, `expected_cells`, `extracted_cells`, `missing_cells`, `completeness_pct` and `verification_status`. Query `include_markdown=false` (default `true`) returns `ocr_markdown: null` for every page, to keep the payload small. |
| GET | `/ocr/company/{code}/year/{fy}/page/{page_number}/markdown` | `get_page_markdown` | `{"page": n, "ocr_markdown": "..."}` (`""` before OCR). `404` when the page has no row. |
| GET | `/ocr/company/{code}/year/{fy}/pdf` | `get_company_year_pdf` | The source PDF (`FileResponse`), or `404`. |

### Report verification

The fiscal year of a report is detected from the upload and can be wrong, so a person can mark it as **checked**.
The mark is stored on the report (`documents.verified_by`, `verified_at`, `verify_note`) and shown to everyone.

- `POST /ocr/company/{code}/year/{fy}/verify` with `{"verified": true, "note": "matches the cover"}` marks it as
  checked by the `X-Username`; `{"verified": false}` clears the mark (optional `note` says why). Verifying again
  replaces who / when / note; clearing an unverified report changes nothing. Response:
  `{instrument_code, fiscal_year, verified, verified_by, verified_at, verify_note}`.
  `404` when the report is not registered, `422` for a bad body or year, `400` without `X-Username`.
- Every call is an [event](#event-log): `REPORT_VERIFY` or `REPORT_UNVERIFY`, with the note and who had verified it
  before (`was_verified`, `previous_verified_by`, `previous_verified_at`), also when nothing changed or it failed.
- The state is returned as `verified`, `verified_by`, `verified_at`, `verify_note` in every `documents[]` entry of
  `GET /company/{code}/status` (and of the `status` in the upload / register responses) and at the top level of
  `GET .../year/{fy}/pages`. A pending report has `verified: false` and the other three `null`.
- **A new PDF clears the mark** (`POST /upload` for the year, or `POST /register-document` with a `file`): the
  checked file is gone. That response has `verification_cleared: true` (for `/upload`, per document in
  `uploaded_documents`) and the `REPORT_UPLOAD` / `REPORT_REGISTER` event has `verification_cleared: true`.
  Registering without a `file`, OCR and extraction leave the mark alone.

### Registration

- Registration is **optional**. `/pipeline/ocr-client-pages` works without it.
  `/pipeline/ocr` auto-registers a PDF it finds on disk. It returns
  `404 {"detail": "Document not registered and no page image supplied ..."}` when a requested page
  has no rendered image and there is no PDF to render it from.
  Each page in its `completed_pages` has `ocr_failed`: when the OCR call fails (e.g. a used-up daily quota) the page's
  stored OCR and extraction are left untouched and `ocr_markdown` carries an error stub that is NOT saved.
- Registering again for the same `(instrument_code, fiscal_year)` **replaces** the stored PDF and
  page count, and the response has `"replaced": true`.
  - Existing pages, OCR Markdown and extracted metrics are kept.
  - When no `file` is sent, the previously stored PDF path is kept.
- The frontend should prevent duplicates within one upload session. Across sessions, the last
  registration wins.

## Pipeline

| Method | Path | Handler | Body |
|---|---|---|---|
| POST | `/ocr/pipeline/ocr-client-pages` | `run_ocr_client_pages` | multipart: `instrument_code`, `fiscal_year`, `page_numbers_json` (JSON list of ints ≥1), `files[]` (PNG/JPEG; the page number is taken from the filename, then from the list, then from the position). The Gemini calls run in a worker thread, so they don't block other requests. |
| POST | `/ocr/pipeline/ocr` | `run_ocr_pipeline` | `OcrRequest {instrument_code, fiscal_year, page_numbers[]}` |
| POST | `/ocr/pipeline/extract` | `run_extraction_pipeline` | `ExtractionRequest` (same fields as `OcrRequest`) |
| POST | `/ocr/pipeline/merge` | `run_merge_pipeline` | `MergeRequest {instrument_code}`. `409` while another merge for the company is running. |
| POST | `/ocr/pipeline/merge/stream` | `run_merge_pipeline_stream` | `MergeRequest`. NDJSON progress stream; see [Merge stream](#merge-stream). |

- **Page handling (OCR and extraction):**
  - Duplicate page numbers are dropped.
  - Up to `OCR_PAGE_PARALLELISM` (default 10) pages are processed in parallel. Results keep the request order.
  - `total_time_consumption_sec` is the longest page duration (wall clock), not the sum.
- **OCR response:**
  - `completed_count`, `usage_report`
  - `pages[]`, each with `page_number`, `ocr_markdown`, `tokens`, `duration_sec`,
    `extraction_status` and `verification_status`. Re-running OCR on a page that was already
    extracted marks it `DEPRECATED`.
  - Client pages also have `image_url: "/ocr/files/pages/{CODE}/{FY}/page_N.png"`.
  - `ocr-client-pages` always returns one `pages[]` entry per uploaded image. A page whose OCR
    failed carries the error Markdown, and a request without images is a `422`.
- **Extraction response:**
  - Item counts and the cumulative usage.
  - `raw_csv_data` (`headers`, `years`, `rows`, `csv_text`, …): the raw dataset for **all**
    pages and years of the company, not just the requested pages.
  - `verification_results[]`: **exactly one entry per requested page**, even when that page's
    extraction raised. A failed page has `is_success: false`, `extraction_status: "FAILED"` and
    zero counts. Each entry has `page_number`, `items_count`, `expected_cells`, `extracted_cells`,
    `missing_cells`, `completeness_pct`, `extraction_status`, `verification_status`, `is_success`
    and `message`, plus the token counts and `label_evidence_failures` (always `{}`: the label evidence check is switched off), `compositions_count` (breakdowns
    the model identified on the page) and `composition_mismatches` (how many of them do not add up) and `unidentified_lines` (lines marked `FIELD_UNIDENTIFIED`, see `/unidentified-lines`). See
    [PIPELINE.md](PIPELINE.md#stage-3-extraction-markdown--structured-metrics).
- **Merge response:** `merged_rows_count`, `merge_count`, `conflicts_count`, `total_pairs` (ambiguous
  links for Gemini), `evaluated_pairs` (questions actually sent; equal to `total_pairs` because Gemini answers are never cached),
  `matched_pairs` (links between reports), `token_usage` (also `missing_answers`: questions the model left out of its answer) and `merged_csv_data` (`headers`, `years`,
  `rows`, `csv_text`, `stating_csv_text`, `merge_report`, `conflicts`, `value_mismatch_conflicts`,
  `micro_variance_merges`, plus `series_count`, `links_by_rule`, `review_open_count` and
  `relabelled` — counts of metrics whose metric type or unit the deterministic relabel changed, and
  `scope_disagreements` with the printed statement titles; scope itself is never rewritten).
  The merge links each report to the next; see [PIPELINE.md § Stage 5](PIPELINE.md#stage-5-merge-series-linking).

### Merge stream

`POST /ocr/pipeline/merge/stream` responds with `application/x-ndjson`: one JSON object per line,
each terminated by `\n`.

| `type` | Fields |
|---|---|
| `init` | `stage: "init"`, `percent: 5`, `total_pairs`, `evaluated_pairs`, `matched_pairs`, `conflicts_count`, `status: "INITIALIZING"`, `subtitle`, plus the [call counters](#gemini-call-progress) (the expected number of Gemini calls is known here) |
| `progress` | one at the start and one at the end of every Gemini call, and one for each retry (none when nothing is ambiguous): `stage` (`gemini_call_started` / `gemini_retry` / `gemini_call_done`), `percent` (10 to 90, by calls done), the [call counters](#gemini-call-progress), `total_pairs` (ambiguous links expected, whole merge), `evaluated_pairs` (answered so far), `matched_pairs`, `conflicts_count` (answered "no match" so far), `current_statement`, `field_a`, `field_b`, `status` (`EVALUATING` / `EVALUATED`), `confidence`, `subtitle`; `call_duration_sec` on `gemini_call_done`; `retry_attempt` and `retry_in_sec` on `gemini_retry` |
| `finalize` | `stage: "saving"`, `percent: 95`, the counts as in `progress` (`matched_pairs` = links), `status: "SAVING"`, `merged_clusters_count` (series), `subtitle`, and the call counters, now final (`estimate_exact: true`) |
| `complete` | `stage: "complete"`, `percent: 100`, `gemini_calls_done` = `gemini_calls_total` (calls made), `estimate_exact: true`, `total_pairs`, `evaluated_pairs`, `matched_pairs`, `conflicts_count`, `value_mismatch_count`, `value_mismatch_conflicts`, `merged_rows_count`, `merge_count`, `token_usage`, `merged_csv_data: {csv_text, stating_csv_text, rows, merge_report, ...}`, `subtitle` |
| `error` | `error` (for example `Merge failed (ValueError)`, `Merge cancelled`, `Merge stopped unexpectedly`, `Merge timed out`), `subtitle` |

#### `merge_stats`

Every `init` / `progress` / `finalize` event of the merge (stream, job progress, [polling](#merge-by-polling-the-sockets-messages-over-plain-http) and the [socket](#merge-socket-off)) carries `merge_stats`, and the last summary has the final one (`complete` event, the `POST /pipeline/merge` response, the `POST /jobs/merge` result, `merged_csv_data.merge_stats`, the `MERGE_RUN` event details):

```json
{"reports": {"total": 5, "done": 3, "remaining": 2},
 "lines": {"total": 410, "done": 252, "remaining": 158},
 "links": {"total": 190, "by_system": 171, "by_user": 4, "by_fintra_ai": 15},
 "fintra_ai": {"questions_total": 31, "questions_answered": 18, "questions_remaining": 13, "linked": 15, "not_linked": 3, "calls_total": 2, "calls_done": 1},
 "reviews": {"created": 22, "auto_resolved": 0, "high": 3, "medium": 5, "low": 14, "complete": false}}
```

`reports`: reports linked to the ones before them. `lines`: the printed line items of all reports (one per report and line); `done` are those of the linked reports, `remaining` the rest. `links`: pairs of lines found to be the same line, by who decided: the system (name, value and gap rules), a user override, or Fintra AI. `fintra_ai`: the ambiguous pairs sent to Fintra AI (answered, remaining, how many it linked or not). `reviews`: review items made so far, by attention level; while the merge runs only those the linking makes are counted, `complete: true` in the final summary means all of them, and `auto_resolved` is how many an existing user override already resolved. A progress event with `stage: "report_linked"` is sent after every report, so a merge that asks Fintra AI nothing still moves.

#### Merge by polling (the socket's messages over plain HTTP)

The hosted server does not pass sockets through, so the merge socket below is **switched off** (`MERGE_SOCKET_ENABLED` unset). The frontend gets the same messages by polling:

1. `POST /ocr/jobs/merge` `{instrument_code}` (needs `X-Username`) returns `202 {job_id, status, position, waiting_for_earlier_job, ...}`.
2. `GET /ocr/jobs/{job_id}/events?after=<n>` every second or two, `after` = the `next` of the previous answer (0 the first time), until `done` is true. It returns `{job_id, status, done, next, messages}`: `messages` are the new messages since `after`, exactly the socket's messages in the socket's order (the table below): `queued` first, `started`, every progress event (`init`, `progress` with `stage`, `finalize`, each with `merge_stats`; no `merged_csv_data`), then one `complete` (the summary with `merge_stats`) or `error`. A message is never sent twice and none is skipped; after `done` the answer stays empty. `404` when the job is unknown or expired (`JOB_TTL_SEC` after it ended; a server restart empties the queue). A job keeps its last 5000 progress events (`jobs.MAX_KEPT_EVENTS`).
3. Cancel with `DELETE /ocr/jobs/{job_id}` (it stops before the next Fintra AI call; the last message is then `error` with `Merge cancelled`). Closing the page does **not** cancel a polled merge.

#### Merge socket (off)

Registered only when `MERGE_SOCKET_ENABLED=1`; the code is kept to switch it back on. `WS /ocr/ws/merge/{code}?username=<name>&api_key=<key>` is the merge with a socket instead of polling or an NDJSON stream. Browsers cannot send headers on a socket, so `username` (the `X-Username`, 1-64 printable characters) and `api_key` (needed when the server has `API_KEY` set) are query parameters. Connecting starts the merge as a job in the same queue as `POST /jobs/merge` (it waits for earlier jobs of the company; recorded as `MERGE_RUN` with `details.socket`). Every message is JSON:

| `type` | When | Fields |
|---|---|---|
| `queued` | first message | `job_id`, `position`, `waiting_for_earlier_job`, `deduplicated` |
| `started` | the job starts running | `job_id` |
| `init`, `progress`, `finalize` | every step of the merge; for every Fintra AI call a `progress` with `stage: "gemini_call_started"` and one with `gemini_call_done` (and `gemini_retry` on a retry) | as the stream: `percent` (never backwards), `subtitle`, `gemini_calls_done`, `gemini_calls_total`, `estimate_exact`, `eta_sec`, `total_pairs`, `evaluated_pairs`, `matched_pairs`, `conflicts_count`, ... |
| `complete` | last message, merge saved | `percent: 100`, `job_id`, `status: "done"`, `merged_rows_count`, `merge_count`, `conflicts_count`, `value_mismatch_count`, `total_pairs`, `evaluated_pairs`, `matched_pairs`, `token_usage`, `subtitle`. No table: read it with `GET /company/{code}/results/merged` |
| `error` | last message | `error`, `subtitle`, `status_code`, `job_id`/`status` once a job exists |

After the last message the server closes the socket (code `1000`; `1008` after an `error` for a refused connection: `401` wrong key, `400` no username, `404` no metrics, `422` bad code, `429` queue full). The client may send `{"action": "cancel"}`; closing the socket cancels the merge (it stops before its next Fintra AI call and nothing is saved). `requirements.txt` needs `websockets` for uvicorn to serve sockets.

#### Gemini call progress

**Naming:** the product shows the model as **Fintra AI**, never "Gemini" or "LLM". Every text the API returns for display (progress `subtitle`s such as "Asking Fintra AI (call 2 of about 5)", review reasons, override notes, job errors) says "Fintra AI". Field and value names stay as they are (`gemini_calls_done`, `stage: gemini_call_*`, `source: "gemini"`, `rule: "GEMINI"`, override `source: "llm"`): the frontend maps them to "Fintra AI" when it shows them.

Gemini is asked once for every report-to-report step that has ambiguous links, plus once for the final
join pass, and a step with more than 50 questions takes several calls. A merge makes about 5 to 15 calls
(CITYBANK 5, BXPHARMA 13, DESCO 7). Whether a step needs a question depends on Gemini's answers for the
earlier steps, so the exact number is only known at the end. Before it starts, the merge makes a free dry run
of the linking (1 to 3 seconds, no Gemini) to **estimate** the number of calls; the estimate matched the
real merges to within one call. It is corrected as each step is reached. These fields are on `init`,
every `progress`, `finalize` and `complete` events:

| Field | Meaning |
|---|---|
| `gemini_calls_total` | Calls expected in the whole merge (an estimate until `estimate_exact` is `true`; it can grow or shrink by a few as steps are reached) |
| `gemini_calls_done` | Calls finished |
| `estimate_exact` | `false` until `finalize`, then `true` |
| `questions_total` | Ambiguous links expected in the whole merge (estimate, same rule); `total_pairs` carries the same number |
| `round`, `round_index`, `rounds_total` | The step now running: the report whose lines are being linked (for example `"2021-22"`) or `"JOIN"` for the final pass; its position (1-based) and the number of steps expected. `round` is `null` on `init` and `finalize` |
| `elapsed_sec` | Seconds since the merge started linking |
| `eta_sec` | Seconds left for the Gemini calls: average duration of the finished calls times the calls left; `null` until one call has finished. It does not include saving (a second or two) |
| `call_in_round`, `calls_in_round` | Which call of the current step this is (progress events only) |

`percent` is `10 + 80 * gemini_calls_done / gemini_calls_total` (at most 90) and never decreases: when the
total grows, the bar waits instead of going back. With no ambiguous links the stream has no `progress`
events and `gemini_calls_total` is `0`: `init` (5), `finalize` (95), `complete` (100).

Guarantees:

- **Exactly one terminal event.** The stream always ends with exactly one `complete` or `error`
  event, even when the merge raises, its worker thread dies, or it is silent for 15 minutes
  (`MERGE_STREAM_IDLE_TIMEOUT_SEC`).
- **`percent` never decreases.**
- **Errors before the stream starts are plain HTTP responses:**
  - `404` when the company has no metrics and no saved merge.
  - `409 {"detail": "A merge is already running for this company"}` when a merge is already
    running. One merge per company runs at a time; the lock is in memory, which is enough for the
    single uvicorn process in `Procfile`.
- **No metrics, but a saved merge:** the stream is a single `complete` event with
  `subtitle: "Loaded existing merged results"` and the saved `merged_csv_data.rows`.
- **Cancelling:** when the client disconnects, for example after aborting the fetch, the merge
  stops before its next Gemini batch and **no merge result is saved** (the deterministic relabel of
  scope, metric type and unit may already have been applied).

## Job queue

OCR, extraction and merge as queued jobs. **One job is one unit of work**: the OCR of one page, the
extraction of one page, or the merge of one company; there is no batch. The frontend sends all its
requests at once (one request per page, then the merge); the backend queues them and runs them in the
order received, within Gemini's rate limits. The submit call returns `202` with the job; poll
`GET /ocr/jobs/{job_id}` (or `GET /ocr/jobs`) until `status` is `done`, `error` or `cancelled`.
Implemented in [app/ocr/jobs.py](../../app/ocr/jobs.py).

| Method | Path | Handler | Body / result |
|---|---|---|---|
| POST | `/ocr/jobs/ocr` | `submit_ocr_job` | `{instrument_code, fiscal_year, page_number}` (ONE page; the old `page_numbers` list is a `422`). The result is the `/pipeline/ocr` response for that page. |
| POST | `/ocr/jobs/ocr-client-pages` | `submit_ocr_client_pages_job` | multipart: `instrument_code`, `fiscal_year`, `page_number`, `file` (ONE PNG/JPEG). The image is validated and saved before the `202`; the result is the `/pipeline/ocr-client-pages` response for that page. |
| POST | `/ocr/jobs/extract` | `submit_extract_job` | `{instrument_code, fiscal_year, page_number}`. The result is the `/pipeline/extract` response for that page. |
| WS | `/ocr/ws/merge/{code}?username=&api_key=` | `merge_socket` | **Off** unless `MERGE_SOCKET_ENABLED=1`. Runs the merge and feeds its status over a socket (see [Merge socket](#merge-socket-off)). |
| POST | `/ocr/jobs/merge` | `submit_merge_job` | `{instrument_code}`. While running, `progress` holds the merge stream's latest progress event (`percent`, `stage`, `gemini_calls_done`, `gemini_calls_total`, `eta_sec`, ...). The result is a summary: `merged_rows_count`, `merge_count`, `conflicts_count`, `value_mismatch_count`, `total_pairs`, `evaluated_pairs`, `matched_pairs`, `token_usage` (read the table with `GET /results/merged`). No metrics: `error` with `status_code` `404`. |
| GET | `/ocr/jobs` | `list_jobs_endpoint` | The queue. Query: `instrument_code`, `status`, `kind`. Returns `{queues: {ocr\|extract\|merge: {queued, running}}, limits: {rate_limit_rpm, workers, queue_max}, jobs: [...]}`, jobs in the order sent, without `result`. |
| GET | `/ocr/jobs/{job_id}` | `get_job` | One job (see below). Unknown or expired id: `404`. |
| GET | `/ocr/jobs/{job_id}/events` | `get_job_events` | Query `after` (default 0). The merge socket's messages by polling: `{job_id, status, done, next, messages}` (see [Merge by polling](#merge-by-polling-the-sockets-messages-over-plain-http)). Unknown or expired id: `404`. |
| DELETE | `/ocr/jobs/{job_id}` | `cancel_job` | Cancels a job: a queued one at once (`status: cancelled`, its event `CANCELLED`); a running merge before its next Gemini batch (`cancel_requested: true`, then `cancelled`). `409` for a running page job (one Gemini call, cannot be stopped) or a finished job. Needs `X-Username`; recorded as `JOB_CANCEL`. |

A job:
```json
{"job_id": "9f...", "kind": "extract", "stage": "extract", "status": "queued",
 "instrument_code": "DBH", "fiscal_year": "2023", "page_number": 200,
 "position": 4, "waiting_for_earlier_job": true, "progress": null, "cancel_requested": false,
 "result": null, "error": null, "status_code": null,
 "created_at": "...Z", "started_at": null, "finished_at": null, "updated_at": "...Z"}
```
- `kind`: `ocr`, `ocr-client-pages`, `extract` or `merge`; `stage` is the Gemini step (`ocr`, `extract`, `merge`).
- `status`: `queued`, `running`, `done` (`result` set), `error` (`error`, `status_code`: the HTTP status the
  synchronous endpoint would have returned, for example `404`) or `cancelled`.
- `position`: 1-based place in its step's queue while `queued` (`null` otherwise).
- `waiting_for_earlier_job`: the job is queued behind an earlier job on the same target (see Order).

**Order and limits.**
- Each step (`ocr`, `extract`, `merge`) has its own queue, because each step has its own API key and
  quota. A queue starts its jobs one by one in the order received. Up to `OCR_JOB_WORKERS` /
  `EXTRACT_JOB_WORKERS` (default `OCR_PAGE_PARALLELISM`, 10) page jobs may be in flight so slow calls
  (20-60 s) overlap, and one merge at a time (`MERGE_JOB_WORKERS`, 1). Every Gemini call still waits for the
  step's rate limiter, so requests per minute never exceed `RATE_LIMIT_RPM`, however many jobs are queued.
  Set a worker count to 1 to run that step strictly one job after another.
- A job waits for the earlier jobs on the same target: a page's extraction starts only after that page's OCR
  job finished (sent earlier), a merge only after every earlier OCR / extraction job of the company, and a
  page job sent after a merge waits for that merge. Jobs of other pages and companies are not held up.
- Sending the same work again while it is still queued returns the queued job with `"deduplicated": true`
  (no second job, no second event).
- More than `JOB_QUEUE_MAX` (default 5000) waiting jobs: `429`.
- Jobs are kept in memory: they don't survive a restart (their events stay `STARTED`), and finished jobs are
  dropped `JOB_TTL_SEC` (default 3600) after they finish.

The synchronous `/ocr/pipeline/*` endpoints still exist (several pages per call) but bypass the queue order;
the frontend should use the jobs.

## Results and reporting

| Method | Path | Handler | Notes |
|---|---|---|---|
| GET | `/ocr/company/{code}/results/raw` | `get_company_raw_results_endpoint` | One chunk (100 rows) of the raw table, with `raw_columns`, `years` and the status fields. Query: `statement_type` (optional, one statement's rows), `offset` (default 0), `view` (`slim` default, or `full` for the former, about 3 times larger rows). Fast: no merge enrichment. See [Results response](#results-response). `404` only when the company has neither results nor extracted metrics. |
| GET | `/ocr/company/{code}/results/merged/row/{series_id}` | `get_company_merged_row_endpoint` | One merged row in full (the former `merged_rows` item, plus `flags`), for the row's modals. `404` for a series that is not a row of the saved merge. See [Merged table rows](#merged-table-rows-slim). |
| GET | `/ocr/company/{code}/results/merged/conflicts` | `get_company_value_conflicts_endpoint` | Optional `statement_type`. `{statement_type, count, value_mismatch_conflicts}`: the value conflicts that used to be in `results/merged`. |
| GET | `/ocr/company/{code}/results/merged/report` | `get_company_merge_report_endpoint` | Optional `statement_type`. `{statement_type, count, merge_report}`: the merge report that used to be in `results/merged`. |
| GET | `/ocr/company/{code}/results/merged` | `get_company_merged_results_endpoint` | One statement of the merged table (consolidated and standalone rows together), with its conflicts and merge report. Query: `statement_type` (optional; default: the first statement that has rows). See [Results response](#results-response). Same `404` rule. |
| GET | `/ocr/company/{code}/cell-details` | `get_cell_details_endpoint` | See [Cell details](#cell-details). |
| GET | `/ocr/company/{code}/cell-by-position` | `get_cell_by_position_endpoint` | Fast cell provenance by markdown coordinates or reference string (e.g. `FY 2002-03 \| P-59 \| T-2 \| R-9 \| C-3`). See [Cell by position](#cell-by-position). |
| GET | `/ocr/company/{code}/verify-extraction` | `verify_company_extractions_endpoint` | Optional query `fiscal_year`. Re-audits every extracted page against its OCR Markdown. Returns totals (`total_pages`, `verified_pages`, `failed_pages`, `is_all_verified`, `total_expected_cells`, `total_extracted_cells`, `total_missing_cells`, `overall_completeness_pct`) and `pages[]`. |
| GET | `/ocr/company/{code}/extracted-cells` | `get_company_extracted_cells_endpoint` | Optional query `fiscal_year`. Extracted cells keyed by `"{fiscal_year}_{page_number}"`, each with `table_num`/`table_row`/`table_col`, `numeric_value`, `concept_name`, `raw_field` and a check against the Markdown cell. Used to highlight cells on the page image. Each cell also has `series_id` (the merged row it belongs to in the last merge, `null` when it is in no series or there was no merge), so a click can open `cell-details?series_id=`. |
| GET | `/ocr/company/{code}/note-cells` | `get_company_note_cells_endpoint` | Optional query `fiscal_year`. Every printed cell that holds a note reference, keyed by `"{fiscal_year}_{page_number}"`: `[{table_num, table_row, table_col, notes: ["4"], concept_name, raw_field, metric_ids}]`. The cell is the Note column cell, or the label cell when the label prints "(note 42.a)". Rows without figures are included too (e.g. a heading "Provision for taxation | 44"): their `metric_ids` is `[]` and `concept_name` is `null`; `raw_field` is the row's printed label. Used to highlight note cells on the page. |
| GET | `/ocr/company/{code}/unidentified-lines` | `get_company_unidentified_lines_endpoint` | Optional query `fiscal_year`. Lines whose figures were extracted but whose page does not show what the line is (the title / parent heading is on another page): `[{metric_id, fiscal_year, page_number, table_num, table_row, raw_field, suggested_field (the model's name for the line), reason, values: {year: number}}]`. A line also lands here when its concept name is only a generic word ("Total", "Subtotal", "Others"...): `suggested_field` is then its heading plus that word. They carry `verification_flag: "FIELD_UNIDENTIFIED"`, are left out of the raw and merged CSV and of the merge, and appear as `FIELD_UNIDENTIFIED` review items after a merge. A user resolves them by naming the line (`POST /line-overrides`; the suggestion is pre-filled) or by re-extracting the page. |
| POST | `/ocr/company/{code}/line-overrides` | `set_company_line_override` | Body `{fiscal_year, page_number, table_num (default 1), table_row, field, note?}`. A user names one printed row (typically a line from `/unidentified-lines`, but any line can be renamed): every metric of that row gets `concept_name = field` (display `field` adds its " (segment)"), `verification_flag = "USER_CONFIRMED"`, `flag_reason = "Named by a user"`, so the line enters the CSV and the merge. Stored by page coordinates and re-applied whenever the page is extracted again. Returns `{id, updated_metrics, merged_stale}`; `404` when no extracted line sits at those coordinates. |
| GET | `/ocr/company/{code}/line-overrides` | `list_company_line_overrides` | Optional query `fiscal_year`. `[{id, instrument_code, fiscal_year, page_number, table_num, table_row, field, note, updated_at}]`. |
| DELETE | `/ocr/company/{code}/line-overrides/{override_id}` | `delete_company_line_override` | Forgets the user's name; the lines keep it until their page is extracted again. `404` for an unknown id. |
| GET | `/ocr/company/{code}/download/raw` | `download_raw_csv` | `{CODE}_financial_data.csv`, built from the stored metrics if no CSV was saved yet; `404` if there is nothing to export. Starts with a UTF-8 BOM. |
| GET | `/ocr/company/{code}/download/raw.xlsx` | `download_raw_xlsx` | `{CODE}_financial_data.xlsx` (same content type as the merged workbook); `404` when there is nothing to export. The raw table as a workbook with the same sheets as the merged one (`Balance Sheet`, `Income Statement`, `Cash Flow`, `Changes in Equity`, `Notes & Others`; only those with rows). Per sheet two header rows (row 1: `Field / Raw Line Item`, `Scope`, `Unit`, then each report, e.g. `FY 2022-23`, merged over its columns; row 2: each column's year, or its printed date for a restated column, whose header cell is darker), then one row per printed line in the table's order (the same rows and counts as `results/raw`), numbers as numbers (two decimals for per-share, % and ratio units), an empty cell where a report does not print the year. Header rows and the first three columns are frozen. |
| GET | `/ocr/company/{code}/download/merged` | `download_merged_csv` | `{CODE}_financial_data_merged.csv`; `403` until a merge has run. Starts with a UTF-8 BOM. |
| GET | `/ocr/company/{code}/results/standard` | `get_company_standard_results` | The **Standard tab**: the merged table in the standard layout as data, the same layout and figures as `/download/standard.xlsx` (see [Standard tab](#standard-tab)). `403` until a merge has run. |
| GET | `/ocr/company/{code}/download/standard.xlsx` | `download_standard_xlsx` | The "Standard Excel", `{CODE}_financial_data_standard.xlsx`: every statement in the standard layout, a sheet per statement (see below); inactive rows left out; `403` until a merge has run. `download/merged.xlsx` is a different export and is unchanged. |
| GET | `/ocr/company/{code}/download/merged.xlsx` | `download_merged_xlsx` | `{CODE}_financial_data_merged.xlsx` (`application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`); `403` until a merge has run. One sheet per statement that has rows, named like the results tabs and in their order: `Balance Sheet`, `Income Statement`, `Cash Flow`, `Changes in Equity`, `Notes & Others` (every other statement type). Per sheet: a header row (`Field / Line Item`, `Unit`, one column per year, oldest first), a bold group row per scope (`Standalone (25)`, `Consolidated first`), then the lines in tree order with the line name **indented by its depth** (Excel cell indent, depth 0 to 15). The tree is the one the results screen draws: a line sits under the total whose current `breakdown.components` lists it in the newest report (the smallest total among those that list it in the same report), in printed order; lines no total lists are top-level, in the merged-row order (alphabetical); totals and subtotals are bold. Values are numbers (`#,##0`, two decimals for per-share, % and ratio units), an empty cell where a year has no value, and a cell that was restated is filled light yellow with a comment "Restated. First printed as …". The first two columns are frozen. |
| GET | `/ocr/company/{code}/usage-report` | `get_company_usage_report` | Query: `fiscal_year`, `runs_limit`, `runs_offset`. See [Usage report](#usage-report). |
| GET | `/ocr/company/{code}/year/{fy}/usage-report` | `get_company_year_usage_report` | Query: `runs_limit`, `runs_offset` |

## Series merge: series, evidence, review queue, overrides

| Method | Path | Handler | Notes |
|---|---|---|---|
| GET | `/ocr/company/{code}/year/{fy}/compositions` | `list_cell_compositions` | Optional query `page_number`. Breakdowns the extraction model identified: which printed cells add up to a total. `{instrument_code, fiscal_year, page_number, count, mismatch_count, compositions: [{id, fiscal_year, page_number, table_num, direction, operation, expression, total, parts, parts_sum, difference, status}]}`; `total` and each part carry `row`, `col`, `value` and the extracted metric of that cell (`field`, `raw_field`, `metric_id`, `year`; `null` when the row was not extracted as a metric). See [Breakdowns](#breakdowns). |
| GET | `/ocr/company/{code}/series` | `list_company_series` | `{instrument_code, series: [{id, partition_key, statement_type, scope, entity_key, segment_key, value_kind, unit, canonical_field, first_report, last_report, members_count, status, created_at, updated_at, members: [{report, field, label_key, node_id}], active}]}` (`active` false when a user switched the row off, see [Active and inactive rows](#active-and-inactive-rows)) |
| PUT | `/ocr/company/{code}/series/{series_id}/active` | `set_company_series_active` | Body `{active: bool}`. Switches one merged row on or off, see [Active and inactive rows](#active-and-inactive-rows). Needs `X-Username`. |
| POST | `/ocr/company/{code}/enter-disclosure` | `enter_disclosure` | Body `{series_id, year, value, note, unit?, date?}`. Enters the user's disclosure of one cell: adds it, or replaces the one the cell already has (one endpoint for add and edit), see [Disclosures added by users](#disclosures-added-by-users-and-the-current-disclosure). `200`. Needs `X-Username`. |
| DELETE | `/ocr/company/{code}/enter-disclosure?series_id=&year=` | `delete_disclosure` | Deletes the user's disclosure of the cell (the extracted ones cannot be deleted); a choice of it as the current disclosure goes with it. `{series_id, year, deleted, current_reset}`. Needs `X-Username`. |
| PUT | `/ocr/company/{code}/cell-current` | `set_cell_current` | Body `{series_id, year, record_key}`. Chooses the current disclosure of a cell. Needs `X-Username`. |
| DELETE | `/ocr/company/{code}/cell-current?series_id=&year=` | `reset_cell_current` | Back to the automatic current disclosure. Needs `X-Username`. |
| PUT | `/ocr/company/{code}/series-active` | `set_company_series_active_bulk` | Body `{series_ids: [int] (1 to 2000), active: bool}`. The same for many rows at once. Needs `X-Username`. |
| GET | `/ocr/company/{code}/series-formation?series_id=` | `get_company_series_formation` | How a merged row was formed (the "Consolidated Disclosures" page): `{series_id, canonical_field, statement_type, scope, unit, value_kind, status, active, merged_values {year: value}, members, unmatched_metrics}`. `members` (oldest report first) are the lines of the series: `{report, field, label_key, node_id, link_rule, link_confidence, metrics}` (`link_rule` null for the first). Each metric: `{metric_id, report, page_number, statement_type, concept_name, field, raw_field, metric_type, scope, aggregation_level, category, values: [{data_id, year, date, restated, value, unit, multiplier, table_num, table_row, table_col}]}`, only the printed cells that are part of the series (from `series_cell`). A metric that fits no member of its report is in `unmatched_metrics`. `404` for an unknown series. |
| GET | `/ocr/company/{code}/series/{series_id}/evidence` | `get_company_series_evidence` | The series plus `links[]`: `{from_report, to_report, from_label, to_label, rule, source, confidence, status, decided_at, gap (reports skipped), evidence: {class, overlap: [{year, a, b, match, sign_flip, w}], weight, from_field, to_field, sections, cached?, override_id?}}`. `404` for an unknown id. |
| GET | `/ocr/company/{code}/merge/review` | `list_merge_review` | Query `status` = `open` (default), `resolved`, `dismissed` or `all`. `{instrument_code, status, count, items: [{id, kind, dedupe_key, payload, status, resolution, override_id, attention, attention_rank, created_at, updated_at}]}` |
| POST | `/ocr/company/{code}/merge/review/{item_id}/resolve` | `resolve_merge_review` | Body `{action: "accept"\|"reject"\|"dismiss", note?}`. For `AMBIGUOUS_LINK`, `VALUE_CONFLICT`, `RESTATEMENT_CHECK`, `STALE_LINK`, `LONG_GAP`, `GAP_BLOCKED` and `MAGNITUDE_JUMP`: `accept` saves a `FORCE_LINK` override for the two lines and reports, `reject` a `FORCE_SPLIT`. Other kinds can be accepted (acknowledged) or dismissed; `reject` is `422`. Returns `{id, status, resolution, override_id, merged_stale}`. |
| GET | `/ocr/company/{code}/merge/overrides` | `list_merge_overrides` | Query `include_inactive`, `source` (`user` or `llm`). `{instrument_code, overrides: [...]}`. Every override has `source`: `"user"` (a person decided it) or `"llm"` (a link Gemini made in the last merge: a `FORCE_LINK` with both reports, plus `confidence` and `link_rule` `GEMINI` / `GEMINI_JOIN`). `created_by` is the `X-Username` of the person who made the override (a created override, an accepted or rejected review item, `make_permanent`, `break_link`); `null` for an LLM override and for old overrides with no event. |
| POST | `/ocr/company/{code}/merge/override/{override_id}/decide` | `decide_llm_override` | Body `{action: "make_permanent"\|"break_link", note?}`; needs `X-Username`. On an active LLM override: `make_permanent` makes the same override a user override (kept by every later merge); `break_link` removes it and saves a user `FORCE_SPLIT` of the two lines. Either way Gemini is not asked about that pair again. `404` unknown id, `422` not an active LLM override. Returns `{override, merged_stale: true}`. Recorded as `OVERRIDE_MAKE_PERMANENT` / `OVERRIDE_BREAK_LINK`. |
| POST | `/ocr/company/{code}/merge/override` | `create_merge_override` | `201`. Body `{kind: FORCE_LINK\|FORCE_SPLIT\|RENAME\|DETACH, statement_type, scope, entity_key?, segment_key?, a_label, a_report?, b_label?, b_report?, new_label?, note?}`. Labels are line names as shown (matched through their normalised label). Without reports a link or split applies to every report. `422` when `b_label` (link/split) or `new_label` (rename) is missing, or only one report is given; `409` when a link names the same report twice. |
| DELETE | `/ocr/company/{code}/merge/override/{override_id}` | `delete_merge_override` | Deactivates it; `404` if unknown. |

Review items about figures that differ are raised only for a HIGH variance: a linked line whose figure in a common year is more than 100 % off the older report's figure (or flips sign on a balance-sheet line, or was 0), as `VALUE_CONFLICT`. A smaller difference is a restatement of the cell (`RESTATED_LOW` / `MID`, see `flags`) and makes no review item; `RESTATEMENT_CHECK` is no longer raised. Conflicts that are about more than the figures (headings that share no word, a line another one matches exactly, a changed definition) are still raised.

**Pair items** (`VALUE_CONFLICT`, `LONG_GAP`, `MAGNITUDE_JUMP`, `GAP_BLOCKED`, `STALE_LINK`, `AMBIGUOUS_LINK`, `RESTATEMENT_CHECK`; no merge creates `RESTATEMENT_CHECK` any more, the endpoints still accept it and it ranks LOW) all carry `payload.a_node` / `b_node` and `payload.comparison` `{years, conflicting_years, year_status, lines: [a, b]}` (when both lines are in the current data; `lines[i]` = `{side, report, field, values: {year: value}, cells: {year: {page_number, table_num, table_row, table_col}}}`, every year each line prints). `year_status[year]` is `MATCH` (both print, agree), `MISMATCH` (both print, differ), `ONLY_A` or `ONLY_B`. Except `VALUE_CONFLICT` (which has `overlap_sources`), they also carry `a_report, a_page_number, a_table_num, a_table_row, a_table_col` and the same `b_*`: the printed cell of each line in the disputed year (a `MAGNITUDE_JUMP`'s `a_year` / `b_year`, else the first `MISMATCH` year, else the first year both print, else the line's first printed year); a side no longer in the data (a stale link's old line) has no cell.

**Where review items come from.** Every item has `origin`. `"extraction"` items (kinds `FIELD_UNIDENTIFIED` and `LABEL_EVIDENCE_FAILED`, with `fiscal_year` and `page_number`) are made when a page is extracted, so they can be reviewed before any merge; extracting the page again deletes its items (any status) and makes new ones, and naming the line (`line-overrides`) closes its item. A merge never touches them. `"merge"` items (every other kind) belong to the last merge: **every merge deletes all of them and makes new ones** (they get new ids; a dismissed item is not kept). An item that an active user override already decides (a `FORCE_LINK` / `FORCE_SPLIT` on the same two lines, a `RENAME` or `DETACH` of its line) arrives `resolved` with `resolution: "override"` and that `override_id`. `ORPHAN_OVERRIDE` items stay open. Overrides of `source: "llm"` never resolve items.

**LLM overrides.** A merge links with the user overrides only, so Gemini decides every pair again; each pair it linked (`SAME`) is then listed as an LLM override (the previous LLM overrides are replaced). A user keeps or breaks such a link with `POST .../override/{id}/decide`; user overrides and user disclosures (`cell_disclosure`, `cell_current`) are never deleted by a re-extraction or a re-merge, only by their own delete endpoints.

Every review item has `attention` (`"HIGH"`, `"MEDIUM"` or `"LOW"`) and `attention_rank` (1, 2, 3), derived when the list is read (old items get them too), and the list is sorted most important first (rank, then kind, then id). **An item about a note (`NOTES_AND_FOOTNOTES`) or the statement of changes in equity (`CHANGES_IN_EQUITY`) is always LOW, whatever its kind** (the statement comes from the payload's `statement_type`, else from its `partition`; extraction items carry `statement_type`). Otherwise, HIGH: figures that may be wrong in the merged table (`MAGNITUDE_JUMP`, `INTRA_NODE_CONFLICT`, `PERIOD_COLLISION`, and a `VALUE_CONFLICT` whose change is above 50 %, flips a balance-sheet sign or was rejected). MEDIUM: a link is missing, doubtful or a structure changed (`AMBIGUOUS_LINK`, `STALE_LINK`, `LONG_GAP`, `GAP_BLOCKED`, `SPLIT_DETECTED`, `MERGE_DETECTED`, `SERIES_INCONSISTENT`, `FIELD_UNIDENTIFIED`, `ORPHAN_OVERRIDE`, and a `VALUE_CONFLICT` that is only about the headings). LOW: `RESTATEMENT_CHECK`, `LABEL_EVIDENCE_FAILED`. `RESTATEMENT_CHECK`: Gemini linked two lines whose figures differ by 50 % or less in a common year (a likely restatement); payload as `VALUE_CONFLICT` plus `change_percent` and `change_year`; it is a link kind (accept = force the link, reject = split), and it is not a merged-row value mismatch.
Review item kinds: `AMBIGUOUS_LINK` (Gemini answered `UNSURE`, or left the question out: `payload.verdict` is `"UNSURE"` or `"NO_ANSWER"`; a `NO_ANSWER` payload also lists the `candidates`),
`VALUE_CONFLICT` (payload may also carry `rejected: true` when the same-name pair was NOT linked because another line matches the old values exactly, and `definition_change` `{a_parts, b_parts}` with a "definition changed" reason when the stored breakdowns show a different make-up; a link proposed although the values differ: by Gemini, or a same-name link whose common-year value changes by more than 50 %, flips sign on a balance-sheet line, or whose headings share no word; those carry `payload.rule` `NAME_EXACT` and `payload.reason`, and their merged-row `value_mismatches` rationale reads "Linked by name, but the values differ on common years"), `SPLIT_DETECTED` / `MERGE_DETECTED`
(2–3 lines sum to one line of the next or previous report; payload `partition`, `statement_type`, `scope`, `year`, `from`, `into`, `node_ids`, plus `timeline`, see below), `SERIES_INCONSISTENT` (a chain was broken
because two of its names are printed in the same report), `STALE_LINK` (a Gemini link of the previous
merge that the data no longer supports), `LONG_GAP` (a line linked by name across 2+ skipped reports;
payload `reports_skipped`), `GAP_BLOCKED` (a name match across 1+ skipped reports that was NOT linked because `veto()` fired or the two headings share no informative word; payload like `AMBIGUOUS_LINK` plus `reports_skipped` and `reason`; `accept` links them, also across the gap), `MAGNITUDE_JUMP` (a gap link that was made but whose values in the nearest years differ by more than 50x, sign ignored; payload `ratio`, `a_year`, `b_year`, `a_value`, `b_value`), `FIELD_UNIDENTIFIED` (payload `report`, `page_number`, `metric_id`, `raw_field`, `table_num`, `table_row`, `table_col`, `reason`, `values`: a line whose page does not show what it is), `PERIOD_COLLISION` / `INTRA_NODE_CONFLICT` (one report prints the same line of the same statement twice with
different figures for one year; payload `alt_periods` holds the other printing, `category` the heading and `kept` the cell that was used per year (`{page_number, table_num, table_row, table_col, value}`); never a restatement. The same
figure printed again in the same report raises nothing), `ORPHAN_OVERRIDE` (an override whose lines no longer exist). Overrides and review
decisions apply from the next merge; they set `merged_stale: true`.

`SPLIT_DETECTED` / `MERGE_DETECTED` payloads also carry `timeline` (additive; items saved before this field existed get it after the next merge). It lists every line involved (`role` `from` = before the change, `into` = after) over all reports, so the whole history can be judged:

```json
"timeline": {"years": [2022, 2023, 2024], "reports": ["2022-23", "2023-24"],
  "lines": [{"role": "from", "report": "2022-23", "field": "Others Liabilities", "label_key": "others liabilities",
             "node_id": "...", "series_id": 41, "first_report": "2022-23", "last_report": "2022-23",
             "values": {"2022": 3300000.0, "2023": 3400000.0, "2024": null},
             "stating_reports": {"2022": "2022-23", "2023": "2022-23"},
             "members": [{"report": "2022-23", "field": "Others Liabilities", "label_key": "others liabilities",
                          "node_id": "...", "page_number": 52, "values": {"2022": 3300000.0, "2023": 3400000.0}}]}]}
```

`years` are all years any involved line has a value; `values` has one string key per year (`null` = no value);
`stating_reports` names the report whose value was used (latest report wins); `members` is one entry per report of the line's series with that report's own printed values; `series_id` is `null` when unknown. Lines are ordered `from` first.

### Results response

The results are two endpoints, so the raw table does not wait for the merged one (which needs the restatement dots of every
row). There is no combined `/results` endpoint. Both responses carry the shared status fields:

| Group | Fields |
|---|---|
| Shared (in both) | `instrument_code`, `analyzed_years`, `years`, `has_raw_csv`, `has_merged_csv`, `merged_stale`, `updated_at` |
| Raw (`/results/raw` only) | `raw_columns`, `raw_rows` (one chunk), and the paging fields below |
| Merged (`/results/merged` only) | `merged_rows`, `value_mismatch_conflicts`, `value_mismatch_count`, `merge_report` (all of ONE statement), and the chunk fields below |

**Raw table in chunks.** `GET /results/raw` returns at most **100 rows** per call (a 1,000-row table is about 0.7 MB per chunk instead of
4.7 MB). The rows have a stable order, grouped by statement (`BALANCE_SHEET`, `INCOME_STATEMENT`, ...). Query: `statement_type`
(optional, case-insensitive: only that statement's rows) and `offset` (where the chunk starts; the first call has none). The
response adds:

| Field | Meaning |
|---|---|
| `statement_types` | `[{"statement_type": "BALANCE_SHEET", "rows": 120}, ...]`: the statements of the WHOLE table with their row counts, in table order (for tabs; the same in every chunk) |
| `total_rows` | Rows matching `statement_type` (all rows without it) |
| `limit` | Always `100` |
| `offset` | The offset of this chunk |
| `has_more` | `true` when more rows follow |
| `next_offset` | The `offset` to send for the next chunk; `null` on the last chunk |

`raw_columns` and `years` are the same in every chunk (the table's columns do not depend on the chunk). An unknown `statement_type`, or an
`offset` past the end, gives `raw_rows: []` with `total_rows` as above. The CSV text is not in this response: use `GET /download/raw`.

**Merged table one statement at a time.** `GET /results/merged` returns ONE statement per call: all its `merged_rows` (consolidated and
standalone together). The statements come in table order (`BALANCE_SHEET`, `INCOME_STATEMENT`, `CASH_FLOW`, `CHANGES_IN_EQUITY`,
`NOTES_AND_FOOTNOTES`, `UNKNOWN`), and a statement with no rows is never offered, so no call is wasted: without `statement_type`
the response is the first statement that has rows (a company without a balance sheet starts with its income statement). Query:
`statement_type` (optional, case-insensitive). Only the rows of the chunk are enriched, so each call is fast. The response adds:

| Field | Meaning |
|---|---|
| `statement_type` | The statement this chunk holds (`null` when nothing is merged) |
| `statement_types` | `[{"statement_type": "BALANCE_SHEET", "rows": 89}, ...]`: the statements that HAVE rows, with their merged-row counts, in order (the same in every chunk; for tabs) |
| `has_more` | `true` when another statement follows this one |
| `next_statement_type` | The statement to ask for next, `null` on the last one |

With `view=full` `value_mismatch_conflicts`, `value_mismatch_count` and `merge_report` hold only the entries of this statement. `years` and the status fields are
the same in every chunk. Asking for a statement that has no rows gives `merged_rows: []` and the first existing statement as `next_statement_type`.
The CSV text is not in this response: use `GET /download/merged`.

Measured on the saved data (server side): raw 0.06 to 0.2 s; merged 0.15 to 0.6 s (it used to be 2.7 to 11 s: the merged rows' dots
called `label_key` about 390,000 times; its results are now cached).

- **Always-present fields:** `raw_rows` and `merged_rows` are always arrays, `raw_columns` is an array of subcategorized column objects, and
  `merged_csv_text` was removed from the results (use the download), and `[]` / `""` is what is returned when there is no data. A stale value from an
  earlier response can therefore always be cleared.
- **`raw_columns`** contains hierarchical column metadata subcategorized by `(Report Fiscal Year, Observation Year)`:
  `[{"key": "2014-15__2015", "report_fiscal_year": "2014-15", "report_year_label": "FY 2014-15", "observation_year": 2015, "header": "2015", "observation_date": null, "restated": false, "group_header": "FY 2014-15"}, ...]`.
  A report that prints a year more than once with a **restated** column ("30-Jun-22" next to "1st July 2022 (Re-Stated)") gets one raw
  column for each of those printings, keyed `report__year@date` and headed by the printed date, in the order printed:
  `{"key": "2022-23__2022@2022-06-30", "observation_year": 2022, "header": "2022-06-30", "observation_date": "2022-06-30", "restated": false, ...}`,
  `{"key": "2022-23__2022@2022-07-01", "observation_year": 2022, "header": "2022-07-01", "observation_date": "2022-07-01", "restated": true, ...}`.
  A year the report prints once keeps the plain key and `header` (the year), and `observation_date` is `null`. `cells` and `values` use these keys.
- **`raw_rows` (slim, the default).** Every row is `{field, raw_field, statement_type, scope, unit, metric_type, cells}` and nothing else: `cells[col_key]` is
  `{value, page_number, table_num, table_row, table_col}` for each column that has a figure (a column without one has no key), the report being the
  column's `report_fiscal_year`. Build the reference of a cell as below and open it with `GET /cell-by-position`. The row's concept, behaviour (`action`,
  `qualifier`, `temporal_state`), `category`, `entity`, `segment`, restatement fields, `provenance` and `values` are not sent. `raw_columns`, `years`
  and the paging fields are the same in both views. Use `view=full` for the rows described next.
- **`raw_rows` (`view=full`)** preserves every publication observation without overwriting comparative years:
  - `cells`: keyed by `col_key` (e.g. `cells["2014-15__2015"]`, or `cells["2022-23__2022@2022-07-01"]` for a restated column), holding the rich cell object: `{value, numeric_value, effective_val, multiplier, unit, report_fiscal_year, observation_year, page_number, table_num, table_row, table_col, raw_field, notes}`.
  - There is no `ref` field any more: the frontend builds the coordinate reference from the cell: `"FY " + report_fiscal_year + " | P-" + page_number + " | T-" + table_num + " | R-" + (table_row ?? 1) + " | C-" + (table_col ?? 1)`, for example `FY 2024-25 | P-94 | T-1 | R-4 | C-3` (a missing row or column counts as 1). `GET /cell-by-position` also accepts the four coordinates (`fiscal_year` = `report_fiscal_year`, `page_number`, `table_row`, `table_col`; `table_num`) instead of `ref`.
  - `values`: maps both `col_key` and calendar year to effective values. Missing observations are `null`/omitted for the frontend to render as empty cells.
- **`merged_stale`** is `true` when an OCR or extraction run finished after the last merge, or a
  user merge override was created or deleted since, or the merge was marked invalid by hand (`python -m app.ocr.maintenance reset-reviews`)
  (`database.is_merge_stale`; the LLM overrides a merge writes do not count). The saved merge is kept,
  so the UI should ask the user to re-run it. The flag is always `false` when there is no merge.
- **Merged rows** are one per series (one line item followed across reports). `field` is the line
  name shown to users: the base name for both scopes (no `Consolidated` prefix: the table and the Excel files
  are grouped by scope, and `scope` / `scope_label` say which group a row is in), plus ` - <entity>` at the end when the line
  belongs to an entity other than the reporting company (`Total assets`, `Total assets - City Brokerage Limited`). Only the flat
  merged CSV, which has no scope column, still puts `Consolidated ` in front of a consolidated line.
  `base_field` is the name without prefix or suffix, `scope_label` is `Consolidated` or `Standalone`, and
  `merged_from[0]` is the stored (unsuffixed) name.
  Besides the raw-row fields they carry `series_id`, `merged_from` (every name of the series, canonical first),
  `constituent_details[]` (one per report: `field`, `raw_field`, `scope`, `values`, `report_year`,
  `page_number`, `table_*`, `alias_id`, `link_rule`, `link_confidence`, `flags`), `link_summary[]`
  (`{from_report, to_report, rule, confidence, gap}`), `alt_periods` (18-/6-month columns of transition
  years), `review_flags` (also `SIGN_CONVENTION_CHANGED`), `sign_convention_years` (years in which a linked line is printed with the opposite sign in
  two reports, e.g. "Less: Energy purchase" +62.7bn then (62.7bn); the merged value is the latest report's printed value), `value_mismatches`, `micro_variances`, `breakdown` and `breakdown_of` (see
  [Breakdowns](#breakdowns)). Units follow the metric type:
  `BDT`, `BDT/share`, `Nos.`, `%`, `Times`; `metric_type` is `CURRENCY`, `PER_SHARE`, `COUNT`,
  `PERCENT` or `RATIO`.
- **Merged table rows (slim).** <a id="merged-table-rows-slim"></a>`GET /results/merged` returns, by default (`view=slim`), only what the table needs
  (about a quarter of the size of the former rows). Each `merged_rows` item is
  `{series_id, field, scope, statement_type, unit, metric_type, level, active, values, flags, breakdown, breakdown_of, merged_from_count, mismatch_years}`:
  `field` is the shown name (the same for both scopes; use `scope` to group), `metric_type` (`CURRENCY`, `PER_SHARE`, `COUNT`, `PERCENT`, `RATIO`) is for the number format, `level` is `LINE_ITEM` / `SUBTOTAL` / `GRAND_TOTAL`, `active` is the
  row's checkbox (see Active and inactive rows), `values` is `{year: number}` for the years that have a value (a missing year is an empty cell),
  `breakdown` is `{operation, status, components: [{series_id, sign}]}` or `null` (every line that was ever part of this total in any report: the latest report's lines in its printed order, a line only an older report printed at its printed place; the full row's components also have `last_report`),
  `breakdown_of` the series ids of the totals this line is part of, `merged_from_count` how many names were merged into the row, and
  `mismatch_years` the years with a value conflict. The response has no `merge_report` and no `value_mismatch_conflicts`, only
  `value_conflicts_count` and `value_merge_count` (all statements). Everything else of a row (provenance, `merged_from` names,
  `constituent_details`, `link_summary`, `value_mismatches`, `micro_variances`, `restatement_variances`, ...) is in
  `GET /results/merged/row/{series_id}`; the two lists in `GET /results/merged/conflicts` and `/results/merged/report`. `view=full` gives the
  former response (rows with all of that, and the two lists) for old callers.
- **Flags: what is behind a cell's dots.** `flags` is `{year: [entry, ...]}` for the years with something to show; it replaces the dots. One entry per
  LATER printing of the cell (a later report, or a restated column of the same report), oldest first, each
  `{type, report, date, value, original_value, original_report, difference, percent_change}`: `value` is that printing's figure, `original_value`
  the FIRST printed one (`original_report` its report), `difference` = value - original_value, `percent_change` = difference / |original_value| x 100
  (null when the first figure was 0). `type` says how it differs from the first one:

  | `type` | When | Old dot |
  |---|---|---|
  | `RESTATED_IDENTICAL` | printed again with the same figure | green |
  | `RESTATED_MICRO` | a very small difference: the two agree to rounding for the value kind (`series_keys.agree`: under one printed unit for money, 0.011 for per-share and percent figures, ...) | purple |
  | `RESTATED_LOW` | more than rounding, up to 25 % of the first figure | yellow |
  | `RESTATED_MID` | more than 25 %, up to 100 % | yellow |
  | `RESTATED_HIGH` | more than 100 %, a sign flip, or the first figure was 0 and this one is not | yellow |
  | `USER_DEFINED` | the current figure of the cell was chosen by a user (see Disclosures added by users): `{type, report, date: null, value, origin, disclosure_id, record_key, set_by, set_at}`, last in the list | none |

  Only `RESTATED_HIGH` raises a review item (see below). The same `flags` are on the rows of `view=full` and of `results/merged/row/{series_id}`.
- **Restatement dots:**
  - **Raw table rows (`raw_rows`)**: Always empty dots (`restatement_dots: {}`). The raw table displays all extracted historical values faithfully side-by-side with zero dots.
  - **Merged table rows (`merged_rows`)**: Carry `restatement_dots` (`{year: ["yellow"|"green"|"purple", ...]}`). Computed across publications for unified series.
  - `/results/merged` and `cell-details` compute dots from the same records with the same rule
    (`router._row_records`, `router._restatement_dots`): only the row's own scope; a merged row's series lines (its `constituent_details`, as `cell-details`
    with `series_id`); one dot per LATER report that printed the cell (the original gets no dot). Only a later
    report can restate. Another printing inside the same report and the same statement (another table or page) is
    never a restatement and gets no dot: with exactly the same figure it is fine (e.g. the number of shares printed
    three times); with a different figure the row lists the year in `duplicate_in_report_years` for review.
  - Raw rows are keyed by statement, name, unit, tag, scope and entity, plus the category when the same name is used under
    different headings in one report: two "Total" lines of different notes are two raw rows, each with its own category,
    page and coordinates. Raw rows always have `duplicate_in_report_years: []` and `restated_variance_years: []`;
    `duplicate_in_report_years` is only filled on merged rows and ignores a same-named line of another heading on the same page.
  - Merged rows also carry `has_micro_variance` / `micro_variance_years`.
  - Conflicts carry `field_a_restatement_dots` / `field_b_restatement_dots`.
- **Where each compared figure came from.** Every `collision_details` entry (in `value_mismatch_conflicts` and in the
  rows' `value_mismatches`) and every `micro_variances` entry carries the printed cell of both values, so a figure can be
  linked to its cell without guessing from the row's lines: `a_report`, `a_page_number`, `a_table_num`, `a_table_row`,
  `a_table_col` for `val_a`, and the same with `b_` for `val_b`. `a_` is the older report. A part is `null` when unknown.
  The `VALUE_CONFLICT` review item's payload has the same cells per year in `overlap_sources` (`{year: {a_*, b_*}}`).
- **The whole rows of a conflict (`comparison`).** `collision_details` only lists the years the two reports share. Every
  value mismatch (`value_mismatches` of a merged row, `value_mismatch_conflicts`) and every `VALUE_CONFLICT` review item
  (`payload.comparison`) also carries `comparison`, the two lines side by side over all the years either prints:
  ```json
  "comparison": {
    "years": [2014, 2015, 2017],
    "conflicting_years": ["2015"],
    "year_status": {"2014": "ONLY_A", "2015": "MISMATCH", "2017": "ONLY_B"},
    "lines": [
      {"side": "a", "report": "2015-16", "field": "Earnings Per Share", "label_key": "eps", "node_id": "2015-16|eps|OCI",
       "section": "OCI", "unit": "BDT/share",
       "values": {"2014": 3.96, "2015": 5.06},
       "cells": {"2014": {"page_number": 36, "table_num": 1, "table_row": 19, "table_col": 6},
                 "2015": {"page_number": 36, "table_num": 1, "table_row": 19, "table_col": 5}}},
      {"side": "b", "report": "2016-17", "...": "same shape for the newer report"}
    ]}
  ```
  `years` are integers in ascending order; the keys of `year_status`, `values` and `cells` are the same years as strings.
  `year_status` is `MATCH`, `MISMATCH` (both print the year, the figures differ) or `ONLY_A` / `ONLY_B` (one report only).
  `values` are the figures as the merge uses them (multiplier applied), `cells` the printed cell of each (open it with
  `GET .../cell-by-position`). `lines[0]` is always `a`, the older report. Present on items saved after this field
  existed (after the next merge); a conflict without it falls back to `collision_details`.
- **`raw_csv_text`** is rebuilt from the stored metrics.

### CSV text

`csv_text` in `raw_csv_data`, `merged_csv_data` and the results endpoints, and the downloads, are protected
against formula injection (`csv_service.csv_safe_cell`):

- A text cell that starts with `=`, `+`, `-`, `@`, a tab or a CR is prefixed with `'`.
- Numbers, including negatives such as `-5`, are left alone.
- `csv_service.parse_csv_text_to_rows` strips that prefix again.
- Only the downloads start with a UTF-8 BOM. The `csv_text` strings in JSON do not, so a client
  that adds its own BOM does not end up with two.

### Cell details

- **Query:** `year` (required, 1900-2200) and one of `series_id` (the merged row's `series_id`) or `metric_id` (a stored metric's id,
  e.g. `metric_id` of a breakdown part or total, of a record, or of a printed cell: its series in the last merge is used; ignored when
  `series_id` is given). The line is the series and the column is `year`: the records are exactly the series' members (the lines the
  merge linked across reports), and the field, statement, scope and category (heading) come from the series, so lines that share a
  name with another series are never mixed in. Unknown series: `404`; a `metric_id` that is in no series (a line the page does not
  name, or not merged yet): `404`; neither `series_id` nor `metric_id`, or no `year`: `422`. There is no lookup by field name (the
  `field`, `statement_type`, `scope`, `merged_from` and `category` parameters were removed; extra query parameters are ignored). For a
  cell of the raw table use `GET /cell-by-position`.
- **Response:**
  - `cell_records`: only records of the series' scope, and
    exactly ONE record per publication (report, page, table, restated column or not): the cell itself. Other printed cells are
    never returned as part of it; how cells add up is in `breakdown` / `part_of`. A raw row of a segment
    column is named `Field (Segment)` and returns that column's cells; the plain `Field` row returns the
    cells without a segment (e.g. the Total column).
    Each has `date` (the printed date of its column, or `null`), `restated` (`true` for a column the report marks
    "Re-Stated") and `status` (`ORIGINAL_STATED` for the first report that printed the value, `RESTATED`
    for the first printing in a later report **and for a restated column of the same report** (it counts for the dots), `REPEATED_IN_REPORT` for another printing in the same report with
    exactly the same figure (fine), `DUPLICATE_IN_REPORT` for another printing in the same report with a different
    figure (`needs_review: true`); neither repeat is a restatement or gets a flag), `restatement_type`
    (for a `RESTATED` record how it differs from the first printed one: `RESTATED_IDENTICAL` / `RESTATED_MICRO` / `RESTATED_LOW` / `RESTATED_MID` / `RESTATED_HIGH`, see [Flags](#merged-table-rows-slim); otherwise `null`), `scope`, `entity`, `unit`, …, `notes` (note references of the record's line) and `note_cell` (`{table_num, table_row,
    table_col}` of the printed cell holding the reference, or `null`), plus `breakdown` (the cells that add up to this record's cell, or `null`) and
    `part_of` (the totals this cell is a part of, one object each, **with their `parts`**: the sibling lines, and this cell's own line among them, so "Part of calculation" can show `Non-current Assets + Current Assets = Total Assets`). Both are
    `{composition_id, direction (ROWS / COLUMNS), operation, expression, status (MATCH / MISMATCH), difference, parts_sum, total: {row, col, value, field, raw_field, metric_id, year, statement_type, scope, series_id}, parts: [{row, col, sign (1 or -1 for a deducted part), value (the number exactly as printed, with its own sign: show this), applied (what the part adds to the total: value, or -abs(value) for a deducted part), field, raw_field, metric_id, year, series_id}]}`.
  - `duplicate_in_report` (true when a record of the year is `DUPLICATE_IN_REPORT`, i.e. one report gives two
    different figures), `restatement_variance` (original vs the latest RESTATED record; `null` without one),
    `flags` (the same entries as a merged row's `flags[year]` for this cell: one per later printing with its `type`, `value`, `original_value`,
    `difference`, `percent_change`, and a `USER_DEFINED` entry last when a user chose the current disclosure), `micro_variance`, `has_micro_variance`
    and `total_disclosures` (extracted ones; added ones are `user_added_count`). There is no timeline and no `restatement_dots` here: the years of a row
    and their flags come from `GET /results/merged`.
- **Contract:**
  - `report_year` is the fiscal-year string of the report that disclosed the value (`"2024-25"`).
  - `restatement_variance` is either `null` or an object
    `{original_value, restated_value, difference, percent_change}`; the values are numeric and `percent_change` is `null` when the original value is 0 (not applicable).
    The merged rows' `restatement_variances[year]` (and the raw CSV rows') follow the same rule, and merged rows add `history[]` (one entry per pair of consecutive reports whose figure for that year differs).
  - `micro_variance` is either `null` or an object whose `val_a`, `val_b`, `diff` and
    `percent_change` are all numbers. A partially recorded variance is returned as `null`.
  - Every record has a normalized `unit`. It falls back to `BDT`, or `Times`
    for ratios, when the source had none.
  - The response always has the full shape. A request whose `field` is blank also carries
    `error: "No valid field specified"`.

### Cell by position

The returned `metric` also has `series_id` (the cell's series in the last merge, or `null`).

Fast, direct cell provenance endpoint designed for the document review Markdown page (`/review/{instrument}/{fy}`). Resolves the LLM extraction metadata and raw Markdown content directly from physical table coordinates or reference strings, completely bypassing field-name matching and segment filtering.

- **Query Parameters:**
  - `ref` (optional, string): Full reference string formatted like `"FY 2002-03 | P-59 | T-2 | R-9 | C-3"`, `"FY 2016 | P-36 | T-2 | R-1 | C-2"`, or `"2016-17 | P-35 | R-16 | C-4"` (table defaults to 1 if omitted; single calendar years automatically resolve to the matching report fiscal year). Any explicitly provided query parameters override parsed reference string tokens.
  - `fiscal_year` (optional\*, string): Report fiscal year, e.g. `"2002-03"`, `"2016"` or `"FY 2002-03"`.
  - `page_number` (optional\*, integer): Page number in the document.
  - `table_num` (optional, integer): Table index on page (default: `1`).
  - `table_row` (optional\*, integer): 1-indexed row number in the Markdown table.
  - `table_col` (optional\*, integer): 1-indexed column number in the Markdown table.
  - *\* Either `ref` or all four coordinate parameters (`fiscal_year`, `page_number`, `table_row`, `table_col`) must be provided (returns `422` if incomplete).*

- **Response:**
  - `found` (`true` \| `false`): Whether a numeric financial metric was extracted at these exact coordinates.
  - `ref`: the cell's reference string, always sent, in the canonical form `FY 2018-19 | P-68 | T-1 | R-4 | C-3` (the report year as stored, table 1 when none was given), whichever way the cell was asked for. The same string is `coordinates.reference` and `coordinates.ref`.
  - `coordinates`: `{ reference, ref, instrument_code, fiscal_year, page_number, table_num, table_row, table_col }`.
  - `raw_cell_content`: The unparsed cell text extracted directly from the page's OCR Markdown table grid (e.g. `"**2,948,053,854**"`).
  - `metric` (when `found: true`):
    - `metric_id`: Integer primary key in `financial_metrics`.
    - `concept_name`: Canonical base concept (e.g. `"Profit after Tax for the Period"`).
    - `raw_field`: Text label as extracted from the row.
    - `display_field`: Stored display field (includes segment suffix if present, e.g. `"Profit after Tax for the Period (January - June 18 Months)"`).
    - `statement_type`, `category`, `aggregation_level`, `scope`, `entity`, `segment`, `temporal_state`, `action`, `metric_type`, `qualifier`, `tag`.
    - `data_year`, `data_date`, `numeric_value`, `multiplier`, `effective_value`, `unit`, `calculation_formula`.
    - `verification_flag`, `flag_reason`.
    - `notes`: Array of note references attached to the line (e.g. `["Note 32"]`).
    - `composition`: Arithmetic composition links (breakdown / part-of) if this cell is part of an equation check.
  - `message` (when `found: false`): Explanatory string indicating why no metric was extracted (e.g. header column, row label, non-numeric cell).

### Disclosures added by users and the current disclosure

A cell of the merged table is a row (`series_id`) and a column (`year`). Its **disclosures** are what each report printed for it (extracted by the
model, one per publication, a restated column of a report being its own) and the one a user may enter.
- **Origin.** Every record of `cell-details` has `origin`: `AI_EXTRACTED` (read from a printed cell) or `USER_ADDED`, plus `disclosure_id` (null
  for an extracted one), `created_by` (the user, null for an extracted one) and `record_key` (see below).
- **One per cell, replaced when entered again.** A cell (series and year) has at most ONE user disclosure. Entering again REPLACES it (the new figure, note, date, user and time; the `disclosure_id` and `record_key` stay, so a choice of it as the current disclosure holds). The extracted disclosures are never touched or deleted. The replaced figure is kept in the event log.
- **`POST /enter-disclosure`** `{series_id, year, value, note, unit?, date?}`: `value` is the amount in the row's unit, already scaled
  (`476000`, not 476 thousand); `note` (1 to 500 chars) says why; `unit` defaults to the row's; `date` is `YYYY-MM-DD`. Works for an empty cell too. It does **not change what the table shows**: the merged table, the dots
  and the current disclosure stay as they were. `200 {disclosure_id, record_key, series_id, year, value, unit, date, note, created_by, created_at, replaced}` (`replaced` is true when it replaced an earlier one; if that one was the current disclosure, the table shows the new figure at once); `404` unknown series; `422` bad body. Recorded as `DISCLOSURE_ENTER`, with the replaced figure in its details.
- **`DELETE /enter-disclosure?series_id=&year=`** deletes the user's disclosure of that cell (only the user's own entry; the extracted ones cannot be deleted). If it was the cell's current disclosure the choice goes with it and the cell is back to its automatic current one (the latest extracted). `200 {series_id, year, deleted, current_reset}`: `deleted` is false when the cell had none, `current_reset` true when a choice was removed; `404` unknown series. Recorded as `DISCLOSURE_DELETE`, with the deleted figure in its details; the cell can be entered again afterwards.
- **Added records** appear in `cell_records` after the extracted ones with `status: "USER_ADDED"`, `user_note`, `created_at`,
  `date`, `effective_value` (= `numeric_value`, `multiplier` 1), `restatement_type` null, `diff_from_original` (against the first extracted figure, null
  when there is none), and the printed-cell fields (`report_year`, `page_number`, `table_*`, `metric_id`, `raw_field`) null; `breakdown` null, `part_of` [].
- **Current disclosure.** Exactly one disclosure of a cell is current (`is_current: true`; the one marked CURRENT). Without a choice it is the
  latest extracted one (the last of the extracted records). `current_record_key`, `current_is_override` (true when a user chose it),
  `current_selection` (`{record_key, set_by, set_at, missing}` or null; `missing: true` when the chosen disclosure no longer exists, e.g. its page
  was extracted again, and the automatic one is used) and `user_added_count` are on the response.
- **`PUT /cell-current`** `{series_id, year, record_key}` chooses it. This is an **overwrite**: one choice per cell, the next replaces it (the
  disclosures themselves are untouched). `record_key` must be one of the cell's `cell_records[].record_key` (`422` otherwise; `404` unknown
  series). **`DELETE /cell-current?series_id=&year=`** removes the choice (`{series_id, year, reset}`). Recorded as `CELL_CURRENT_SET` /
  `CELL_CURRENT_RESET` (the event log keeps who chose what and when).
- **`record_key`** names a disclosure: `ai:{report}|{page}|{table}|{row}|{col}` for an extracted cell (page coordinates, so it stays valid after a
  re-merge) and `user:{id}` for an added one. Treat it as an opaque string.
- **Effect on the tables.** In `GET /results/merged` the year's value of a row (`values[year]`) is the chosen disclosure's value, `stating_years[year]`
  its report (`"USER"` for an added one), and the row has `overrides[year] = {record_key, origin, disclosure_id, set_by, set_at}`. The merged Excel
  download shows the chosen value (comment "Chosen by ..."). The merge is not re-run, and the choices are kept by a re-merge (they are stored
  by the series' identity, not only by its id). The merged CSV text, the restatement dots and the `restatement_variance` are not changed by a choice.
  The cell's `flags` get a `USER_DEFINED` entry when the choice is a user's (on `cell-details` and on the merged row).

### Active and inactive rows

A user can leave merged rows out of the Excel downloads (the "checkbox" on each row of the merged table).
- `PUT /ocr/company/{code}/series/{series_id}/active` with `{"active": false}` (or `true`), and `PUT /ocr/company/{code}/series-active` with
  `{"series_ids": [...], "active": false}`. Both need `X-Username` and are recorded as `SERIES_DEACTIVATE` / `SERIES_ACTIVATE`.
  Response: `{active, updated: [ids changed], unchanged: [ids already in that state], unknown: [ids that are no series of the company],
  inactive_series_count}`. The single call returns `404` for an unknown series; the bulk call ignores unknown ids and lists them.
- The choice is stored on the series (`series.excluded`), needs no new merge, and stays while a re-merge keeps the series id (ids are
  kept stable between merges); a series that gets a new id starts active.
- `GET /results/merged`: every merged row has `active` (`true` unless switched off) and the response has `inactive_series_count`.
  Inactive rows stay in the results so they can be switched back on.
#### Standard tab

`GET /ocr/company/{code}/results/standard` returns `{instrument_code, years, merged_stale, updated_at, sheets, checks}`. `sheets` are in the order of the workbook (`Balance Sheet`, `Income Statement`, `Cash Flow`, `Changes in Equity`, `Notes & Others`; only those with rows; a statement with both scopes is one sheet per scope), each `{name, statement, scope, style: "balance" | "flow", rows}`. The rows are exactly the rows of the xlsx sheet, in order, with the figures already calculated (Excel would calculate the "As calculated" and check rows, so they are computed here with the same arithmetic). Every row has `kind`, `name`, `depth` (indent), `unit`, `series_id` and `values` (`{year: number | null}`):

| `kind` | What | Extra fields |
|---|---|---|
| `banner` | section title (`ASSETS`, `PER SHARE DATA`, a statement type on Notes & Others) | none |
| `header` | group title over its own lines (Balance Sheet); the name ends with a colon (`Cash total:`) and, like every banner, header, line and total name, starts with a capital letter (the first letter is upper-cased when the merged name starts in lower case; nothing else changes) | none |
| `line` | a merged row with no parts (an input; blue in the xlsx) | `restated` `{year: original value}` (yellow cell), `chosen` `{year: {set_by, origin}}` (a user's current disclosure) |
| `total` | a merged row that has parts (bold on grey). **Its `values` are the figures the reports print** (a user's chosen disclosure included); only for a year no report prints a total for, the calculated figure stands in | `calculated` (false: no usable breakdown, or not a plain sum, so the merged value is kept and no `calculated` / `check` rows follow); when calculated: `parts [{series_id, sign}]` (the tree's children, the union across reports), `parts_by_year {year: [{series_id, sign, value, field?}]}` (the breakdown the report that states that year printed; only for years that have one, see below), `calculated_years` (years the parts add up to a figure: the figures of the `calculated` row), `calculated_only` (years where the reports print no total at all: `values[year]` is the calculated figure, light blue with a comment in the xlsx), `reported_only` (years where no part has a figure, usually because the older reports print the total without its breakdown: only the printed figure is shown, the `calculated` row and the check are `null`), `restated`, `chosen` |
| `calculated` | **"As calculated"**: the total added up from its parts (grey italic; formulas over the parts' main cells in the xlsx); `null` for a year no part has a figure | `of` (the total's `series_id`) |
| `check` | **"Check (calculated - reported)"**: the `calculated` row minus the total's printed figure (grey italic); `null` for a year the reports print no total, no part has a figure, or the stating report stored no breakdown (nothing to check) | `of`, `ok` `{year: bool}` (within rounding), `tolerance` (5 for amounts, 0.01 for per-share and ratios), `notes {year: text}` (only for a year whose stored breakdown does not add up to its own printed total, status `MISMATCH`: why the check cannot match, an extraction or OCR error; show it as a tooltip, the xlsx has it as a cell comment) |
| `balance_check` | Balance Sheet only: total assets minus total equity and liabilities (a per-share total such as "Net Asset Value Per Share" is never taken as one of the two; a name that starts with "Total assets" / "Total ... liabilities" is preferred) | `refs {assets, funding}` (series ids), `ok`, `tolerance` |
| `gap` | the empty row after a check | none |

A calculated total adds, **for each year, the parts the report that states that year printed** (`parts_by_year[year]`: the merged row's `breakdown.by_year[year]`, only a breakdown whose own parts add up to its printed total, status `MATCH`; a year whose stored breakdown does not add up, `MISMATCH`, is still checked, against the union): a part is counted with its series' figure for the year, or, when the part is in no series or its series is not on that sheet, with the figure as printed (`value`). **A year whose stating report stored no breakdown at all has nothing to check: its `calculated` and `check` cells are `null` (the year is not in `ok`, it is in `reported_only`; the xlsx cells are empty), and a total with no such year gets no `calculated` / `check` / `gap` rows (`calculated: false`)**. Exceptions that are still added up from `parts` (the union of the breakdowns across reports, each line counted once, under the total that lists it in the newest report, so the latest layout of the statement is the one drawn; among totals that list it in the same report the smallest wins, then the row order; it also decides where a line is drawn): a plus part adds its value, a minus part subtracts its size (`-ABS`), an empty part counts as 0 (a part that is itself a total counts with its main value: the printed figure, or the calculated one for a year no report prints it). `checks` is `{total, off}`: the check cells over all sheets and those outside rounding. `xlsx_export.standard_layout` builds it and `build_standard_workbook` writes the same rows as Excel formulas, so the tab and the file never disagree.

- `GET /download/standard.xlsx` (Standard Excel) has the sheets `Balance Sheet`, `Income Statement`, `Cash Flow`, `Changes in Equity` and `Notes & Others` (only those with rows; a statement with both scopes gets one sheet per scope, `Income Statement - Consolidated`, shortened to `- Cons.` when the name would pass Excel's 31 characters). Every sheet is laid out from the breakdown tree, with the printed totals, the "As calculated" formula rows and the check rows (below). **Income Statement, Cash Flow, Changes in Equity and Notes & Others** read as the statement is printed: a group's lines first (indented one level, blue), then its total at the group's level, with no banners, no group headers and no added "Total"; a total that is part of a larger one stays at the same level, so the results read down one column (Revenue, Cost of Sales, Gross Profit, ..., Net Profit; or the cash flow sections, Net Increase, Opening Cash, Closing Cash). A minus-signed part is subtracted as `-ABS(...)` (a deduction counts whichever way it is printed, as in the extraction's own check). Lines in no total follow their section; per-share lines go under a `PER SHARE DATA` banner at the end of the sheet. Notes & Others has a banner per statement type (`NOTES AND FOOTNOTES`, ...). Only the Balance Sheet has the assets / equity-and-liabilities check. **The Balance Sheet** is laid out as follows: header `Line Item | Unit | years`; a banner row per top-level total (ASSETS, EQUITY AND LIABILITIES: the total's name without "Total"); a group header (bold, blank values) over each group that has lines of its own, its lines indented one level under it in blue, then the group's grey bold `Total <name>` row (a group made only of other groups has no header, only its total). A total is an **Excel formula** over its direct children in the tree (the union of the parts across reports, each line counted once): `=SUM(C4,C5,C6)` for lines, `=C7+C13` for a total made only of calculated totals, a minus-signed part subtracted as `-ABS(...)`. The total row itself holds the figure the reports state (the merged value; a calculated figure, blue, only for a year no report prints a total). Under each calculated total: a grey `As calculated` row (the formula over the parts) and a grey `Check (calculated - reported)` row (`=calculated - reported`, 0 when they match; up to 5 units are rounding). Group headers end with a colon (`Cash total:`). Every check row (these and the assets/equity one) is followed by one empty row. A total that is not a plain sum (another operation) or has no children keeps its merged value and has no such rows. Then an italic grey `Check: <assets total> - <equity and liabilities total>` row (formulas over the printed totals), and the `PER SHARE DATA` (per-share / non-currency lines in no total) and `OTHER LINES` banners. Zero shows as `-`, restated cells are yellow with a comment. Line values are the merged values; a year with no printed figure under a total leaves its formula cell empty. It leaves the inactive rows out like the merged one.
- `GET /download/merged.xlsx` leaves the inactive rows out (their children keep their place, indented one level less when their
  parent is out). `GET /download/raw.xlsx` leaves out the raw rows whose cells all belong to inactive series (a row in no series, or
  with any cell in an active series, stays). The CSV downloads and `results/raw` are not filtered.

### Breakdowns

During extraction the model lists, per table, which cells add up to a total (`"k"` in the extraction
answer, see [DATA_MODEL.md](DATA_MODEL.md#extracted-metric-json)): a total row and its component rows
(applied to every value column), or a total column and its component columns (applied to every row).
The code expands that to cells, reads the printed numbers and checks the arithmetic. `operation` says
how the parts make the total: `SUM` (default; the signed parts add up, within half a unit of the last
printed digit per number), `RATIO` (first part ÷ second, e.g. earnings per share), `PERCENT` (first ÷
second × 100) or `PRODUCT` (the parts multiplied). For the last three the check allows half a printed
unit of the total plus 0.1% (the inputs are printed rounded). `status` is `MATCH` when it holds,
otherwise `MISMATCH` (kept, not dropped; `difference` = computed value − printed total, `parts_sum` =
the computed value, both `null` when a part is empty or the divisor is zero). A deducted part
("Less: …") has `sign: -1` and always counts as a deduction, whichever way it is printed. A part's `value` is always the printed number with its own sign
(a printed `(1,079,484,428)` is `-1079484428`); show it as printed and show the `+` / `-` from `sign` beside it; `applied` is
what it contributes. For `RATIO`
and `PERCENT`, `parts[0]` is the numerator and `parts[1]` the denominator. Every composition also has
an `expression`, a readable formula built from the part names (`Net profit / Number of shares`,
`Government + Others`). Breakdowns are stored per page and replaced when the page is
re-extracted.

Every `total` and every part of a composition (also in `cell-details` `breakdown` / `part_of` and in `GET /compositions`) carries
`series_id`: the merged row of that cell in the last merge, `null` when the cell is in no series (a row that is no metric, a line the page
does not name, or no merge yet). Open its history with `GET /cell-details?series_id={series_id}&year={year}`, or with
`?metric_id={metric_id}&year={year}`.

Where they appear:
- `GET /ocr/company/{code}/year/{fy}/compositions[?page_number=]`: every breakdown of a report.
- `cell-details`: `breakdown` / `part_of` on each record.
- merged rows: `breakdown` = `{report, direction, operation, status, components: [{series_id, field, sign, last_report}], by_year?}`
  (the UNION of the parts of every report that prints the total: the latest report's order and signs, a line that only an older report printed inserted at its printed place; `report`, `operation` and `status` are the latest report's; `last_report` is the latest report that printed the component, so a line the latest report no longer prints has an older `last_report`, and the parts of the latest year can then exceed the total if a line was reclassified; `series_id` is `null` for a part that is not a
  merged row) or `null`, and `breakdown_of` = series ids of the totals this row is a part of. `by_year` (always present on a row with a breakdown, empty when no year has one) = `{year: {report, status, components?: [{series_id, sign, value, field?}]}}`: the breakdown the report that states that year printed for the very cell that year's value comes from (SUM breakdowns only; `components` only when `status` is `MATCH`, that is the parts add up to the printed total; a `MISMATCH` entry has the status only; `value` is the figure as printed, `field` only for a part in no series). A year with no entry has no stored breakdown: the Standard tab does not check it. The Standard tab adds each year up from it; the slim `/results/merged` rows do not carry it.
- extraction results: `compositions_count`, `composition_mismatches` per page.

### Usage report

- **`recent_runs[]`:**
  - `id`, `process_type` (`OCR`, `EXTRACTION`, `MERGE`).
  - `fiscal_year`: `null` for company-wide `MERGE` runs.
  - `fiscal_year_label`: the stored label, for example `2023-24-2024-25`.
  - `fiscal_years`: the years the run covered.
  - `total_pages`, `duration_sec`, token counts, `model` and `created_at` (ISO-8601 with `Z`).
  - With a fiscal-year filter, MERGE runs covering that year are included.
- **`by_year[]`:**
  - Per-year OCR, extraction and merge usage: `merge_tokens` and `merge_duration_sec` hold the
    year's share of each MERGE run, split evenly across the years the run covered. `total_*`
    includes merge.
  - `page_count`.
- **`summary`:**
  - Totals for `ocr`, `extraction`, `verification` and `merge`.
  - With a fiscal-year filter, `merge` and the combined totals count only that year's share.
- **`pages[]`:** per-page usage plus `extraction_status`, `extraction_message` and
  `verification_status`. See [Statuses](#conventions).

## Event log

Who did what. Every route that needs `X-Username` writes one row to the `event_log` table
([DATA_MODEL.md](DATA_MODEL.md#event_log)) when the action starts (`status: STARTED`) and
finishes it as `SUCCEEDED`, `FAILED` (the error and HTTP status are in `details`) or `CANCELLED`
(a merge stopped by a client disconnect or the idle timeout). Responses of those routes are
unchanged. A job (`/ocr/jobs/*`) writes its row when it is queued (so a queued job is visible), stores the
`job_id`, and the worker finishes the row; a job cancelled before it started is `CANCELLED`. Not recorded: reads, the
automatic PDF registration done by `GET .../pages` and `GET .../pdf`, requests refused before
anything happened (`400` missing username, `401`, `409` merge already running), and the LVaR routes.
Implemented in [app/ocr/events.py](../../app/ocr/events.py).

| Method | Path | Handler | Notes |
|---|---|---|---|
| GET | `/ocr/events` | `list_events` | Newest first. Query: `username`, `instrument_code`, `action`, `status` (`STARTED`, `SUCCEEDED`, `FAILED`, `CANCELLED`), `since`, `until` (ISO date or timestamp; an `until` date includes that whole day, UTC), `limit` (1 to 500, default 100), `offset`. Returns `{total, events: [...]}`; `total` counts all matches, not just the page. Needs only `X-API-Key`. |

An event:
```json
{"id": 620, "created_at": "2026-10-27T04:46:48.177162Z", "finished_at": "2026-10-27T04:46:48.2Z",
 "username": "Fin001", "action": "REVIEW_ACCEPT", "status": "SUCCEEDED",
 "instrument_code": "DESCO", "fiscal_year": null,
 "target_type": "review_item", "target_id": "812", "job_id": null, "source": "api",
 "details": {"review_kind": "VALUE_CONFLICT", "note": "same line", "override_id": 41, "status": "resolved"}}
```
`fiscal_year` is set for actions on one report. `target_type` / `target_id` name the thing that
was changed (`review_item`, `merge_override`, `line_override`, `pipeline_run`); `target_id` is
always a string. `source` is `api`, or `backfill` for history recorded afterwards from the stored
tables (see below). `details` depends on the action.

| `action` | Route | `details` |
|---|---|---|
| `REPORT_UPLOAD` | `POST /ocr/upload` | `documents: [{filename, fiscal_year, total_pages, verification_cleared?}]` |
| `REPORT_REGISTER` | `POST /ocr/register-document` | `filename`, `total_pages`, `replaced`; `verification_cleared: true` when a new file cleared the checked mark |
| `REPORT_VERIFY`, `REPORT_UNVERIFY` | `POST .../year/{fy}/verify` | `note`, `was_verified`, `previous_verified_by`, `previous_verified_at`; target `document` / its id |
| `PAGES_OCR` | `POST /ocr/pipeline/ocr`, `/ocr/pipeline/ocr-client-pages`, `/ocr/jobs/ocr`, `/ocr/jobs/ocr-client-pages` | `pages`, `completed_count`, `total_tokens`; `job` and `client_pages` when they apply |
| `PAGES_EXTRACT` | `POST /ocr/pipeline/extract`, `/ocr/jobs/extract` | `pages`, `items_extracted_count`, `total_tokens`, `failed_pages` (pages whose extraction failed; the action itself still succeeds); `job` for a job |
| `MERGE_RUN` | `POST /ocr/pipeline/merge`, `/ocr/pipeline/merge/stream`, `/ocr/jobs/merge` | `merged_rows_count`, `merge_count`, `conflicts_count`, `total_pairs`, `total_tokens`; `stream: true` for the stream, `job` for a job |
| `REVIEW_ACCEPT`, `REVIEW_REJECT`, `REVIEW_DISMISS` | `POST .../merge/review/{id}/resolve` | `review_kind`, `note`, `override_id` (the override a link decision created), `status` |
| `OVERRIDE_FORCE_LINK`, `OVERRIDE_FORCE_SPLIT`, `OVERRIDE_RENAME`, `OVERRIDE_DETACH` | `POST .../merge/override` | the request fields (`kind`, labels, reports, `new_label`, `note`) |
| `OVERRIDE_REMOVE` | `DELETE .../merge/override/{id}` | none |
| `OVERRIDE_MAKE_PERMANENT`, `OVERRIDE_BREAK_LINK` | `POST .../merge/override/{id}/decide` | `note`; `split_override_id` for a break |
| `LINE_NAME_SET` | `POST .../line-overrides` | `page_number`, `table_num`, `table_row`, `field`, `note`, `updated_metrics` |
| `LINE_NAME_REMOVE` | `DELETE .../line-overrides/{id}` | none |
| `DISCLOSURE_DELETE` | `DELETE .../enter-disclosure` | `year`, `deleted`, `disclosure_id`, `previous` (`{value, unit, date, note, created_by, created_at}`), `current_reset` |
| `DISCLOSURE_ENTER` | `POST .../enter-disclosure` | `year`, `value`, `note`, `disclosure_id`, `replaced`, `previous` (`{value, unit, date, note, created_by, created_at}` of the replaced one) |
| `CELL_CURRENT_SET` / `CELL_CURRENT_RESET` | `PUT` / `DELETE .../cell-current` | `year`, `record_key` (set), `reset` (reset) |
| `SERIES_ACTIVATE` / `SERIES_DEACTIVATE` | `PUT .../series/{id}/active`, `PUT .../series-active` | `series_ids` (first 100), `count`, `updated`, `unknown` |
| `OCR_MD_SYNC` | `GET` / `POST /ocr/sync-ocr-md` | none |
| `JOB_CANCEL` | `DELETE /ocr/jobs/{job_id}` | `kind`, `page_number`, `status_before`; target `job` / the job id |

A `FAILED` event has `details.error` and `details.status_code` (for example `404` for an unknown
override). A `STARTED` event that never finishes means the server stopped during the action.

History before this feature was recorded once with
`python -m app.ocr.maintenance backfill-events --user Fin001`: uploads, OCR / extraction / merge
runs, line names, and any overrides and review decisions, each with the time the tables stored
(`source: backfill`, `status: SUCCEEDED`). Failed runs, deleted line names and review notes were
never stored, so they are not in it. The command backs the database up first and does nothing
when backfill events already exist.

## OCR Markdown maintenance

| Method | Path | Handler | Notes |
|---|---|---|---|
| GET/POST | `/ocr/sync-ocr-md` | `sync_ocr_markdown_endpoint` | Needs `X-Username` (event `OCR_MD_SYNC`). Writes every `pages.ocr_markdown` from the DB to `ocr-md-files/`. `storage_path` in the response is the relative label `storage/ocr-md-files`. |
| GET | `/ocr/ocr-md-files` | `list_ocr_markdown_files` | Lists the Markdown files on disk with their sizes |
