# OCR Processing Pipeline

The engine processes each `(instrument_code, fiscal_year)` document in five stages. The first four
run per year. The merge runs once per company, across all of its years. See
[ARCHITECTURE.md](ARCHITECTURE.md) for context, [API.md](API.md) for auth and validation rules, and [DATA_MODEL.md](DATA_MODEL.md) for the tables
each stage writes.

```
Stage 1 Ingest ─▶ Stage 2 OCR ─▶ Stage 3 Extraction ─▶ Stage 4 Raw CSV ─▶ Stage 5 Merge
 (per year)        (per page)     (per page)            (per company)      (per company)
```

---

## Stage 1: Ingest and page rendering

| Entry point | Behaviour |
|---|---|
| `POST /ocr/upload` (`router.upload_documents`) | Validates every file (signature, `MAX_UPLOAD_MB`, fiscal year), then saves each PDF as `app/ocr/storage/uploads/{CODE}_{FY}.pdf`, renders **all** pages with `ocr_service.render_pdf_to_images` (pypdfium2, 150 DPI), then calls `database.upsert_document` and `database.upsert_page` for each page. |
| `POST /ocr/register-document` (`router.register_document_fast`) | Saves the PDF and registers the document with a client-supplied page count. Renders nothing. Optional: OCR works without it. Registering the same year again replaces the PDF and page count (`replaced: true`) and keeps the existing pages and metrics. |
| `POST /ocr/pipeline/ocr-client-pages` (`router.run_ocr_client_pages`) | Receives page PNGs that the browser already rendered, saves them to `app/ocr/storage/pages/{CODE}/{FY}/` (`_save_client_pages`), and runs live OCR on them right away in a worker thread (`_ocr_saved_client_pages`). `POST /ocr/jobs/ocr-client-pages` does the same as a background job. |
| `router.resolve_pdf_path` | Finds the PDF from the DB record or the `uploads/` naming convention, and repairs `documents.filepath` when it is stale. |

When a page has no PNG yet, the OCR and extraction stages render it on demand with
`ocr_service.render_single_pdf_page`.

## Stage 2: OCR (image → Markdown)

Entry: `POST /ocr/pipeline/ocr` → `router.run_ocr_pipeline(OcrRequest)`, or the same as a
queued job per page through `POST /ocr/jobs/ocr` (Stage 3: `POST /ocr/jobs/extract`, the merge: `POST /ocr/jobs/merge`). See
[API.md § Job queue](API.md#job-queue).

**Page handling**
- PDF opening and page rendering go through `ocr_service.PDF_LOCK` (PDFium is not thread-safe; parallel
  renders of one PDF failed with "Failed to load page"). The Gemini calls still run in parallel.
- `_run_per_page` drops duplicate page numbers and processes up to **`OCR_PAGE_PARALLELISM` (default 10) pages in parallel**. All
  Gemini calls go through the step's own limiter (`rate_limiters["ocr"]` / `["extract"]`).
- A page without a PNG is rendered from the document's PDF. If no document row exists but
  `uploads/{CODE}_{FY}.pdf` does, the document is registered first.
- If a requested page has no rendered image and there is no PDF to render it from, the request
  fails with `404` before any OCR runs. A page whose image cannot be rendered or OCR'd gets the
  generic error Markdown.

**OCR call:** `ocr_service.run_gemini_ocr_on_image(image_path, context=router._ocr_context(code, fy, page))`
- The model is told the company, security code, fiscal year and PDF page, and writes them as the first
  line of the Markdown: `Source: <company> (<code>) | FY <fy> | Page <n>` (`ocr_service.page_source_line`).
  When the model leaves the line out, the code adds it.
  That line is system context, not printed text: the extraction prompt says never to quote it, and the
  label-evidence check removes it before matching quotes.
- Every OCR result then passes `ocr_markdown.normalize_note_tables`, which guarantees two rules the prompt
  asks for but the model does not always follow (the text is never changed, only where it sits):
  descriptive text in the Note column moves into that row's Particulars cell (values never move), and a
  note's number and title printed as a row without figures (in the Particulars or the Note cell) becomes
  a `### <number> <title>` heading with the table split there and its header repeated. On a primary
  statement page only a title row at the top of a table is lifted (on a notes page every one is), so numbered sections inside a
  statement stay in one table.
- The system prompt is [app/ocr/OCR Instruction.md](../../app/ocr/OCR%20Instruction.md), or the
  built-in prompt if that file is missing. The prompt requires:
  - the physical column order *Particulars | Note | Year…*, never shifted
  - section letters kept in Particulars, never in Note, and joined to the description in one cell
    (never a cell of their own)
  - exactly as many cells per row as the header
  - one amount per cell
  - an inferred `**Total …**` label on unlabeled total rows
  - a label stays on the figures of its own printed line (an unlabeled sum above a labelled total
    becomes its own `Subtotal` row)
  - on notes pages, one table per note: the note number and title are a `###` heading above the
    note's own table (never a row, never in the Particulars / Note cells), and a year header printed
    once at the top is repeated in every note's table
  - every value column header carries its year (segment headers such as `Conventional Taka` get the
    year as a suffix: `Conventional Taka 2013`)
  - dates, units and currency scale kept; the unit as a plain line above the table, never as a
    separate table
  - calculation values with no year column (base amount, rate, quantity) kept inside the
    Particulars text, never in their own cell
  - (see the one-table-per-note rule above)
  - narrative paragraphs written outside tables
- The call is wrapped in `rate_limiter.execute_with_retry`.

**Saving and logging**
- `database.update_page_ocr` saves the result: it writes `pages.ocr_markdown` and mirrors it to
  `app/ocr/storage/ocr-md-files/{CODE}/{FY}/page_N.md` through `save_ocr_md_to_disk`.
  - A page that was never extracted gets extraction and verification status `PENDING`.
  - An already-extracted page gets `DEPRECATED` instead.
  - Both statuses are returned for each page.
- The run is logged with `database.record_pipeline_run(process_type="OCR")`. Its duration is the
  longest single page, because pages run in parallel.

## Stage 3: Extraction (Markdown → structured metrics)

Entry: `POST /ocr/pipeline/extract` → `router.run_extraction_pipeline(ExtractionRequest)`.

Pages are processed the same way as in Stage 2: duplicates are dropped and up to `OCR_PAGE_PARALLELISM` pages run in
parallel.

**Markdown source.** The page Markdown comes from the first of these that exists:
1. DB `pages.ocr_markdown`
2. the disk OCR `.md`
3. live OCR of the page image

**Extraction.** `extraction_service.extract_financial_data_from_markdown(md, company_name,
security_code)` does the work. To keep output tokens low, Gemini only **classifies** rows; code
reads labels, note numbers and values from the Markdown cells.
- **Table numbering.** `extraction_verifier.parse_markdown_tables` is the single source of table,
  row and column numbers.
  - A table is a pipe block with a `|---|` divider and at least one body row.
  - Every line before the divider is a header line.
  - Body rows and columns are numbered from 1.
- **Annotation.** `annotate_markdown_tables_with_coordinates` marks the tables with
  `[Table X: Row Y]` / `[Col C]` using that numbering, so the coordinates Gemini returns always
  point at the cells that code reads.
- **Model output.** `EXTRACTION_SYSTEM_INSTRUCTION` asks for a compact answer, enforced by Gemini
  structured output (`EXTRACTION_RESPONSE_SCHEMA`). The schema guarantees syntactically valid JSON
  and valid enum codes.
  - Each table is described once: statement, scope, unit, multiplier, label column, note column,
    the value columns with their years, and the section ranges. Sections are required, so every
    metric row gets a category. A range starts AT a printed section header row, also when that
    header row prints its own total ("Current Assets 30,117,446,368" opens the current-assets range, it is not a
    subtotal of the section above); a balance sheet gets one range per section (non-current assets, current assets,
    equity, non-current liabilities, current liabilities), and grand-total rows and lines printed after them stay inside a range.
    A range is named after the NEAREST printed section header above its rows, never a side heading such as "ASSETS" or
    "EQUITY AND LIABILITIES"; the model lists the printed section header rows first and draws exactly one range per header row.
    The linker's section key (`series_keys.section_key`) reads this category, so a mis-drawn range separates CA from NCA
    lines and cuts their series; if a page still looks wrong, re-extract it.
  - Metric rows are listed as `{"r":row,"c":name or "=","l":level}`, plus only the extras that
    differ from the defaults.
  - The full format is in [DATA_MODEL.md § Extracted metric JSON](DATA_MODEL.md#extracted-metric-json).
- **Decoding.** `decode_compact_extraction(payload, md)` expands this into the grouped
  `concept` / `scope` / `behavior` items that `save_extracted_metrics` stores.
  - `raw_field`, `notes` and every `numeric_value` come from the cells
    (`parse_numeric_cell_value`), so numbers are never re-typed by the model.
  - The scale stays in `multiplier`.
  - Columns with a different scope, entity or segment become separate items.
  - Every value column needs a year. The prompt and schema require `y` (the year the column's period
    ends) and `d` (the end date) when a date is printed. If `y` is still missing, the decoder takes
    the year of `d`, then the period-end year from the column's printed header
    (`_column_year`).
  - **Year guard:** the router passes the report's fiscal year. A column whose header prints no year is
    the report's own year, and a year outside [report end − 6, report end] is replaced by the year
    printed in the header (10-year summaries keep their years) or by the report year (a guessed year
    such as 2024 on a 2017-18 page becomes 2018).
  - Only the structured-output object shape is accepted: `{"t": [{..., "y": [{"c", "y", …}],
    "sec": [{"a", "b", "t", "sc"}], "r": [{"r", "c", "l", …}], "k": [...]}]}`.
  - `_parse_json` is a plain `json.loads`. An answer that is not valid JSON, or that decodes to no
    items, is logged as an error with the start of the raw answer, and the page yields no items.
- **Gemini settings.** The call goes through `gemini_json.generate_json`: JSON mode,
  `temperature=0.1`, and the thinking budget from `GEMINI_THINKING_BUDGET`. The default `auto`
  sends no thinking config; `gemini-3.5-flash-lite` doesn't think by default, rejects a budget
  of 0, and returns 0 thought tokens even with a budget of 2048 or `thinking_level` MEDIUM (tested
  2026-11-18 on a CITYBANK page, see audit AU-7), so a thinking budget gives it no room to reason.
- **Target entity context.** The prompt includes a block from `build_target_entity_context`. It
  keeps the company name out of the concept name and records subsidiaries as entity `"e"`.

**Verification.** `extraction_verifier.verify_extraction(md, items)` then compares the extracted
cells with the numeric value cells it counts in the Markdown tables. Note columns and the label
(Particulars) column are not counted, even when a label looks like a number, e.g. a section number
such as `50.1`.
- A page is `VERIFIED` when at least 85% of the cells were captured, or when no more than 4 cells
  are missing. Otherwise it is `FAILED`,
  with `failure_type` set to `ZERO_EXTRACTION` or `INCOMPLETE_EXTRACTION`. A page with no value
  cells is `NO_DATA_PAGE`.
- **Automatic retry:** if nothing was extracted from a page that has value cells
  (`ZERO_EXTRACTION`), extraction runs once more. The retry prompt includes a `corrective_hint`
  that states how many cells and rows the table has.
  - The retry result is kept if it verifies or returns more items.
  - Its tokens are always added to the page's usage.

**Saving.**
- Metrics are stored with `database.save_extracted_metrics`, then `database.relabel_company(code,
  fiscal_year)` sets metric type and unit by rule (see Stage 5, step 1). Scope and statement type
  stay as the model labelled them.

**Scope and statement labels with evidence.** The model labels every table's statement type (`s`) and
scope (`sc`) itself, and must back each with a verbatim quote of the printed text: `st` (the statement
title or note heading) and `sh` (the heading, sub-heading or column header that shows the scope). A
section range may carry its own `sc` when its heading names a different scope (e.g. "… – The City Bank
Limited" inside a consolidated report). To label consistently it gets a **report context**
(`database.get_report_label_context`, built once per extraction request): whether the report has
consolidated statements (a "Consolidated …" title or column header on any of its pages), the titles
printed on its pages, and how the previous report labelled its headings. The prompt's rules 6 and 7
define the statement types (B/I/C/E only for the primary statements; notes and reconciliations such
as "Increase/(decrease) of other liabilities" are N) and the scope (C only when the heading says
Consolidated/Group; S when it names the company itself, and always in a report without consolidated
statements). **Scope is decided per table from that table's own title**, not per page: a page can hold several
statements, so the model must not carry the scope of an earlier title over to a table under a new title (for example a
balance sheet that names only the company, printed after a "Consolidated" cash flow statement on the same page is S);
only a table with no title of its own keeps the scope of the table above, and `sh` quotes the table's own title.

Rule 8 covers rows whose printed label cannot identify the line ("Total", "Others", "In Bangladesh",
"Outside Bangladesh", "Equity holders of the bank"): the concept name is the parent heading plus that
label (`Net profit after tax attributable to total`, `Balance with other banks and financial
institutions in Bangladesh`), written the same way in every report, and never the bare parent
heading, so the same row gets the same name in every report and its series holds together.

**Calculation rows ("Less:" / "Add:").** Some statements are printed as a running calculation. An operator word printed before a
label is a calculation step, never part of the name ("Less: Operating Expenses" is "Operating Expenses"), and the value keeps its printed
sign. A "Less:" / "Add:" row that prints an amount and has indented rows below it has two roles and gets two breakdowns: it is the total
of the rows below it, and the next running total is the previous running total minus ("Less:", in `"n"`) or plus ("Add:") it.

**Breakdowns.** The model also lists, per table, which cells add up to a total (`"k"`: a total row
and its component rows, or a total column and its component columns; deducted parts in `"n"`).
`extraction_service.decode_compositions` expands them to cells, reads the printed numbers and marks
each `MATCH` or `MISMATCH`: by default the signed parts must add up within printed rounding (every printed figure is rounded to its own
precision, so a total printed to the whole Taka matches a part printed with decimals), and a
breakdown with `"o"` is checked as a ratio, percentage or product (earnings per share = profit ÷
shares); the router saves them
with `database.save_cell_compositions` (replacing the page's earlier breakdowns) and reports
`compositions_count` / `composition_mismatches` per page. The model identifies; the code only checks
the arithmetic and never adds, removes or re-asks a breakdown: there is no follow-up call. `"k"` is
required in the schema, and the prompt walks the model through finding them in the same answer:
every row it labels S / G, every row whose label prints a formula ("(C = A-B)", "(i+ii)",
"(A+B+C)") and every ratio row computed from rows of the same table (EPS "(i/ii)") needs one; a column
breakdown names columns (one object for all rows, never one per row); `"o"` is only for division,
percentage or product, never for a subtraction formula, letters and roman numerals name the rows printed with them, an unlabelled
summed row is a total, a subtotal adds only the rows back to the previous subtotal, only direct
parts are listed, `"n"` (deducted parts) is decided by a three-step procedure in the prompt: (1) add every part as printed, a bracketed
figure as a negative number, in EVERY value column, and if all columns equal the printed total `"n"` is empty; (2) only if a
column fails, a part may go in `"n"` when subtracting its printed size fixes every column and it is printed without brackets
in every column; (3) the label ("provision", "write-down", "diminution", ...) never decides (worked example: a "Total
provision" with one line bracketed in one year and positive in the other). Breakdowns that still do not add up stay
`MISMATCH`; the fix for those is a sharper instruction, not another call.

The scope / statement labels (and their quotes) are stored as the model gave them: there is no evidence check and no
retry, so no `LABEL_EVIDENCE_FAILED` flag is produced any more.

**Lines the page cannot name (rule 9).** A table that continues from the previous page has no title on this page.
The model must not borrow the next table's heading or guess: a row whose label does not identify the line (a
bare "Total", "In Bangladesh", ...) gets its printed label and `"x":1`. Such a line is stored with
`verification_flag = "FIELD_UNIDENTIFIED"` and its figures, but `csv_service` and `series_linker.build_nodes` skip it:
it is in neither the raw CSV nor the merged CSV. `GET /ocr/company/{code}/unidentified-lines` lists them, the page
result counts them (`unidentified_lines`), and a merge raises a `FIELD_UNIDENTIFIED` review item for each. A concept name that is still only a generic word (`extraction_service.GENERIC_NAMES`) cannot name its line either, so
it gets the same flag, with its heading plus the word as the suggested name. The raw CSV rows are keyed by statement,
name, unit, tag, scope and entity, plus the category when the same name is used under different headings in one report
(`csv_service.build_raw_csv_from_metrics`), so two "Total" lines of different notes are never one row. A user resolves it by naming the line
(`POST /ocr/company/{code}/line-overrides`, which stores a `line_override` by page coordinates, sets
`verification_flag = "USER_CONFIRMED"`, and is re-applied on every later extraction of the page), or by
re-extracting the page. A heading belongs to the table printed below it, never to the one above it (the prompt gives a
generic example). Every row named from a parent heading (rule 8) carries `"cf"`, an integer 0-100 for how sure the model
is that the heading is the title of the row's table; a row with `cf` below `extraction_service.MIN_NAME_CONFIDENCE`
(50) is treated like an `"x"` row, and its `flag_reason` states the confidence.
`python -m app.ocr.maintenance label-audit CODE` reports, per company, how often the same printed heading
gets a different statement type or scope in different reports (an upper bound: a line printed on both a
statement and a note legitimately has two types).
- `database.update_page_extraction` records the page status (`COMPLETED` or `FAILED`; the
  number of missing cells is in `missing_cells`) and the verification counts (`expected_cells`, `extracted_cells`,
  `missing_cells`, `completeness_pct`, `verification_status`).
- The response includes these counts per page in `verification_results`: exactly one entry per
  requested page. A page whose extraction raised is still reported, with `is_success: false`,
  `extraction_status: "FAILED"` and zero counts.
- At the end, the raw CSV for **all** of the company's years is rebuilt (Stage 4) and saved with
  `database.save_company_results`.

**Auditing later.**
- `GET /ocr/company/{code}/verify-extraction` re-runs the same audit over the stored data.
- `GET /ocr/company/{code}/extracted-cells` returns each page's cells with their coordinates, for
  highlighting on the page image.

## Stage 4: Raw multi-year CSV

`csv_service.build_raw_csv_from_metrics(metrics, company_name=None, instrument_code=None)` is a
pure function.

1. Each observation becomes `effective_val = numeric_value × multiplier`. The unit is normalized
   with `normalize_unit` (`Taka`/`Tk` → `BDT`, `ratio` → `Times`, `sqft` → `Sft.`, and so on).
2. The row key is `(statement_type, normalize_field_name(field), unit, tag, scope, entity)`.
   A row has exactly one scope, `CONSOLIDATED` or `STANDALONE`; entity is the normalized subsidiary
   name. Subsidiary rows are therefore kept apart from the parent. Values are always numbers from
   the decoder.
3. The same `(row, year)` can be reported by several annual reports, because each report also
   restates the prior year. All of these are kept in `all_observations`. The value from the latest
   report wins; within one report, a non-zero value beats a zero. `stating_years[year]` records
   which report supplied that value.
4. Sibling rows from the same report, page and table are never treated as restatements: for the
   variance check each report/page/table keeps one representative (a summary or `Total` row, else
   the first non-zero value).
5. Restatement variance: when distinct reports disagree on a value, the row gets
   `restated_variance_years` and `restatement_variances[year] = {original_value, restated_value,
   difference, percent_change, history}` (original = earliest report's figure, restated = latest;
   `percent_change` is `null` when the original is 0; `history[]` lists every consecutive pair of reports that changed the figure:
   `{from_report, to_report, from_value, to_value, difference, percent_change, within_tolerance}`).
6. Rows are sorted by `STATEMENT_TYPE_ORDER` (Balance Sheet → Income → Cash Flow → Equity →
   Highlights → Notes → Unknown), then by the normalised line name, scope and series id
   (`series_linker.series_to_rows`). Within a statement the merged rows and the CSVs are therefore in A to Z order, not in
   the order the company prints them; only the Standard tab and `standard.xlsx` (and the tree order of `merged.xlsx`) follow the
   statement's structure (REQ-9 in `docs/reports/feedbacks/feedback-1/requirements.html`).
7. Observations with no year or no value go to `remaining_data`. An observation without a year
   takes the year of its date (`extract_year_int`, which reads dates before fiscal-year ranges, so
   `2023-06-30` is 2023, and `2023-24` is 2024).

**Breakdown and total with one name.** When a table prints a breakdown (`In Bangladesh`, `Outside
Bangladesh`) and its total under the same concept name, the node takes the total row's value
(`series_linker._is_summary_row`: level `SUBTOTAL` / `GRAND_TOTAL`, or a printed "Total" label). The parts are
not part of that cell anywhere; how they add up is recorded as a breakdown.

**Every printed value column is its own cell.** Columns that print the same year must differ by scope,
entity or segment; the prompt asks the model to set `"g"` / `"e"` / `"sc"` from the header. A column whose header says
"Re-Stated" is none of these: it is the same line printed again for that year. The prompt asks the model to give it the
same `"y"`, its own printed date `"d"` and `"rs":1`; it is stored as an observation with `date` and `restated = 1` and never
becomes a segment or a row ("... (1st July Re-Stated)" no longer exists). When the model gives a same-year column neither, `extraction_service._same_year_segments` names each such column's segment after its printed
header without the year and currency words ("Conventional Taka 2013" → `Conventional`; a "Total" column
keeps none). Raw rows of a segment are `Field (Segment)`; merged rows `Field - Segment`.
A printed row is decoded once (a row number repeated in `"r"` adds nothing), and when the columns name
the segments a row-level `"g"` is ignored (it would give every column the same segment). The scale
multiplier `"m"` must be a real scale (1, 1,000, 100,000, 1,000,000, 10,000,000, 1,000,000,000);
any other number is ignored.

**Breakdowns on merged rows.** `series_linker.series_breakdowns` maps every breakdown cell to the
series holding it (`node["cells"]`), so a merged row gets `breakdown` (its components as series ids,
as the union of its parts in every report, the latest report's order first and a line only an older report printed at its printed place, each with `last_report`) and `breakdown_of` (the totals it belongs to), so a line dropped by the latest report (a nil provision) is still a component. Besides the union, `breakdown.by_year[year]` keeps the breakdown that the report stating that year printed for that year's cell (`year_breakdowns`; a stored breakdown that does not add up is listed with its status only), because every report builds its totals with its own layout: the Standard tab adds up each year from it, so a year is checked against the report that printed it, not against the layout of another report.

**Line names.** Each merged row's `field` is built by `series_linker.display_name`: the series' base
name (no `Consolidated` prefix for either scope: the table and the Excel files group by scope; only the flat merged CSV adds it),
` - <entity>` when the line belongs to an entity other than the reporting company, and ` - <segment>` for
a segment's line (`Segment liabilities - Conventional`).
So `Total assets` (either scope) and `Total assets - City Brokerage Limited`. `base_field` has no prefix or suffix, and
Returns `{headers, raw_columns, years, rows, csv_text, stating_csv_text, remaining_data}`.
`raw_columns` contains subcategorized column definitions `(report_fiscal_year, observation_year)`.
`rows` (`raw_rows`) contains `cells` mapping `col_key` to rich cell metadata including the cell's coordinates
(`report_fiscal_year`, `page_number`, `table_num`, `table_row`, `table_col`). `/results/raw` serves the rows in chunks of 100. In accordance with the raw table specification, `raw_rows`
displays all values faithfully with zero restatement dots (`restatement_dots: {}`); restatement dots are
computed and displayed exclusively on `merged_rows`.

## Stage 5: Merge (series linking)

Entry: `POST /ocr/pipeline/merge` or `POST /ocr/pipeline/merge/stream` (NDJSON progress). Both take
the company's merge lock (`router._acquire_merge_lock`; a second merge gets `409`), then call
`router._execute_merge`, which runs `series_merge.run_series_merge(code, progress_callback,
should_cancel)` and saves the result.

The merge decides which line in report N+1 continues each line of report N. A **node** is one line
item in one report; a **series** is a chain of nodes, one per report. The merged table has one row per
series.

**Step 1: deterministic relabel** (`database.relabel_company`, rules in `series_keys.py`; also run
after every page extraction). No AI is involved:
- **Value kind and unit** (`value_kind`, `unit_for_kind`): `metric_type` becomes `CURRENCY`,
  `PER_SHARE`, `COUNT`, `PERCENT` or `RATIO`, and every observation gets the matching unit `BDT`,
  `BDT/share`, `Nos.`, `%` or `Times`.
- **Scope is not rewritten.** It is labelled by the model with printed evidence at extraction time
  (Stage 3). The relabel only counts `scope_disagreements` against the report's printed statement
  titles (`resolve_report_scopes`), as a cross-check for the label audit.

**Step 2: nodes** (`series_linker.build_nodes`). One node per (report, statement, scope, entity,
segment, section, label key). When two printed lines share all of that but print different figures for
the same year, they are told apart: the node id gets the heading they sit under (`|c:<heading>`) and,
inside one heading, the printed row label (`|r:<row label>`); the node keeps `cat` (heading key) and `disc`
(that suffix). This covers the same line name under two headings (administrative vs selling expenses) and record
lists whose rows all got one concept name (asset disposals, related parties). Nodes that never collide keep
the plain id. A node that gets one year twice from the same printed row, once as printed and once from a restated column,
keeps the restated figure as the year's value and the other print as an earlier step (`node["prints"]`); this is not a
conflict. `resolve_series_values` orders a year's observations by report, then restated last, so the restated figure stays
unless a later report prints that year again, and the in-report step feeds `restate` (yellow dot). Two nodes with different `disc` are never linked (name match, unique value, Gemini question and join pass all
refuse them, through `veto()`), a unique head prefers the node under the same heading, and Gemini is shown such a line as
`<name> [heading: <heading>; row: <printed row label>]` so it can see the lines differ. Two rows with the same heading and the same printed label stay one node and raise
`INTRA_NODE_CONFLICT`, whose payload names the heading and the kept cell per year. `label_key` normalizes case, punctuation, note references, a leading
"Consolidated/Standalone" and common synonyms (sales/turnover → revenue, loan → borrowings …). It also folds the variants one line is
printed with in different reports: a leading note number ("31. Other comprehensive income"; years and "18 months" are kept),
"(s)" and plurals (`ies` → `y`; abbreviations such as `navps` are kept), British/American spelling (-ise/-ize), a repeated word
("EPS EPS") and a bracketed alternative ("Profit/(Loss)"). The word "total" stays in the key, because "X" and "X total" are often a part and
its subtotal in one table; instead `series_keys.core_key` (the key without "total") is stored on the node as `core`, and a name match
accepts two lines with the same `core` ("Deposits and other accounts" ↔ "… total") when they are never printed in the same report
(the coexistence veto), the keyword/aggregation contrast is skipped for them.
`section_key` maps the model's category to a fixed set (CA, NCA, CL, NCL, EQ, REV, COS, OPEX, FIN,
TAX, OCI, CF_OP, CF_INV, CF_FIN, CF_CASH, PER_SHARE, OTHER). Columns of 18- or 6-month periods
(transition years) go to `alt_periods` and never take part in matching. Statement × scope × entity ×
segment is a **partition**; nodes of different partitions are never linked (so single-entity years
and consolidated years stay separate series).

**Step 3: links, nearest report first** (`series_linker.link_company`), one-to-one. Reports are
counted in **company order**: a report in which this statement/scope has no lines still counts as a
step, so a line printed in 2012-13 and next in 2023-24 is 10 reports apart. Rounds R0–R3 run against
the series that ended in the immediately previous report; only lines still unmatched then widen step by
step (R4):

| Round | Links when | Rule |
|---|---|---|
| R0 | a user `FORCE_LINK` override names both reports | `USER` |
| R1 | same label key, same informative printed label, "X" = "X total" (`core`), or a user alias (`FORCE_LINK` without reports); two passes, exact same-label pairs first, so a looser match never takes a node an exact one needs. A name link whose values disagree keeps confidence 1.0 for a small change (a restatement), but gets 0.8 and a `VALUE_CONFLICT` item (`payload.rule` `NAME_EXACT`, `payload.reason`) when a common year changes by more than 50 %, a balance-sheet line flips sign, or the two headings share no word When the change is large and exactly one other unmatched line of the next report matches the old values exactly (nobody else could claim it), that line is linked instead (`VALUE_UNIQUE`, `evidence.preferred_over_name`) and the same-name pair becomes a `VALUE_CONFLICT` item with `rejected: true`. A link whose common-year values have the same size but opposite sign keeps confidence 1.0 and gets `evidence.sign_convention_changed` (years); the merged row lists them in `sign_convention_years` and `review_flags` has `SIGN_CONVENTION_CHANGED`. For a same-name `VALUE_CONFLICT` the stored breakdowns are compared: when the total is made of different parts in the two reports (or was a single line), the reason ends with "definition changed: A + B -> C" and `payload.definition_change` lists the parts | `NAME_EXACT`, `ALIAS_EQUIV` |
| R2 | the comparative-year value is identical and unique both ways, and the lines share a section or have clear name support | `VALUE_UNIQUE` |
| R2s | 2–3 lines of one report sum to one line of the other (split or merge), tested in that line's own latest year shared with a candidate part (`_line_overlap_year`), over the 12 candidates with the most similar names (`SUBSET_POOL`); parts must have a real (non-zero) value, and the lines involved are not sent to Gemini (a Gemini question carries no sum, so it could link a total to one of its parts) | no link; `SPLIT_DETECTED` / `MERGE_DETECTED` review item |
| R3 | ambiguous cases only, judged by Gemini. Pairs that only an **entity-word rule** blocks (`Entity scope contrast`, `Entity/segment breakdown`, `Distinct entity or segment breakdown`, see below) are asked too when their figures do not disagree; they are never linked by name or value | `GEMINI` |
| R4, 1 report skipped | a name match, or a unique comparative value; a name match is refused (review item `GAP_BLOCKED`) when `veto()` fires or the two headings share no informative word; a link with a >50x jump between the nearest years is kept and gets a `MAGNITUDE_JUMP` item (`evidence.magnitude_ratio`); a user-accepted link between the two reports is applied (`USER`) | `GAP_NAME`, `GAP_VALUE` |
| R4, 2+ reports skipped (up to `MERGE_MAX_GAP_REPORTS`, default 3) | a name match only (same `GAP_BLOCKED` / `MAGNITUDE_JUMP` checks); confidence 0.75 and a `LONG_GAP` review item | `LONG_GAP` |
| R5, after all reports are linked | a line where a series stops and a line where a series starts up to 8 reports later (same statement and scope, similar but not identical names, no hard veto (an entity-word rule does not block a question), no value disagreement, labels specific enough to identify a line). All such questions of the company are gathered and sent to Gemini together, in batches of 50 questions (5 candidates each); answers are cached. Accepted at confidence ≥ 0.90; 2+ skipped reports also create a `LONG_GAP` review item | `GEMINI_JOIN` |

A closer report always wins: a line continued at a smaller gap is no longer available to a wider one.
Beyond the limit nothing is linked. Every link records `gap` (reports skipped).

Values are compared per kind (`series_keys.agree`): currency exactly, within one printed unit of the
coarser scale (Taka vs Taka '000); per share and ratios within 0.011; percent within 0.011 points
(15.3 = 0.153); counts within 0.5. `compare_values` classifies a pair as `AGREE`, `DISAGREE` or `NONE`
(no overlapping year), weighting small and round numbers lower.

**Hard vetoes** (a link is never made, except by a user override; the soft entity-word vetoes named below are the one exception: they only keep the pair from being linked without Gemini): different value kind or unit; an
accounting contrast (`merge_service.has_unmergable_keyword_contrast`: current vs non-current,
operating vs investing, assets vs liabilities, gross vs net, basic vs diluted, opening vs closing,
component vs total …; a part against its own subtotal is a contrast, but `X subtotal` vs `X total` (both names carry a total word and the rest of the words is identical, e.g. one report prints the sum row of a block as "Subtotal", another as "Total") is **not**: the same line, so the normal rules decide, `merge_service._same_line_subtotal_vs_total`; the entity-word contrasts (a line with a subsidiary / segment / entity word in only one of the two names, e.g. `Investment in Associate Companies` vs `Investment in Associate and Subsidiary Companies`) are **soft vetoes**, `series_linker.SOFT_VETO_REASONS`: R1, R2 and R4 still never link such a pair, but R3 and R5 show it to Gemini, which decides); the two labels printed side by side in the
same report (so `Total Assets` never links to `Total Equity and Liabilities`); contrasting sections
(CA/NCA, CL/NCL, cash-flow sections); a parent heading about assets in one report and liabilities in
the other (`series_keys.parent_side`, used for lines with no known section, so `Branch adjustment`
under "other assets" never links to the same name under "other liabilities"; the side is also part
of the node id); a user `FORCE_SPLIT`.

**Gemini (R3)** is asked only when the evidence is ambiguous: a unique value match with weak name
support, several candidates, no overlapping year with a plausible name, or a plausible name with
different values. `merge_service.judge_link_questions` sends one question per line: the line, up to 5
candidates with their section and a value-evidence token (`=2020:1,234,567` identical, `≠2020:+3.2%`
differs, `∅` no overlap), using `LINK_SYSTEM_INSTRUCTION` and `LINK_RESPONSE_SCHEMA`
(`{"t": account-type letters, "d": [{"q": id, "c": candidate or -1, "v": "SAME" | "UNSURE" | "DIFFERENT"}]}`).
The prompt tells Gemini that an entity or ownership word alone (company, parent, group, subsidiary, associate) is a re-wording, not a different account, when the evidence token shows an identical value (`=`), that a standard statement total whose name differs only by filler words is the same line even with `∅`, and that differently NAMED subsidiaries or segments are never the same line.
Acceptance is by **verdict**, not by a number: `SAME` links (when the values of a common year disagree, a `VALUE_CONFLICT` review item only if the change is HIGH (more than 100 % of the older figure, `series_linker.high_variance`) or a balance-sheet line flips sign; a smaller difference is a restatement and makes no review item (`RESTATEMENT_CHECK` is no longer raised)),
`UNSURE` becomes an `AMBIGUOUS_LINK` review item (`payload.verdict: "UNSURE"`), `DIFFERENT` is a "no"; different account types
are never linked. A question the model leaves out of its answer is **not** a "no": it becomes an `AMBIGUOUS_LINK` item with
`verdict: "NO_ANSWER"` and the candidate labels, and is counted in `token_usage.missing_answers`.
Answers are **not cached**: Gemini is asked every time the merge runs (a pin needs a user override; a link Gemini drops later
appears as `STALE_LINK`). To keep the answers as repeatable as possible the prompt is a pure function of the data: questions
are sorted by content (partition, line, report, candidates), numbered 0..n-1 inside each batch (not by a global counter), batches
are cut on partition boundaries, and the call uses temperature 0.

**Step 4: series and values** (`build_series`, `resolve_series_values`). Links form chains; a chain
whose labels are printed side by side in some report is broken at its weakest non-user link
(`SERIES_INCONSISTENT` review item); the side-by-side test only counts reports up to the last report in which the chain still used the earlier
name, so a name that comes back later as a NEW line (a redefinition) does not cut the chain. Per year the latest report wins (`stating_years`); a difference
beyond the kind's tolerance between ANY two consecutive reports that print the year is a restatement (`restatement_variances`; a figure that
changes in a middle report and goes back still counts), a drift where every step is within the tolerance and the end values differ is a
`micro_variances` entry, and a rounding drift that comes back shows nothing. The row's `field` is the latest report's line name (a user `RENAME` wins);
`merged_from` lists every name of the series.

**Variance and review items.** A restatement is graded against the FIRST printed figure of the cell: identical, micro (rounding), low (up to 25 %), mid (up to 100 %) or high (more than 100 %, `series_keys.restatement_type`); `/results/merged` returns the grade per later printing as `flags`. During the link step a name link or Gemini link whose common-year figures differ becomes a `VALUE_CONFLICT` review item only when the difference is HIGH (`series_linker.high_variance`: more than 100 % of the older figure) or has another reason (sign flip, unrelated headings); `RESTATEMENT_CHECK` items are no longer made.

**Step 5: save** (`database.save_series_run`, one transaction). Tables `series`, `series_member`, `series_cell`
(each printed cell of each member line with its series), `series_link` (with `evidence_json`), `merge_review_item` (every `origin = 'merge'` item is deleted and rewritten; `series_merge.auto_resolve` first marks the items an active user override already decides as resolved; `FIELD_UNIDENTIFIED` and `LABEL_EVIDENCE_FAILED` are not made here, they are made at extraction by `database.replace_page_review_items`), and the links Gemini made as `merge_override` rows of `source = 'llm'` (the merge links with the user overrides only, so Gemini decides again); series ids are kept stable between merges.
Before saving, `series_merge._attach_timelines` gives each `SPLIT_DETECTED` / `MERGE_DETECTED` item a
`payload.timeline`: the resolved value per year (latest report wins) and each report's own entry for every
line involved, from the first to the last report of its series. The payload of an existing open item is
refreshed on every merge (status is kept), so older items get the timeline after the next merge.
Everything is keyed on natural keys (partition, report, label key), never `financial_metrics` ids, so
re-extraction keeps links and overrides. A Gemini link the data no longer
supports becomes a `STALE_LINK` review item; an override whose lines no longer exist an
`ORPHAN_OVERRIDE` item.

Returns `{headers, years, rows, csv_text, stating_csv_text, merge_report, conflicts,
value_mismatch_conflicts, value_mismatch_count, micro_variance_merges, total_pairs (Gemini questions),
evaluated_pairs (questions sent), matched_pairs (links), token_usage, series_count, links_by_rule,
review_open_count, relabelled}`. It is stored with `database.save_company_results` and logged as
`process_type="MERGE"`. The CSV text is escaped against formula injection (`csv_service.csv_safe_row`).

**Staleness.** OCR or extraction after a merge does not delete it, and neither does a new or deleted
user override. `/results/raw` and `/results/merged` then report `merged_stale: true` (`database.is_merge_stale`) until the merge is
re-run.

**Maintenance.** `python -m app.ocr.maintenance clear-judgements [--all]` removes the retired Gemini cache rows; `clear-merges` (backs up the DB, then deletes every
saved merge), `relabel CODE|--all`, and `merge-report CODE` (read-only: what the linker would do,
without Gemini).

### Streaming protocol (`/ocr/pipeline/merge/stream`)

The merge runs in a worker thread. `progress_callback` pushes events into an `asyncio.Queue`, which
is streamed as `application/x-ndjson`. There is one JSON object per line, each with `type` and
`percent`. Event types: `init`, `progress`, `finalize`, `complete` or `error`; the field list is
in [API.md § Merge stream](API.md#merge-stream).

Gemini is asked once per report step (and once for the join pass), and which questions a step has depends on
the answers of the steps before it, so the call count is only exact at the end.
`series_merge.plan_gemini_calls` therefore runs the linker once without Gemini to estimate the calls per
step, and `series_merge.GeminiProgress` turns the per-call events of `merge_service.judge_link_questions`
into whole-merge progress (`gemini_calls_done` / `gemini_calls_total`, `eta_sec`, `percent` by calls done),
correcting the plan whenever a step turns out different. Before this, `percent` went from 10 to 90 inside the
first call and then stayed at 90 for the rest of the merge.

- **Percent never decreases.** The stages report on their own scales, so `percent` is clamped to
  the highest value sent so far.
- **Exactly one terminal event.** The stream always ends with exactly one `complete` or `error`
  event:
  - An exception in the merge becomes `{"type": "error", "error": "Merge failed (<ExceptionType>)"}`.
    The details are only in the server log.
  - If the worker thread dies without a terminal event, the generator sends
    `Merge stopped unexpectedly`.
  - After `MERGE_STREAM_IDLE_TIMEOUT_SEC` (900 s) without events, it sends `Merge timed out` and
    cancels the merge.
- **Cancellation.** When the client disconnects (checked every second), the merge is cancelled:
  `merge_service.MergeCancelled` is raised before the next Gemini call, and no merge result is saved
  (the deterministic relabel of step 1 may already have run).
- **Lock release.** The merge lock is released when the worker finishes.
- **Cached merge.** If the company has no metrics but has a cached merge, a single `complete`
  event is returned from `company_results`.

## Reading results

- **`GET /ocr/company/{code}/results/raw` and `/results/merged`** are the two parts of the results. `/results/raw` rebuilds the raw rows from the
  metrics; `/results/merged` reads the merged rows and conflicts from `company_results` (and is the one that computes the dots), one statement per call, and only that statement's rows get dots.
  Both work as soon as metrics exist, even before any results were saved.
  - It adds **restatement dots** to every non-empty cell (`restatement_dots{year: [...]}`) of the
    merged rows, the merged rows' `constituent_details`, and both sides of each value-mismatch conflict.
    **Raw rows get none**: they carry `restatement_dots: {}`, `duplicate_in_report_years: []` and
    `restated_variance_years: []`, because the raw table shows every printed value side by side
    (`raw_columns` / `cells`). The response also returns `raw_columns`.
  - For each value, every report that disclosed it (in the row's own scope only) is grouped by
    report/page/table. Sibling rows of one table are collapsed, preferring a `Total` row. The
    original disclosure gets no dot; each LATER report that prints it gets one. Only a later report can
    restate. Another printing in the same report and statement is not a restatement (no dot): with exactly
    the same figure it is fine; with a different figure it is listed in `duplicate_in_report_years` and needs
    review (the merge raises `INTRA_NODE_CONFLICT` / `PERIOD_COLLISION`). A line repeated in a note is another
    statement and is not compared. `cell-details` uses the same
    functions (`_row_records`, `_publications`, `_restatement_dots`), so both always agree.

  | Dot | Meaning |
  |---|---|
  | green | A later report repeats the value unchanged |
  | yellow | A later report restated the value with a different figure |
  | purple | The merge accepted a small difference within tolerance (`micro_variances`). Merged rows only. |

  `restated_variance_years` is recomputed as the years with a yellow dot.
- **`GET /ocr/company/{code}/cell-details?series_id=&year=`** (or `?metric_id=&year=`)
  returns the provenance of one cell of the merged table. It lists every stating and restating observation across
  reports, with page, table and cell coordinates. The records are exactly the series' members (`series_id` is
  required; there is no lookup by field name), and field, statement, scope and category come from the series.
  - Only records of the series' scope are returned. They are
    tagged `ORIGINAL_STATED`, `RESTATED` (first printing in a later report, or a restated column of the same report, with a yellow or green
    `dot_color`), `REPEATED_IN_REPORT` (printed again in the same report with the same figure: fine, no dot) or
    `DUPLICATE_IN_REPORT` (printed again in the same report with a different figure: no dot, `needs_review`), one per publication
    (the cell itself; other printed cells are never shown as part of it).
  - Records also include the disclosures users added (`origin: USER_ADDED`) and say which one is current (`is_current`, `current_record_key`); see API.md "Disclosures added by users".
  - The response carries the cell's `restatement_dots` and its `micro_variance`, and each year in
    the timeline has its own `restatement_dots`.
- **`GET /ocr/company/{code}/cell-by-position?ref=...`** (or with individual query parameters `fiscal_year`, `page_number`, `table_num`, `table_row`, `table_col`)
  returns the extraction details of a specific markdown cell by spatial coordinates.
  - Provides a direct lookup for review pages to view extracted metrics and note links directly from table cell positions without needing the field name or series ID.
  - Accepts standard coordinate strings like `"FY 2002-03 | P-59 | T-2 | R-9 | C-3"`.
  - Returns the raw markdown cell content (`cell_text`), matched extracted metrics (`metrics`), arithmetic composition breakdown (`breakdown` and `part_of`), and attached note details (`notes`).
- **`GET /ocr/company/{code}/download/raw|merged`** returns the CSV downloads. The raw CSV is built
  from the metrics if none was saved yet.
- **`GET /ocr/company/{code}/download/raw.xlsx`** returns the raw table as a workbook (`xlsx_export.build_raw_workbook`) from the same rows as `results/raw`: a sheet per statement, the report over its columns, restated columns headed by their date.
- **`GET /ocr/company/{code}/results/standard`** returns the same standard layout as data for the frontend's Standard tab (`xlsx_export.standard_layout`: totals show the figure the reports print (a calculated figure stands in only for a year no report prints a total, `calculated_only`), an "As calculated" row (each year added up from the breakdown its stating report printed, `parts_by_year`; the union when that report has none) and a "Check (calculated - reported)" row under each (`calculated_years`, `reported_only`; a year whose report stored no breakdown has nothing to check: blank cells, and no rows at all for a total with none; a year whose stored breakdown does not add up keeps its check with a `notes` reason on the `check` row), group headers ending with a colon, the Balance Sheet's balance check, `checks: {total, off}`); the xlsx below is written from the same rows.
- **`GET /ocr/company/{code}/download/standard.xlsx`** returns every statement of the saved merged rows in the standard layout (`xlsx_export.build_standard_workbook`, a sheet per statement and scope). The other statements read as printed (lines, then their total; the result chain down one column). The Balance Sheet has banners, group headers, grey total rows with the printed figures (with an "As calculated" row of Excel formulas over their lines and a "Check (calculated - reported)" row under each), an assets-minus-equity-and-liabilities check row, per-share lines in their own section (layout in `docs/ocr/API.md`). The merged workbook below is unchanged.
- **`GET /ocr/company/{code}/download/merged.xlsx`** returns the saved merged rows as a workbook
  (`xlsx_export.build_merged_workbook`): one sheet per statement, names indented by their depth in the breakdown tree,
  numbers as numbers, restated cells highlighted. Nothing is recalculated; it reads `merged_rows_json`.
