# OCR Module Index

Every Python file in [app/ocr/](../../app/ocr/), with its public symbols and its role. Line counts
are approximate. [ARCHITECTURE.md](ARCHITECTURE.md) explains how the modules fit together.

## `router.py` (2,516 lines): `/ocr` routes and orchestration

- **Router:** `router = APIRouter(prefix="/ocr", dependencies=[Depends(require_auth)])` (every
  route needs `X-API-Key`; every data-changing route also takes `user: Username`, the `X-Username`
  header, and records its action in the event log), plus `public_router = APIRouter(prefix="/ocr")` holding only the
  unauthenticated `GET /ocr/health`
- **Validation:**
  - Patterns: `INSTRUMENT_CODE_PATTERN`, `FISCAL_YEAR_PATTERN`
  - Annotated parameter types: `CodeParam`, `FyParam`, `CodeForm`, `FyForm`
  - `_validate_fiscal_year`
- **Upload helper:** `_read_upload(upload, magics, kind)` enforces the size limit (413, naming the
  limit) and checks the file signature (415).
- **PDF helper:** `_pdf_page_count(pdf_path)` counts pages with pypdfium2.
- **Page helpers:**
  - `_page_verification_fields(page)` returns a page row's extraction and verification fields
    (with defaults for pages not in the DB, plus `extraction_message`).
  - `_page_image_url(page)` returns the `/ocr/files/...` URL of a rendered page, or `None`.
  - `_list_document_pages(code, fy)` builds the page list for `get_document_pages`.
  - `_save_client_pages(...)` validates and saves uploaded page images;
    `_ocr_saved_client_pages(code, fy, saved_images)` OCRs them. Both the synchronous endpoint
    (in a threadpool) and the job endpoint use them.
  - `_run_per_page(fn, page_numbers)` drops duplicate page numbers and runs up to `config.PAGE_PARALLELISM` pages in
    parallel (`ThreadPoolExecutor`), keeping the request order.
- **Models:** `OcrRequest`, `ExtractionRequest`, `MergeRequest`. All of them validate their fields
  with patterns.
- **Merge helpers:**
  - `_acquire_merge_lock(code)` takes the per-company merge lock, or raises 409.
  - `_execute_merge(code, all_metrics, progress_callback, should_cancel)` merges, saves and
    records the MERGE run. It raises `MergeCancelled` without saving when cancelled.
  - `_merge_fy_label(years)`, `MERGE_STREAM_IDLE_TIMEOUT_SEC`.
- **Cell-details helpers:** `_complete_or_none(record, numeric_keys)`, `parse_coordinate_ref(ref_str)`
- **Other helper:** `resolve_pdf_path(code, fy)`
- **Endpoints:** see [API.md](API.md).
  - Pipeline handlers: `run_ocr_pipeline`, `run_ocr_client_pages`, `run_extraction_pipeline`,
    `run_merge_pipeline`, `run_merge_pipeline_stream`
  - Job queue: `submit_ocr_job`, `submit_ocr_client_pages_job`, `submit_extract_job` (one page each), `submit_merge_job`, `list_jobs_endpoint`, `get_job`, `get_job_events` (the merge socket's messages by polling, built by `_merge_poll_messages`; the socket `merge_socket` is registered only when `config.MERGE_SOCKET_ENABLED`), `cancel_job`
  - Pages: `get_document_pages`, `get_page_markdown`
  - Coverage: `get_coverage` (+ `_company_state`): the sector-wise matrix, built from `database.get_coverage_data` and `jobs.active_by_target`
  - `_company_results(code, include_raw, include_merged, chunk_merged, merged_statement)` builds the body of `/results/raw` and `/results/merged` (without `include_merged` the merged rows are neither loaded nor enriched); `get_company_raw_results_endpoint` cuts the raw rows into chunks of `RAW_PAGE_ROWS` = 100; `_merged_chunk` keeps one statement's merged rows (before they are enriched), conflicts and merge report and works out `statement_types` / `next_statement_type`.
  - Shared by the `/results/merged` dots and `cell-details`: `_load_cell_candidates`, `_row_records` (the row's
    own-scope records: full normalised label, series lines, or subsidiary fallback), `_publications`,
    `_representative`, `_narrow_by_category` (same name under two headings on one page), `_publication_statuses` (a restated column of the same report is RESTATED; ORIGINAL_STATED / RESTATED / REPEATED_IN_REPORT / DUPLICATE_IN_REPORT), `_restatement_dots` (one dot per
    later report), `_duplicate_in_report`
  - Provenance: `get_cell_details_endpoint` (records carry `breakdown` / `part_of`), `get_cell_by_position_endpoint` (`GET /company/{code}/cell-by-position`), `list_cell_compositions`
    (`GET /company/{code}/year/{fy}/compositions`), helper `_composition_view`
  - Verification: `verify_company_extractions_endpoint`, `get_company_extracted_cells_endpoint`,
    `get_company_note_cells_endpoint` (`GET /company/{code}/note-cells`),
    `get_company_unidentified_lines_endpoint` (`GET /company/{code}/unidentified-lines`),
    `set_company_line_override` / `list_company_line_overrides` / `delete_company_line_override` (`/company/{code}/line-overrides`)
  - File serving: `get_storage_file`

## `config.py` (63 lines): settings

- **Gemini and rate limits:** `gemini_api_key(stage)` (`OCR_API_KEY` / `EXTRACT_API_KEY` / `MERGE_API_KEY`), `GEMINI_STAGES`, `DEFAULT_MODEL`, `GEMINI_THINKING_BUDGET`
  (`None` for `auto`, the default: no thinking config is sent), `RATE_LIMIT_RPM`,
  `MIN_REQUEST_INTERVAL`
- **Uploads:** `MAX_UPLOAD_MB`, `MAX_UPLOAD_BYTES`
- **Statuses:** `VERIFICATION_STATUSES`, `EXTRACTION_STATUSES`
- **Jobs:** `JOB_TTL_SEC`, `JOB_WORKERS` (`OCR_JOB_WORKERS`, `EXTRACT_JOB_WORKERS`, `MERGE_JOB_WORKERS`), `JOB_QUEUE_MAX`
- **Paths:** `BACKEND_DIR`, `DATA_DIR`, `STORAGE_DIR`, `UPLOADS_DIR`, `PAGES_DIR`, `OCR_MD_DIR`,
  `COMPANIES_CATALOG_PATH`, `DB_PATH`
- **`init()`:** creates the folders. It is called from the app lifespan, so importing `config` has
  no side effects.

## `events.py` (~100 lines): event log (who did what)

- `require_username` (FastAPI dependency: the `X-Username` header, 1-64 printable characters, else
  `400`) and `Username = Annotated[str, Depends(require_username)]`, the parameter type of every
  data-changing route.
- `record(user, action, instrument_code=None, fiscal_year=None, target_type=None, target_id=None,
  details=None, event_id=None, keep_open=False)`: context manager. Writes a `STARTED` row, yields an
  `Event` (`.details` dict, `.target(type, id)`), then sets `SUCCEEDED`, `FAILED` (`error`,
  `status_code` in the details) or `CANCELLED` (`merge_service.MergeCancelled`); always re-raises, and
  a failure to write the log is only logged. With `event_id` it finishes the row made by `start`
  (background jobs); `keep_open=True` leaves it `STARTED` on success.
- `start(...)` writes a `STARTED` row and returns its id.
- In `router.py`: `_run_logged` (runs a pipeline function inside an event; `_result_summary` keeps its
  token and page counts), `_submit_logged_job` (event when the job is queued, finished by the job worker
  `_job_worker`, `CANCELLED` when cancelled before it started), `_merge_job` (the merge as a job), `_merge_event_details`, and `GET /ocr/events` (`list_events`).
- The pipeline bodies are `_ocr_pipeline` and `_extraction_pipeline`; `run_ocr_pipeline` and
  `run_extraction_pipeline` are the endpoints that wrap them.

## `jobs.py` (~230 lines): in-memory job queue

- One job = one page's OCR, one page's extraction, or one company's merge.
- `submit(kind, stage, fn, *args, instrument_code, fiscal_year, page_number, takes_job, on_cancel, on_queued)` queues
  `fn` on its step's queue and returns the job view. The same queued work is returned (`deduplicated`) instead of
  queued twice; more than `config.JOB_QUEUE_MAX` waiting jobs is a `429`.
- Workers: `config.JOB_WORKERS[stage]` threads per step (`ocr`, `extract`, `merge`), started on the first submit. A
  worker takes the oldest queued job of its step that is not blocked by an earlier active job on the same target
  (`_conflicts`: same page, or same company when either job is a merge).
- `subscribe(job_id, callback)` / `unsubscribe`: `callback({"event": "progress", "progress": {...}})` on every `JobContext.progress` and `callback({"event": "end"})` when the job finishes or is cancelled (used by the merge socket `router.merge_socket`).
- `JobContext.progress` also keeps every progress event on the job (the last `MAX_KEPT_EVENTS`); `events_since(job_id, after)` returns `{job, events, total, next}` for `GET /jobs/{job_id}/events`.
- (merge progress numbers: `series_merge.MergeStats` builds `merge_stats`, fed by `series_linker.link_company(on_step=...)`; see API.md `merge_stats`.)
- `takes_job=True` passes a `JobContext` (`progress(dict)`, `cancelled()`); the function raises `JobCancelled` to stop.
- `cancel(job_id)`, `get(job_id)`, `list_jobs(code, status, kind)`, `counts()`, `active_by_target()` (queued / running jobs per
  company and report, for the coverage dashboard).
- A job view: `{job_id, kind, stage, status, instrument_code, fiscal_year, page_number, position,
  waiting_for_earlier_job, progress, result, error, status_code, cancel_requested, created_at, started_at,
  finished_at, updated_at}`. Finished jobs are dropped after `config.JOB_TTL_SEC`, and jobs are lost on restart.

## `rate_limiter.py` (92 lines): Gemini quota protection

This module sits in `app/ocr/`, not in `services/`, so there is exactly one shared limiter.

- `SafeGeminiRateLimiter`: a sliding-window limiter with `wait_sync()` and `wait_async()`, guarded by
  both a thread lock and an asyncio lock.
- `rate_limiters`: one limiter per step (`"ocr"`, `"extract"`, `"merge"`), since each step has its own key.
- `execute_with_retry(fn, stage, max_retries=4, initial_backoff=12.0)`: waits for that step's limiter, then
  retries with exponential backoff on 429 errors; a used-up daily quota fails at once.

## `ocr_service.py` (490 lines): Stage 2

- `load_ocr_system_prompt()`, `OCR_SYSTEM_PROMPT`. The prompt is read from
  [app/ocr/OCR Instruction.md](../../app/ocr/OCR%20Instruction.md). Its rules cover strict column
  order, the Note column, one amount per cell, inferred `Total` labels, calculation values kept in
  Particulars, the unit as a line above the table, one table per note under a `###` note heading,
  labels kept on their own printed line, the year on every value column header, and prose outside
  tables. If the file is missing,
  a built-in prompt is used.
- `get_genai_client()`
- `render_pdf_to_images(pdf_path, code, fy, dpi=150)`: renders every page.
- `render_single_pdf_page(pdf_path, code, fy, page, dpi=150)`: renders one page when it is needed.
- `ocr_markdown.normalize_note_tables(md)` (`services/ocr_markdown.py`): note numbers only in the Note
  column; a note's number and title becomes a `###` heading above its own table (applied to every OCR result).
- `page_source_line(context)`: the `Source: <company> (<code>) | FY | Page` first line.
- `run_gemini_ocr_on_image(image_path, model_name, context=None)` returns
  `{markdown, tokens, prompt_tokens, candidates_tokens, duration_sec}`.

## `extraction_service.py` (1,091 lines): Stage 3

- `EXTRACTION_SYSTEM_INSTRUCTION`: the compact, table-oriented output format (see
  [DATA_MODEL.md](DATA_MODEL.md#extracted-metric-json)). The model only classifies rows.
- `EXTRACTION_RESPONSE_SCHEMA`: the Gemini structured-output schema for that format. It
  guarantees valid JSON and enum codes, and requires the section ranges.
- Code tables: `STATEMENT_CODES`, `SCOPE_CODES`, `LEVEL_CODES`, `TEMPORAL_CODES`, `ACTION_CODES`,
  `METRIC_TYPE_CODES`, `QUALIFIER_CODES`, `TAG_CODES`
- `build_target_entity_context(company_name, security_code)` tells the model which entity is the
  parent company, so subsidiaries get tagged as entity `"e"`.
- `annotate_markdown_tables_with_coordinates(md)` marks tables, rows and columns using
  `parse_markdown_tables` numbering.
- `build_extraction_prompt(md, company_name, security_code, corrective_hint=None, report_context=None)`;
  `build_report_context_block(report_context)` writes the REPORT CONTEXT (consolidated or not, page
  titles, previous report's labels).
- `decode_note_cells(payload, md)`: every Note-column cell (and "(note 12.a)" label) of the page, rows
  with or without figures; returned as `note_cells` by `extract_financial_data_from_markdown`.
- `decode_compositions(payload, md)`: the model's breakdowns (`"k"`) as cell compositions, each
  checked against the printed numbers (`MATCH` / `MISMATCH`; sum, ratio, percent or product via
  `COMPOSITION_OPERATIONS` / `_combine`); returned as `compositions` by
  `extract_financial_data_from_markdown`.
- `check_label_evidence(payload, md, report_context)` returns `{table: reason}` for tables whose
  `st`/`sh` quotes are not printed or don't support `s`/`sc`. It is NOT called by the pipeline any more
  (checking and retry are off); it remains as a utility.
- `decode_compact_extraction(payload, md, report_end=None)` expands the compact answer into grouped
  items. It reads labels, notes and values from the cells. `_column_year` fills a missing `y` from
  `d`, then from the column's printed header; with `report_end` it applies the year guard (no printed
  year → report year; out-of-range year → header year or report year). Only the structured-output
  object shape (`{"t": [...]}`) is accepted; `_parse_json` is a plain `json.loads`.
- `extract_financial_data_from_markdown(md, company_name, security_code, model_name,
  corrective_hint=None, fiscal_year=None, report_context=None)` returns `{items, tokens,
  prompt_tokens, candidates_tokens, thoughts_tokens, duration_sec}`. `candidates_tokens` includes thought
  tokens.
  The router passes `fiscal_year`, `report_context` and, on the automatic retry, `corrective_hint`.

## `gemini_json.py` (89 lines): shared Gemini JSON call

- `generate_json(client, model, prompt, system_instruction, response_schema=None)`:
  - JSON mode with `temperature=0.1`.
  - The optional structured-output schema.
  - The `GEMINI_THINKING_BUDGET` thinking config. If a model rejects it, the call retries once
    without it and remembers that for the model.
- `usage_tokens(response)` returns `{prompt_tokens, candidates_tokens (output + thoughts),
  thoughts_tokens, total_tokens}`.
- `response_text(response)`
- Used by extraction and merge.

## `extraction_verifier.py` (683 lines): Stage 3 verification, pure functions

This module is deterministic and uses no AI. It counts the numeric value cells in the OCR Markdown
tables and compares them with what the extraction returned.

- **Table numbering:** `parse_markdown_tables(md)` is the single source of `(table, row, col)`
  numbers. The prompt annotator and every grid lookup use it.
  - A table is a pipe block with a divider and at least one body row.
  - All lines before the divider are header lines.
- **Cell parsing:** `NIL_MARKERS`, `NOTE_COLUMN_NAMES`, `LABEL_COLUMN_NAMES`, `clean_cell_text`,
  `is_numeric_value_cell`, `parse_numeric_cell_value`, `extract_markdown_table_grid`
- **Counting:** `analyze_markdown_tables(md)`, `count_extracted_cells(items)`
  - Note columns and the label column are never counted as values. The label column is the one
    headed `Particulars`/`Description`/`Item`; without such a header, a table with a Note column has
    its labels in the first non-note column. So number-like labels (`50.1`, `2024`) are not counted.
- **`verify_extraction(md, items, tolerance_pct=85.0)`** returns:
  - `is_success`
  - `status`: `VERIFIED`, `FAILED` or `NO_DATA_PAGE`
  - `failure_type`: `None`, `ZERO_EXTRACTION` or `INCOMPLETE_EXTRACTION`
  - counts: `expected_rows`, `expected_cells`, `extracted_items`, `extracted_cells`,
    `missing_cells`, `missing_items`, `completeness_pct`, `table_count`
  - `message`, `value_verification`
- **`verify_cell_values(md, cells)`** checks each extracted value against the cell at its
  `(table_num, table_row, table_col)` in the Markdown grid.
- **DB audits:** `audit_page_in_db`, `audit_company_extractions(code, fiscal_year=None, db_path)`
  and `backfill_all_pages_verification(db_path)`. Each takes a `db_path`; the router passes
  `config.DB_PATH`.
- **CLI:** `python -m app.ocr.services.extraction_verifier` fills in the verification columns for
  every page in `config.DB_PATH` and prints a summary.

## `csv_service.py` (747 lines): Stage 4, pure functions

- **CSV injection:** `csv_safe_cell` / `csv_safe_row` prefix `'` to text cells that start with
  `= + - @`, a tab or a CR. Numbers are left alone. `csv_unescape_cell` reverses it.

- `STATEMENT_TYPE_ORDER`, `normalize_statement_type`
- `normalize_field_name`, `normalize_unit`, `format_numeric_value`
- `extract_year_int`: the year of a value, date or column header; when several are printed, the
  period-end year. Dates are read before fiscal-year ranges (`2023-06-30` → 2023, `2023-24` → 2024, a single-year report `2024` → 2024).
- **Entities:** `GENERIC_LEGAL_SUFFIXES`, `normalize_entity_name`,
  `is_distinct_subsidiary_entity(entity, parent_candidates)`. The last one is also used by
  `cell-details`.
- `build_raw_csv_from_metrics(metrics, company_name=None, instrument_code=None)` returns
  `{headers, raw_columns, years, rows, csv_text, stating_csv_text, remaining_data}`. Columns in `raw_columns`
  are subcategorized by `(report_fiscal_year, observation_year)`. Rows preserve every publication disclosure
  without overwriting, attaching the coordinates (page, table, row, column, report year) to every cell in `row["cells"]` (no `ref` string). Raw rows carry no restatement dots.
- `parse_csv_text_to_rows(csv_text)` (un-escapes cells, automatically detects single-tier or two-tier subcategorized headers)

## Stage 5: the series merge (see [PIPELINE.md § Stage 5](PIPELINE.md#stage-5-merge-series-linking))

### `series_keys.py` (321 lines): deterministic classifiers, pure functions
- **Scope (cross-check only):** `page_heading`, `page_title_scope`, `resolve_report_scopes(pages,
  model_scopes)` (statement titles → `CONSOLIDATED` / `STANDALONE`). Used for the report context and the
  label audit; scope itself is labelled by the extraction model.
- **Value kind and unit:** `value_kind(label, units, model_type, values)` → `CURRENCY` /
  `PER_SHARE` / `COUNT` / `PERCENT` / `RATIO`; `unit_for_kind` → `BDT`, `BDT/share`, `Nos.`, `%`, `Times`.
- **Keys:** `section_key(category, statement)` (fixed set), `sections_contrast`,
  `parent_side(category, section)` (`A` / `L` for lines with no known section under an assets or
  liabilities heading), `label_key(field)` (cached with `lru_cache`; folds note numbers, plurals, -ise/-ize, repeated words and a "/(Loss)" alternative; keeps "total"), `core_key(label)` (the key without "total", used by `_name_match`),
  `name_similarity`, `period_months(header)`, `column_headers(md)`, `report_end_year(fy)`.
- **Values:** `agree(a, b, kind, mult_a, mult_b)` (per-kind tolerance), `informative_weight`,
  `compare_values` → `AGREE` / `DISAGREE` / `NONE` with the overlapping years.

### `series_linker.py` (1,193 lines): the linker, pure functions (no DB, no AI calls)
- `build_nodes(metrics, pages_md, company_name, code)`: one node per report × statement × scope ×
  entity × segment × section × label key (plus `|c:<heading>` / `|r:<row label>` when two lines of a report collide; a restated column of the same row is kept as `prints` and the restated figure is the year's value),
  with year→value, provenance, `alt_periods`, `cat` / `disc` and flags. `_identity` and `_category_key` are its helpers; `_disc_clash` (used by `veto` and `_name_match`) and `_shown` (the name Gemini sees) handle lines told apart this way.
- `link_company(nodes, overrides, judge)`: rounds R0 user, R1 name (two passes, exact label first; `_value_match_instead_of_name` links the one line whose values match exactly when a same-name link jumps, `_link` adds `sign_convention_changed`; `_name_link_doubt` flags a disagreeing name link as `VALUE_CONFLICT` above `NAME_DOUBT_CHANGE` = 50 %, a balance-sheet sign flip or unrelated headings, confidence `NAME_DOUBT_CONFIDENCE` = 0.8), R2 unique value,
  R2s split/merge detection (`_detect_splits` / `_detect_merges`, `_subset_matches`, per-line year `_line_overlap_year`, pool of `SUBSET_POOL` = 12 ranked by name similarity), R3 Gemini (injected `judge`; asked every time, nothing is cached),
  R5 `_join_series` (series that stop vs series that start later, one batched Gemini pass; `_join_questions`), `_resolve_questions` (one batched Gemini call, no cache), R4 widening (`_widen_link`, with `_heading_compatible` and `_magnitude_ratio`; vetoed or heading-incompatible gap pairs become `GAP_BLOCKED` items, >50x jumps `MAGNITUDE_JUMP`; `veto` ignores an entity/segment contrast when both nodes print the same row label: 1 skipped report by name or unique value, 2+ by name as `LONG_GAP`, up to
  `max_gap` / `MAX_GAP_REPORTS`); reports counted in company order, nearest first; one-to-one; hard
  vetoes (`veto`; the entity-word reasons in `SOFT_VETO_REASONS`, tested with `_soft_veto`, are soft: such pairs go into `soft_cache` and become R3 / R5 questions for the judge, never automatic links). Returns links (each with `gap`), review items and question counts.
- `build_series(nodes, links)`: chains links into series and breaks chains whose names coexist in
  a report.
- `high_variance(ev)` / `_doubt_needs_review` (a name or Gemini link becomes a VALUE_CONFLICT review item only for a HIGH variance or another reason), `display_name(base, scope, entity, segment=None)`: the shown line name (`Consolidated ` first for a consolidated
  line, nothing for a standalone one, ` - <entity>` last for a non-reporting entity).
- `series_breakdowns(series, series_ids, compositions)`: breakdown cells lifted to series
  (`breakdown`, `breakdown_of` on merged rows): a total's components are the union of its parts across all reports (`_union_parts`, `last_report` per component).
- `year_breakdowns(series, series_ids, compositions)`: the breakdown each report printed for each total cell (SUM compositions; a `MISMATCH` one with its status only), as series; `series_to_rows` puts the entry of the cell a year's value comes from into `breakdown.by_year[year]` (the stating report's own breakdown).
- `resolve_series_values`, `series_to_rows(series, series_ids, overrides, conflicts, compositions=None)` (the merged-row
  shape), `rows_to_csv`, `value_conflicts_from_review` (`VALUE_MISMATCH` conflicts for merged rows' `value_mismatches`).

### `xlsx_export.py`: the merged and raw tables as Excel workbooks (openpyxl)
- `build_tree(rows)`: one statement and scope as `[(row, depth, has_children)]` in tree order. A line's parent is the total that
  lists it in the newest report (the component's `last_report`, `series_keys.report_end_year`; unknown is oldest), so the latest layout wins;
  among totals that list it in the same report the smallest (fewest components) wins, then the row order; children follow the printed order of those components;
  lines no total lists are top-level in the given order.
- `build_merged_workbook(rows, years=None)`: workbook bytes. Sheets `Balance Sheet`, `Income Statement`, `Cash Flow`,
  `Changes in Equity`, `Notes & Others` (only those with rows); scope group rows, indent = depth, bold totals, numeric cells,
  restated years filled with a comment. Used by `router.download_merged_xlsx`.
- `standard_layout(rows, years=None)`: the standard layout as data (`{years, sheets: [{name, statement, scope, style, rows}]}`; served by `router.get_company_standard_results`, `GET /results/standard`); `layout_checks(layout)` counts the check cells and those off; `_layout_rows(sections, years, style)` makes one sheet's rows (kinds banner / header / line / total / calculated / check / balance_check / gap; a total shows the printed figure and the "As calculated" / "Check (calculated - reported)" rows hold the sum of its parts and the difference, the calculated figure stands in only for a year no report prints (`calculated_only`); headers end with a colon; `restated` / `chosen` only for years with a figure); `_render_sheet` writes those rows into an openpyxl sheet. `router._export_rows(code)` loads the rows for both standard outputs and the merged workbook.
- `build_standard_workbook(rows, years=None)`: every statement in the standard layout (`standard_layout` written by `_render_sheet`), a sheet per statement and scope (`_sheet_title`), Notes & Others with a section per statement type (`_layout_rows(sections, years, style)`: style `balance` for the Balance Sheet, `flow` for the others (lines then their total, result chain at one level, PER SHARE DATA at the end); balance style: banners, group headers, printed totals with "As calculated" formula rows (`_formula_parts`: the total's tree children and signs, the union; each year is added up from `breakdown.by_year[year]` when it has one: `parts_by_year`, a part not on the sheet or in no series as printed; a printed year with no stored breakdown is not calculated or checked, a total with none has no calculated / check rows; the `check` row's `notes` give the reason for a year whose stored breakdown does not add up; `balance_check` skips per-share totals) and check rows, assets/equity check (`balance_check` row), per-share section; `_nest` turns `build_tree`'s flat order into nested nodes). Used by `router.download_standard_xlsx`; `build_merged_workbook` is unchanged.
- `build_raw_workbook(rows, raw_columns)`: workbook bytes of the raw table, same sheets; two header rows (report group merged over its
  columns, then year or printed date), numeric cells, frozen panes. Used by `router.download_raw_xlsx`.

### `series_merge.py` (~225 lines): orchestrator
- `run_series_merge(code, progress_callback, should_cancel, judge=None)`: relabel → nodes → links
  (Gemini via `merge_service.judge_link_questions`) → series → `database.save_series_run` → merged
  rows and CSV. Returns `{merged_res, metrics}`; called by `router._execute_merge`.
- `plan_gemini_calls(nodes, overrides)`: dry run of the linker without Gemini (on a copy), returns
  `[{round, questions, calls}]`: the estimate of the Gemini calls. `GeminiProgress(emit, plan)`: turns the per-call
  events of `judge_link_questions` into whole-merge progress (`fields()`, `round_started`, `on_event`).
- `_attach_timelines(review, series, nodes)`: after `build_series`, adds `payload["timeline"]` (every year of every
  involved line across all reports) to `SPLIT_DETECTED` / `MERGE_DETECTED` items, in place; derived from the
  series only, no extra Gemini call. `database.save_series_run` then adds `series_id` to each timeline line.
- `series_linker.review_attention(kind, payload)` / `ATTENTION` / `ATTENTION_RANK` / `LOW_PRIORITY_STATEMENTS` (an item about NOTES_AND_FOOTNOTES or CHANGES_IN_EQUITY is always LOW; HIGH / MEDIUM / LOW per review item, used by `database.list_review_items`, which adds `attention` and `attention_rank` and sorts most important first), `value_change(evidence)` (largest relative change of the disagreeing common years); `_attach_comparisons(review, nodes)`: adds `payload["comparison"]` (both lines' whole rows with their printed cells, per-year MATCH / MISMATCH / ONLY_A / ONLY_B) to every pair item (`PAIR_REVIEW_KINDS`: VALUE_CONFLICT, LONG_GAP, MAGNITUDE_JUMP, GAP_BLOCKED, STALE_LINK, AMBIGUOUS_LINK, RESTATEMENT_CHECK) and the flat `a_*` / `b_*` printed-cell fields (not VALUE_CONFLICT) to the others; `_attach_definition_changes(review, nodes, compositions)`: for a same-name `VALUE_CONFLICT`, compares the stored breakdowns of the two reports and adds `payload["definition_change"]` and a "definition changed" reason when the total is made of different parts.

### `merge_service.py` (522 lines): lexical vetoes and the Gemini link judge
- **Lexical layer:** `normalize_text`, `get_tokens`, `SYNONYM_REPLACEMENTS`, `apply_synonyms`,
  `CONTRAST_SPECS`, `has_unmergable_keyword_contrast`, `has_aggregation_level_contrast` (a subtotal against a component is a contrast;
  `X subtotal` vs `X total` is not: `_same_line_subtotal_vs_total`, `_without_total_words`),
  `extract_entity_or_segment_qualifier`, `normalize_entity_qualifier`, `extract_core_concept_head`
  (no company-specific rules), `are_candidate_pair`.
- **Gemini judge:** `get_gemini_client`, `LINK_SYSTEM_INSTRUCTION` (rules for evidence tokens, entity words, standard totals and the never-link list), `LINK_RESPONSE_SCHEMA`,
  `ACCOUNT_TYPE_CODES`, `build_link_batch_input(questions)`, `decode_link_batch_output(parsed, labels)`,
  `judge_link_questions(questions, batch_size=LINK_BATCH_SIZE (50), progress_callback, should_cancel)` (emits a `stage` event before and after every call, and on every retry), `split_link_batches` (content-sorted batches cut on partition boundaries), `VERDICT_CONFIDENCE` (SAME / UNSURE / DIFFERENT; a missing answer is `NO_ANSWER`). The call uses temperature 0 and prompt ids 0..n-1 per batch. Calls go through
  the merge step's limiter (`rate_limiters["merge"]`, key `MERGE_API_KEY`) and are retried on 429/503.
- **Cancellation:** `MergeCancelled` is raised before the next Gemini call once `should_cancel()`
  is true.

### `app/ocr/maintenance.py` (195 lines): CLI
`python -m app.ocr.maintenance clear-merges` (backs up the DB, then `database.clear_all_merges`),
`clear-judgements [--all]` (backs up the DB, then deletes the retired `pair_judgement` rows: the "no" rows, or all),
`backfill-series-cells [--code CODE]` (backs up the DB, then fills `series_cell` for merged companies from the stored metrics and the saved `series_member`, no Gemini: `database.rebuild_series_cells`),
`backfill-events --user NAME` (backs up the DB, then records the stored history as that user's events once, `database.backfill_events`),
`relabel CODE|--all` (`database.relabel_company`), `merge-report CODE ...` (read-only evaluation of
the linker; no writes, no Gemini), `label-audit CODE ...` (read-only: how often the same printed heading
gets a different statement type or scope across reports, and failed label evidence).

## `database.py` (2,348 lines): persistence

- **Setup:**
  - `get_connection` opens connections with a 60 s timeout, WAL, foreign keys on, and
    `sqlite3.Row` rows.
  - `init_db` is called from the app lifespan.
  - `populate_companies_catalog`
- **Response helpers:** `iso_utc` (timestamps with `Z`), `extraction_message`
- **Companies:** `get_company_info`, `get_company_status`, `get_all_companies`,
  `get_companies_basic`, `upsert_company` (a call without a name or sector keeps the stored ones), `get_coverage_data`
  (the aggregate queries behind `GET /coverage`)
- **Documents and pages:** `upsert_document` (`new_file=True` clears the checked mark), `get_document_by_year`,
  `set_document_verification` / `verification_fields` (the checked mark), `_add_missing_columns` (older databases), `upsert_page`,
  `get_pages_for_year`
  - `update_page_ocr` returns the page's new `extraction_status` and `verification_status`.
  - `update_page_extraction` also stores the verification counts.
- **OCR Markdown copies on disk:** `save_ocr_md_to_disk`, `sync_all_ocr_md_to_disk`
- **Metrics:**
  - `save_extracted_metrics` accepts grouped or flat items.
  - `get_all_metrics_for_company`
  - `get_extracted_cells_for_company(code, fiscal_year=None)`
  - `get_extracted_cell_by_position(code, fiscal_year, page_number, table_num, table_row, table_col)`
    (loads raw markdown cell text, matched metrics, arithmetic breakdown, and attached notes for an exact coordinate)
  - `save_page_note_cells(code, fy, page, note_cells)`, `get_note_cells_for_company(code, fiscal_year=None)`
    (every printed cell holding a note reference, by page, with the row's metrics),
    `get_note_cells_for_metrics(metric_ids)` (`notes` and `note_cell` per metric, for `cell-details`)
- **Results:** `save_company_results`, `get_company_results`, `is_merge_stale` (also true after an
  override change), `clear_all_merges`
- **Breakdowns:** `save_cell_compositions(code, fy, page, compositions)` (replaces the page),
  `get_cell_compositions(code, fiscal_year=None, page_number=None)` (cells resolved to metrics),
  `get_cell_composition_links(code, fy, page, table, row, col)` (`breakdown` / `part_of` of one cell)
- **Labels:** `get_pages_markdown`, `relabel_company(code, fiscal_year=None)` (metric type and unit;
  counts scope disagreements only), `get_report_label_context(code, fiscal_year)` (has consolidated
  statements, page titles, previous report's labels) for the extraction prompt
- **Series merge state:** `save_series_run` (series, members, links, review items (all `origin = 'merge'` items replaced), LLM overrides; stable series
  ids), `load_active_links`, `list_series`,
  `get_series_evidence`, `get_series_formation` (members, metrics and values of a series, from `series_member` + `series_cell`), `enter_cell_disclosure` (add or replace) / `delete_cell_disclosure` / `list_cell_disclosures` / `set_cell_current` / `reset_cell_current` / `list_cell_current` / `get_record_value` (user disclosures and the current choice of a cell, stored by series identity), `ai_record_key`, `set_series_active` / `get_inactive_series_ids` (rows switched off for the Excel downloads), `get_series_id_for_metric`, `series_cell_map`, `list_review_items`, `get_review_item`, `set_review_item_status`,
  `add_merge_override`, `load_merge_overrides(code, active_only, source)`, `get_merge_override`, `make_llm_override_permanent`, `deactivate_merge_override`,
  `replace_page_review_items` (the extraction review items of a page, called by `save_extracted_metrics` and `set_line_override`), `rebuild_extraction_review_items`
- **Event log:** `add_event`, `finish_event` (merges the details), `set_event_job`, `list_events(username, instrument_code,
  action, status, since, until, limit, offset)` (`{total, events}`, newest first), `backfill_events(username)`
- **Usage:** `record_pipeline_run`, `get_pipeline_usage_report` (MERGE runs split across years via
  `_merge_run_years`)
