# OCR Module Architecture

## Purpose

The module digitizes the financial statements in annual-report PDFs of companies listed on the
Dhaka Stock Exchange. An *instrument* is a company's trading code, such as `CITYBANK` or
`SQURPHARMA`. For each company the module builds one table: each row is an accounting line item
and each column is a year. This works even though the same item is worded differently across a
decade of reports.

Related docs: [PIPELINE.md](PIPELINE.md) · [DATA_MODEL.md](DATA_MODEL.md) · [API.md](API.md) ·
[MODULE_INDEX.md](MODULE_INDEX.md) · the shared service: [../ARCHITECTURE.md](../ARCHITECTURE.md)

## System context

```
┌────────────────────┐  HTTP/NDJSON + API key  ┌──────────────────────────────────┐  HTTPS  ┌───────────────┐
│ OCR front-end      │ ──────────────────────▶ │ app/main.py (FastAPI)            │ ──────▶ │ Google Gemini │
│ localhost:3030     │ ◀────────────────────── │  ├─ /lvar → app/lvar/router.py   │ ◀────── │ (google-genai)│
│ (separate project) │  /ocr/files/... images  │  └─ /ocr  → app/ocr/router.py    │         └───────────────┘
└────────────────────┘                         └────────────────┬─────────────────┘
                                         ┌──────────────────────┴──────────────────┐
                                         ▼                                         ▼
                         SQLite: app/ocr/data/financial_ocr.db      Filesystem: app/ocr/storage/
                                                                    uploads/ pages/ ocr-md-files/
```

The front-end is a separate project. `app/main.py` configures CORS from `CORS_ORIGINS` (default
`http://localhost:3030,http://127.0.0.1:3030`) and allows the `X-API-Key`, `X-Username` and
`Content-Type` headers (and the `DELETE` method).

## Layers

| Layer | Modules | Responsibility |
|---|---|---|
| App composition | `app/main.py`, `app/security.py` (shared with LVaR) | Loads `.env`, sets up CORS, runs the lifespan hooks (`config.init()`, `database.init_db()`), JSON error handlers, provides `require_auth` |
| API / orchestration | `router.py`, `jobs.py` | `/ocr` endpoints, request models, input validation, upload limits, pipeline orchestration (up to `OCR_PAGE_PARALLELISM` pages in parallel), verification retry, restatement dots, NDJSON streaming with a per-company merge lock and cancellation, background jobs, authenticated file serving |
| AI services | `ocr_service.py`, `extraction_service.py`, `merge_service.py` | Gemini calls: image → Markdown, Markdown → JSON metrics, and the judge for ambiguous cross-report links (plus the lexical accounting vetoes). The OCR prompt is `OCR Instruction.md`. |
| Verification | `extraction_verifier.py` | Deterministic check of extracted cells against the numeric cells in the OCR Markdown, and per-company audits |
| Transformation | `csv_service.py` | Pivots per-page metrics into multi-year rows per scope and entity, tracks stating and restated years |
| Excel export | `xlsx_export.py` | The raw table and the merged rows as workbooks: a sheet per statement, the line names indented as the results tree (from each row's breakdown), restated cells marked |
| Series merge | `series_keys.py`, `series_linker.py`, `series_merge.py` | Deterministic scope / value kind / unit, line items linked report to report into series (name, comparative-year value, Gemini only when ambiguous), merged rows |
| Event log | `events.py` | `X-Username` dependency and the `record` context manager: who did what (`event_log`, `GET /ocr/events`) |
| Persistence | `database.py`, `maintenance.py` | SQLite schema (no migrations; `init_db` creates it), CRUD, series-merge state, usage reporting, copying OCR Markdown to disk; maintenance CLI |
| Cross-cutting | `config.py`, `rate_limiter.py` | Paths, environment variables, one shared Gemini rate limiter with retry and backoff |

## Module dependency graph

All imports inside the package are relative (`from . import database`, `from .config import ...`).

```
app.main ──┬─▶ app.lvar.router
           ├─▶ app.ocr.router ──┬─▶ app.security.require_auth
           │                    ├─▶ config, jobs ──▶ config
           │                    ├─▶ database ──▶ config, extraction_verifier, series_keys (both lazy)
           │                    ├─▶ ocr_service ──▶ config, rate_limiter.execute_with_retry
           │                    ├─▶ extraction_service ──▶ config, rate_limiter.execute_with_retry, csv_service
           │                    ├─▶ extraction_verifier    (pure; its DB audits take a db_path)
           │                    ├─▶ csv_service            (pure: no I/O, no project imports)
           │                    ├─▶ series_keys            (pure) ──▶ csv_service, extraction_verifier
           │                    └─▶ series_merge ──▶ database, series_linker (pure) ──▶ series_keys, csv_service,
           │                                         merge_service ──▶ config, rate_limiter.rate_limiters["merge"]
           ├─▶ app.ocr.config    (init() runs in the lifespan)
           └─▶ app.ocr.database  (init_db() runs in the lifespan)

```

These parts are pure functions, so they are the easiest to unit test:
- `csv_service`
- `extraction_verifier`
- `series_keys` and `series_linker` (Gemini is an injected `judge`)
- the lexical half of `merge_service`: `normalize_text`, `apply_synonyms`,
  `has_unmergable_keyword_contrast`, `are_candidate_pair`

`tests/test_ocr.py` checks the verification and restatement-dot behavior on seeded metrics, and
`tests/test_series_merge.py` the series merge on synthetic reports, without calling Gemini.
`tests/test_raw_table_subcategorization.py` checks the raw table columns and cells, and
`tests/test_cell_by_position.py` the `cell-by-position` lookup and `parse_coordinate_ref`.

## End-to-end data flow

1. **Ingest.** `POST /ocr/upload` saves the PDF and renders every page immediately.
   `POST /ocr/register-document` only saves the PDF. Either way the PDF is stored as
   `app/ocr/storage/uploads/{CODE}_{FY}.pdf`.
2. **Page rendering.** `ocr_service.render_pdf_to_images` (every page) or `render_single_pdf_page`
   (one page) uses pypdfium2 at 150 DPI and writes `app/ocr/storage/pages/{CODE}/{FY}/page_N.png`.
   Alternatively, the client can render the pages itself and send them to
   `/ocr/pipeline/ocr-client-pages`.
3. **OCR.** `ocr_service.run_gemini_ocr_on_image` produces Markdown tables with the columns
   *Particulars | Note | Year…*. The result is stored in `pages.ocr_markdown` and also written to
   `app/ocr/storage/ocr-md-files/{CODE}/{FY}/page_N.md`.
4. **Extraction.** Gemini only classifies table rows, in a compact format that keeps output tokens
   low. `extraction_service.decode_compact_extraction` then reads labels, notes and values from
   the Markdown cells, and builds metrics grouped as `concept` / `scope` / `behavior`, plus
   `raw_field`, `notes`, and `data[]` with exact table coordinates.
   - `extraction_verifier.verify_extraction` checks that the cells were captured. If none were,
     extraction is retried once with a corrective hint.
   - The result is stored in the `pages` verification columns.
   - `database.save_extracted_metrics` normalizes the metrics into the four `financial_metric*`
     tables; `database.relabel_company` then sets metric type and unit by rule. Scope and statement
     type are the model's labels, stored as given.
5. **Raw CSV.** `csv_service.build_raw_csv_from_metrics` organizes all observations into hierarchical
   subcategorized columns by `(Report Fiscal Year, Observation Year)` (`raw_columns`). Rows are keyed by
   (`statement_type`, `field`, `unit`, `tag`, scope, entity). No publication disclosures are overwritten:
   comparative years appear distinctly under each report. Every cell carries its coordinates (`report_fiscal_year`, `page_number`, `table_num`, `table_row`, `table_col`).
6. **Merge (series linking).** `series_merge.run_series_merge` links each report's line items to the
   next report's, one-to-one, into series: by name, then by the comparative-year value each report
   reprints (so a rename such as "Accounts Payable" → "Creditors and Other Payables" is linked), and
   only ambiguous cases go to Gemini. Hard accounting contrasts and lines printed side by side in one
   report are never linked. User overrides pin decisions; Gemini answers are never cached, so Gemini decides again on every merge.
7. **Results.** `database.save_company_results` stores the raw, stating, merged and merged-stating
   CSVs, the merge report and the conflicts in `company_results`; the series, links and review queue
   are in the series tables. They are served by
   `/ocr/company/{code}/results/raw` and `/results/merged`, the CSV downloads and `/cell-details`. Restatement dots
   (yellow, green, purple; see [PIPELINE.md § Reading results](PIPELINE.md#reading-results)) belong
   strictly to the Merged Table (`merged_rows`), while the Raw Table (`raw_rows`) displays pure extracted
   values across all publications with zero dots.

Every OCR, extraction and merge run is logged in `pipeline_runs` with its token counts and
duration. `database.get_pipeline_usage_report` aggregates these logs.

## Key design decisions

- **Gemini free-tier safety.** `SafeGeminiRateLimiter` allows at most `RATE_LIMIT_RPM` calls
  (default 12) in any 60-second window, with at least `MIN_REQUEST_INTERVAL` seconds (default 4.5)
  between calls.
  - `execute_with_retry` makes up to 4 attempts. On 429 or quota errors it backs off exponentially
    (12 s × 1.8ⁿ); on other errors it retries after 4 s.
  - Each step has its own API key and its own limiter (`rate_limiter.rate_limiters["ocr" | "extract" | "merge"]`),
    so each key's per-minute and daily quota is spent separately.
  - A used-up DAILY quota (429 "...PerDay...") is not retried: the call fails at once.
- **Always Gemini.** OCR and extraction always call Gemini.
- **Parallel pages.** OCR and extraction process up to `OCR_PAGE_PARALLELISM` (default 10) pages at once in a thread pool, after
  dropping duplicate page numbers. The merge asks Gemini once per report transition, and only
  about ambiguous links. The shared
  rate limiter still caps the total request rate, and SQLite's WAL mode plus a 60 s busy timeout
  absorb the concurrent writes.
- **Lazy rendering.** When no PNG exists yet, OCR and extraction render just the page they need
  (`render_single_pdf_page`). Large PDFs never have to be fully rasterized.
- **Where extraction gets page Markdown, in order:** the database (`pages.ocr_markdown`), then
  `ocr-md-files/` on disk, then a live OCR call.
- **Verified extraction.** Every extracted page is checked by deterministic cell counting
  (`extraction_verifier`). A page that yields nothing is retried once with a corrective prompt.
  Pages that are still incomplete are stored as `FAILED` (with `missing_cells`; responses add
  `extraction_message: "Missing N cells"`), not hidden.
- **Concept, entity, scope and statement type are labelled at extraction time, with evidence; metric
  type and unit are deterministic.**
  - The extraction prompt asks Gemini for a clean accounting `concept.name` with no company name.
  - Subsidiaries are named in `scope.entity`.
  - Scope and statement type are not patched by code. The model gets the report context (consolidated or
    not, page titles, last year's labels) and must quote the printed heading behind each label; the quotes
    are kept as audit evidence but not checked. `label-audit` measures consistency across years.
  - Single-entity and consolidated years are separate series.
  - Raw rows, merges, restatement dots and `cell-details` all respect scope and entity.
- **Conservative, evidence-based merging.**
  - Hard accounting contrasts are never linked: current vs non-current, operating vs investing,
    gross vs net, component vs total, and so on; nor two lines printed in the same report.
  - Links are one-to-one, nearest report first (reports counted in company order), so chains can't
    snowball. A line that skipped reports is linked only after closer candidates are used, by name,
    up to `MERGE_MAX_GAP_REPORTS` (default 3) skipped reports; 2+ skipped reports are flagged `LONG_GAP`.
  - A value link needs an identical comparative-year figure that is unique both ways; a Gemini link
    needs 0.80–0.95 confidence depending on the evidence. Everything else goes to the review queue.
  - Every decision is stored with its evidence and keyed on natural keys, so user overrides
    survive re-extraction.
- **One merge per company, saved only when complete.**
  - Both merge endpoints take an in-memory per-company lock (`_acquire_merge_lock`). A second
    merge gets `409`. That is enough for the single uvicorn process in `Procfile`.
  - The stream cancels its merge when the client disconnects (`MergeCancelled`, checked before
    each Gemini batch). A cancelled merge saves nothing.
  - An OCR or extraction run, or an override change, after the last merge makes `/results/raw` and `/results/merged` report
    `merged_stale: true`. The old merge is kept until it is re-run.
- **Long requests off the event loop.** The blocking Gemini work in `ocr-client-pages` runs in a
  worker thread. `jobs.py` is an in-memory job queue (one page's OCR or extraction, or one company's merge, per job; a queue per
  Gemini step, run in order, a page's extraction after its OCR, a merge after its company's page jobs) that clients poll,
  for proxies that cut long requests.
- **No schema migrations.** `database.init_db` creates the schema (`CREATE TABLE IF NOT EXISTS`).
  To start over, delete the database file (see [DATA_MODEL.md](DATA_MODEL.md)).

## Security

The module uses the same approach as LVaR (see
[../ARCHITECTURE.md § Security model](../ARCHITECTURE.md#security-model)):

- **Auth.** `router` is declared as
  `APIRouter(prefix="/ocr", dependencies=[Depends(require_auth)])`.
  - Every route on it requires the header `X-API-Key: <API_KEY>`, compared in constant time. No
    other credential (bearer token, session, `?token=`) is accepted; user login is handled by the
    frontend, not here.
  - Routes that change data also depend on `events.require_username` (the `X-Username` header, `400`
    without it) and record the action in the `event_log` table through `events.record`. The value is
    trusted, not verified: it is attribution, not authentication.
  - `GET /ocr/health` lives on a separate `public_router` (same prefix, no dependency) so load
    balancers and uptime checks can call it without the key. `app/main.py` includes both routers.
  - See [API.md § Conventions](API.md#conventions).
- **Path safety.** Instrument codes and fiscal years must match `INSTRUMENT_CODE_PATTERN` and
  `FISCAL_YEAR_PATTERN`. Neither pattern allows `.` or `/`.
  - The check applies to path parameters, form fields, request bodies and every value in
    `fiscal_years_json`.
  - It runs before any of these values is used to build a file path (`uploads/{CODE}_{FY}.pdf`,
    `pages/{CODE}/{FY}/`).
- **File serving.** `/ocr/files/{path}` resolves the requested path and only serves regular files
  inside `config.STORAGE_DIR`.
- **Uploads.** `_read_upload` reads at most `MAX_UPLOAD_MB` + 1 byte and returns `413` if the file is
  larger. It also checks the file signature and returns `415` if it is wrong. Every file in a
  request is checked before any of them is written.
- **Bounded inputs.** A page-number list can hold at most 2000 entries, and the `cell-details` query
  strings have length limits. The optional `fiscal_year` on `verify-extraction` and
  `extracted-cells` must match `FISCAL_YEAR_PATTERN`.
- **No leaks.** `/ocr/health` returns only `{"status": ...}` (no model, key or path details), and
  the Markdown-listing endpoints return no absolute paths. Exceptions
  are logged with `logger.exception`, and clients only see generic text.

## Configuration

`load_dotenv()` in `app/main.py` reads `.env` from the project root once. Real environment
variables take precedence. The template is [../../.env.example](../../.env.example).

| Variable | Default | Used by |
|---|---|---|
| `API_KEY` | (empty = no auth) | `require_auth` (shared with LVaR): the expected `X-API-Key` value |
| `JOB_TTL_SEC` | 3600 | `jobs.py`: how long finished jobs are kept |
| `OCR_JOB_WORKERS` / `EXTRACT_JOB_WORKERS` | `OCR_PAGE_PARALLELISM` | `jobs.py`: page jobs of that step in flight at once (the rate limiter still caps the calls per minute; 1 = strictly one by one) |
| `MERGE_JOB_WORKERS` | 1 | `jobs.py`: merges run at once |
| `JOB_QUEUE_MAX` | 5000 | `jobs.py`: waiting jobs before submits get `429` |
| `MERGE_SOCKET_ENABLED` | off | `config.MERGE_SOCKET_ENABLED`: `1` registers `WS /ocr/ws/merge/{code}` (`router.merge_socket`); off, the merge is polled with `GET /ocr/jobs/{job_id}/events` |
| `MERGE_MAX_GAP_REPORTS` | 3 | `series_linker.MAX_GAP_REPORTS`: reports a line may skip and still be linked (2+ → `LONG_GAP`) |
| `OCR_API_KEY` / `EXTRACT_API_KEY` / `MERGE_API_KEY` | (empty) | Gemini key of each step (`config.gemini_api_key(stage)`): OCR, extraction, merge |
| `GEMINI_MODEL` | `gemini-3.5-flash-lite` | `DEFAULT_MODEL` |
| `GEMINI_THINKING_BUDGET` | `auto` (no thinking config sent) | Thinking budget for extraction and merge calls (`gemini_json.generate_json`). A number sets `thinking_budget`. `gemini-3.5-flash-lite` rejects `0`, doesn't think by default and ignores a positive budget or `thinking_level` (0 thought tokens in tests). A rejected config is retried without it. Thought tokens are counted in `candidates_tokens`. |
| `RATE_LIMIT_RPM` | 12 | `SafeGeminiRateLimiter` |
| `MIN_REQUEST_INTERVAL` | 4.5 | `SafeGeminiRateLimiter` |
| `OCR_PAGE_PARALLELISM` | 10 | `config.PAGE_PARALLELISM`: pages of one request processed at once. One call takes 20-60 s, so about `RATE_LIMIT_RPM` × seconds ÷ 60 calls must be in flight to reach the limit; the limiter still caps starts per minute |
| `MAX_UPLOAD_MB` | 50 | `config.MAX_UPLOAD_BYTES` |
| `OCR_DATA_DIR` | `app/ocr/data` | `config.DATA_DIR` (holds `financial_ocr.db` and `companies.json`) |
| `OCR_DATABASE_PATH` | `<OCR_DATA_DIR>/financial_ocr.db` | `config.DB_PATH` |
| `OCR_STORAGE_DIR` | `app/ocr/storage` | `config.STORAGE_DIR` (`UPLOADS_DIR`, `PAGES_DIR` and `OCR_MD_DIR` are inside it) |
| `CORS_ORIGINS` | `http://localhost:3030,http://127.0.0.1:3030` | `app/main.py` |

Path constants:
- `BACKEND_DIR` is `app/ocr/`.
- `DATA_DIR` is `app/ocr/data/`. It holds `companies.json` and `financial_ocr.db`.
- `COMPANIES_CATALOG_PATH` is `app/ocr/data/companies.json`.

Importing `config` has no side effects. The folders are created by `config.init()` when the app
starts.

## Known issues and technical debt

| # | Location | Issue |
|---|---|---|
| 1 | `upload_documents` | A file that isn't listed in `fiscal_years_json` silently gets FY `2024-25`. |
| 2 | `database.get_connection` | `with get_connection() as conn` commits but doesn't close the connection. It sets WAL mode and a 60 s timeout. |
| 3 | `database.py` | Uses `datetime.utcnow()`, which is deprecated from Python 3.12. |
| 4 | Cloud Run | The SQLite database and `storage/` are lost on restart unless `OCR_DATABASE_PATH` and `OCR_STORAGE_DIR` point at a mounted volume. |
| 5 | Lint | Several OCR modules don't follow the project's ruff rules. They trigger style warnings (old typing syntax, broad `except`) and have unused imports. |
