# API Reference

Base URL: `http://localhost:8000` locally, or your Cloud Run service URL.

## Conventions

- **Prefixes:** LVaR endpoints are under `/lvar`, OCR endpoints under `/ocr` (OCR reference: [ocr/API.md](ocr/API.md)).
- **Auth:** every endpoint except `GET /` and `GET /ocr/health` needs the header
  `X-API-Key: <API_KEY>`. No other form is accepted (no `Authorization: Bearer`, no `?token=`, no
  user sessions). A bad or missing key returns HTTP `401 {"detail": "Not authenticated"}`. If
  `API_KEY` is not configured on the server, no key is required.
- **Username (OCR only):** every `/ocr` route that changes data also needs `X-Username: <logged-in user>`
  (1 to 64 printable characters) and returns `400` without it. The value is trusted, not verified, and is
  recorded in the event log (`GET /ocr/events`). Details: [ocr/API.md](ocr/API.md#event-log). LVaR routes do not use it.
- **Errors:** every error response is JSON with a string `detail`. `422` validation errors also
  carry the original FastAPI list in `errors`, and unhandled errors return
  `500 {"detail": "Internal server error"}`.
- **Status in the body:** most handlers return **HTTP 200** even when the operation fails. Read the outcome from the `status` / `message` fields in the JSON body. This applies to the LVaR endpoints; OCR endpoints use real HTTP status codes. For LVaR, the real HTTP status is non-200 only for `401` (auth), `422` (request body or path validation by FastAPI) and an unhandled `500`.
- **Row format:** `GET` endpoints that return table rows give each row as a JSON **array in column order** (see [Table columns](#table-columns)), not as an object.
- **Body size:** each list in a request body is capped at `MAX_REQUEST_LIST_ITEMS` (default 50 000). Larger lists get `422`.

## Endpoint summary

| Method | Path | Purpose |
|---|---|---|
| GET | `/` | Health check (no auth) |
| GET | `/lvar/get-last-entry` | Freshness timestamps of stored market data |
| POST | `/lvar/get-lvar-data` | Store market data **and** calculate LVaR for client holdings |
| GET | `/lvar/get-mkt-day-end-data/filter={filter_type}/{filter_value}` | Read daily price rows |
| GET | `/lvar/get-mkt-security-data` | Read all security metadata |
| POST | `/lvar/project_init/{project_name}/{password}` | Create the database and tables (admin) |
| POST | `/lvar/project_delete/{project_name}/{password}` | Drop tables and delete the database (admin, opt-in) |
| POST | `/lvar/project_data_init_mde` | Bulk upsert daily price rows |
| POST | `/lvar/project_data_init_msi` | Bulk upsert security metadata |
| PUT | `/lvar/rename-security-code/{old_code}/{new_code}` | Rename a ticker in every table, atomically |

### OCR module (`/ocr`)

Full reference: [ocr/API.md](ocr/API.md). Same auth as above (except the public `/ocr/health`); routes that change data also need the `X-Username` header. Instrument codes and fiscal years are
pattern-validated, and uploads are limited in size and file type.

| Method | Path | Purpose |
|---|---|---|
| GET | `/ocr/health` | Public health check (no auth): `{"status": "pass" \| "warn" \| "fail"}` as `application/health+json` |
| GET | `/ocr/coverage` | Sector-wise coverage matrix: companies x years, report stage, checked mark, running jobs |
| GET | `/ocr/companies` · `/ocr/companies/catalog` | Companies in the DB (`?fields=basic` for a light list) · raw `companies.json` |
| GET | `/ocr/company/{code}/status` | Analyzed years, documents, page progress |
| POST | `/ocr/upload` | Upload annual-report PDFs (multipart) and render their pages |
| POST | `/ocr/register-document` | Register a PDF without rendering it |
| POST | `/ocr/company/{code}/year/{fy}/verify` | Mark a report's fiscal year as checked by a person, or clear the mark (recorded as an event) |
| GET | `/ocr/company/{code}/year/{fy}/pages` · `/pdf` | Page list (`?include_markdown=false` to leave out the Markdown) · source PDF |
| GET | `/ocr/company/{code}/year/{fy}/page/{page_number}/markdown` | OCR Markdown of one page |
| POST | `/ocr/pipeline/ocr-client-pages` | OCR on page images rendered by the browser |
| POST | `/ocr/pipeline/ocr` · `/ocr/pipeline/extract` | OCR pages · extract financial metrics, with per-page verification (pages run in parallel, `OCR_PAGE_PARALLELISM`, default 10) |
| POST | `/ocr/pipeline/merge` · `/ocr/pipeline/merge/stream` | Link every line item across reports into series (by name and comparative-year value; Gemini only for ambiguous links). The stream variant returns NDJSON progress and always ends with `complete` or `error`; `409` while a merge is running |
| WS | `/ocr/ws/merge/{code}?username=&api_key=` | **Off** unless `MERGE_SOCKET_ENABLED=1` (the hosted server does not pass sockets): runs the merge and pushes its status over a socket |
| POST | `/ocr/jobs/ocr` · `/ocr/jobs/ocr-client-pages` · `/ocr/jobs/extract` · `/ocr/jobs/merge` | Queue ONE page's OCR / extraction, or one company's merge (`202 {job_id, ...}`); run in order within Gemini's rate limits |
| GET · DELETE | `/ocr/jobs` · `/ocr/jobs/{job_id}` | The queue, one job's status / progress / result, cancel a job |
| GET | `/ocr/jobs/{job_id}/events?after=` | The merge's messages by polling (the same as the merge socket's): `{job_id, status, done, next, messages}` |
| GET | `/ocr/company/{code}/results/raw` (slim rows; `?view=full` for the former) · `/results/merged` (slim rows with `flags`; `?view=full` for the former) · `/results/merged/row/{series_id}` · `/results/merged/conflicts` · `/results/merged/report` · `/cell-details?series_id=&year=` or `?metric_id=&year=` (merged table cell history, by series) | Consolidated results as two parts, the raw table (in chunks of 100 rows, `?statement_type=&offset=`) and the merged table (one statement per call, `?statement_type=`; `raw_columns`, `raw_rows` with cell coordinates, `merged_rows` with restatement dots) · provenance of one cell (`?series_id=` for exactly one series) |
| GET | `/ocr/company/{code}/year/{fy}/compositions` | Breakdowns: which printed cells add up to a total (`?page_number=`), each checked MATCH / MISMATCH |
| GET | `/ocr/company/{code}/series` · `/series/{series_id}/evidence` · `/series-formation?series_id=` | Merged series and their members · the links of one series with their evidence · the members, metrics (metric_id) and values that formed it |
| POST · PUT · DELETE | `/ocr/company/{code}/enter-disclosure` (POST, DELETE) · `/cell-current` | Enter the user's disclosure of a merged cell (adds it, or replaces the cell's earlier one) or delete it; choose, or reset, which disclosure of a cell is the current one (an overwrite; `cell-details` shows `origin` and `is_current`; needs `X-Username`) |
| PUT | `/ocr/company/{code}/series/{series_id}/active` · `/series-active` | Switch one merged row, or many, on or off; inactive rows are left out of the Excel downloads (`merged_rows[].active`, needs `X-Username`) |
| GET/POST | `/ocr/company/{code}/merge/review` · POST `/merge/review/{item_id}/resolve` | Merge review queue · accept / reject (saved as an override) / dismiss an item |
| GET/POST/DELETE | `/ocr/company/{code}/merge/overrides` · POST `/merge/override` · DELETE `/merge/override/{override_id}` | User merge decisions: force link, force split, rename, detach. `source` is `user` or `llm` (a Gemini link); `POST /merge/override/{override_id}/decide` makes an LLM override permanent or breaks it |
| GET | `/ocr/company/{code}/verify-extraction` · `/extracted-cells` | Extraction completeness audit · extracted cells with table coordinates |
| GET | `/ocr/company/{code}/download/raw` · `/download/merged` · `/download/raw.xlsx` · `/download/merged.xlsx` · `/results/standard` (the Standard tab as data: the standard layout with calculated totals and checks) · `/download/standard.xlsx` (every statement in the standard layout, totals as formulas with checks) (inactive rows left out) | Subcategorized two-tier Raw CSV download · Merged CSV download · Raw and merged Excel workbooks (a sheet per statement; the merged one with the indented tree) |
| GET | `/ocr/company/{code}/usage-report` · `/year/{fy}/usage-report` | Token and time usage |
| GET/POST | `/ocr/sync-ocr-md` · GET `/ocr/ocr-md-files` | Copy OCR Markdown to disk · list those files |
| GET | `/ocr/events` | Event log: who did what, newest first, with filters (needs only the API key) |
| GET | `/ocr/files/{path}` | Stored page images, PDFs and Markdown (needs `X-API-Key` like every other route) |

---

# LVaR endpoints

## GET `/`

No authentication. Returns `{"message": "Hello World"}`.

## GET `/lvar/get-last-entry`

Shows how fresh the stored data is.

```json
{
  "status": 200,
  "message": "Last entry fetched successfully",
  "market_day_end_data": {"mded_date": "2025-11-02", "mded_create_at": "...", "mded_update_at": "..."},
  "market_security_info": {"msi_create_at": "...", "msi_update_at": "..."}
}
```

## POST `/lvar/get-lvar-data`

The main endpoint. It does three things, in order:
1. Upserts `mde` into `mktDayEndData`.
2. Upserts `msi` into `mktSecurityInfo`.
3. Runs the LVaR pipeline on `csh` (client holdings).

See [ARCHITECTURE.md](ARCHITECTURE.md#lvar-methodology) for the maths.

**Request body**

```json
{
  "status": 200,
  "message": "any string",
  "mde": [ /* day-end rows, same shape as /lvar/project_data_init_mde items */ ],
  "msi": [ /* security rows, same shape as /lvar/project_data_init_msi items */ ],
  "csh": [
    {
      "fintr_customer_id": "C001",
      "full_name": "Jane Doe",
      "security_code": "BATBC",
      "quantity": 1000,
      "avgcost": 510.0,
      "todaysPrice": 525.5,
      "total_cost": 510000,
      "total_market_value": 1250000,
      "equity": 400000,
      "liabilities": -850000
    }
  ]
}
```

`csh` has one row per (client, security). `total_market_value`, `equity` and `liabilities` describe the **whole client portfolio**, so repeat the same values on each of that client's rows. Debt is taken from the first **negative** `liabilities` value. A client with no negative value counts as having no debt.

`mde` and `msi` can be empty lists if the database is already up to date.

**Response (success)**

```json
{
  "status": 200,
  "message": "Data stored and LVR calculated successfully",
  "data": {
    "status": 200,
    "message": "Success",
    "data": [
      {
        "Investor Code": "C001",
        "Investor_TotalMarketValue": 525500.0,
        "Investor_LVaR_95": 61234.5,
        "Investor_VaR_95": 48000.0,
        "Investor_Equity": 400000.0,
        "Investor_Debt": 850000.0,
        "WorstCase_MarketValue": 464265.5,
        "WorstCase_Equity": 338765.5,
        "WorstCase_EDR": 0.3985,
        "Current_EDR": 0.4706,
        "Default_Flag": false,
        "LVaR_to_VaR_Ratio": 1.2757,
        "EDR_Category": "Category 5: 0.25<=SE<0.50 & CE<0.50 - Margin Call Zone (Now) - ..."
      }
    ]
  }
}
```

**Failure bodies** (still HTTP 200, except where marked):

| Body | Meaning |
|---|---|
| `{"status": 500, "message": "Data stored failed", "mde": {...}, "msi": {...}}` | An upsert failed; the nested objects show which one |
| `{"status": 500, "message": "LVR calculation failed"}` (HTTP 500) | The pipeline raised an exception, e.g. no price history for the requested securities |

## GET `/lvar/get-mkt-day-end-data/filter={filter_type}/{filter_value}`

| `filter_type` | `filter_value` | Result |
|---|---|---|
| `date` | `YYYY-MM-DD` | All rows for that date |
| `security_code` | ticker, e.g. `BATBC` | Full history of that security, newest first |
| `limit` | integer | Latest N rows. Clamped to 1–5000; invalid or ≤0 falls back to 300 |
| anything else | – | Latest 300 rows |

```json
{"status": 200, "message": "Market day-end data retrieved successfully", "records_count": 2, "data": [[1, "2025-11-02", "BATBC", ...], ...]}
```

## GET `/lvar/get-mkt-security-data`

Returns every row of `mktSecurityInfo`. The shape is the same as above, with `message: "Market security data retrieved successfully"`.

## POST `/lvar/project_init/{project_name}/{password}`

Creates the SQLite file and both tables if they don't exist. Running it again is safe. `project_name` / `password` must match `PROJECT_ADMIN_NAME` / `PROJECT_ADMIN_PASSWORD`.

| Body | Meaning |
|---|---|
| `{"message": "Project initialized successfully", "project_name": "..."}` | OK |
| `{"message": "Project initialization failed", "project_name": "..."}` | DDL failed |
| `{"message": "Project not found or invalid credentials"}` | Wrong credentials, or admin env vars not set |

## POST `/lvar/project_delete/{project_name}/{password}`

**Destructive:** drops both tables and deletes the database file. Requires valid admin credentials **and** `ALLOW_PROJECT_DELETE=true` on the server. Otherwise it returns `{"message": "Project deletion failed", ...}` and changes nothing.

## POST `/lvar/project_data_init_mde`

Bulk upsert of daily prices. The key is `(date, security_code)`: existing rows are updated. Rows where `un_close`, `un_volume`, `ad_close` or `ad_volume` is `0` are silently skipped.

```json
{
  "status": 200,
  "message": "any string",
  "data": [
    {
      "date": "2025-11-02", "security_code": "BATBC",
      "un_open": 520, "un_high": 530, "un_low": 515, "un_close": 525.5, "un_volume": 120000, "total_value": 63.1,
      "ad_open": 520, "ad_high": 530, "ad_low": 515, "ad_close": 525.5, "ad_volume": 120000,
      "adst_ratio_value": 1.0, "pct_change": 0.85, "pct_change_with_dsex": 0.4,
      "create_at": "2025-11-02 16:00:00", "update_at": "2025-11-02 16:00:00"
    }
  ]
}
```

All 18 keys are required on every row. `un_*` are unadjusted prices and `ad_*` are adjusted ones. `total_value` is the traded value in **millions of BDT**. The DSE index is stored as the security code `00DSEX`.

Response: `{"message": "Project data initialized successfully", "data_count": <rows stored>}`. On failure the message is `"Project data initialization failed"`, and `data_count` holds the error object.

## POST `/lvar/project_data_init_msi`

Bulk upsert of security metadata, keyed on `security_code`. The body has the same envelope as above. Each item:

```json
{
  "security_code": "BATBC", "company_name": "British American Tobacco Bangladesh", "sector": "Food & Allied",
  "is_active": "Y", "marginable": "Y",
  "un_close": 525.5, "ad_close": 525.5, "volitility": 0.21, "betas": 0.9,
  "outstanding_shares_mn": 540.0, "floating_percent": 27.1, "floating_shares_mn": 146.3,
  "create_at": "2025-11-02 16:00:00", "update_at": "2025-11-02 16:00:00"
}
```

Only `security_code` is required. Missing text fields default to `""`, `is_active` to `"Y"` and `marginable` to `"N"`. Missing numeric fields become `0`. `volatility` is accepted as an alternative spelling of `volitility`. **`floating_shares_mn` is needed for LVaR:** securities without it are skipped in the liquidity calculation.

Response: `{"message": "Project security data initialized successfully", "data_count": N}`.

## PUT `/lvar/rename-security-code/{old_code}/{new_code}`

Renames a ticker in `mktSecurityInfo` and `mktDayEndData` in a single transaction. Both codes must match `^[A-Za-z0-9_.\-]{1,64}$` and must be different from each other.

| Body `status` | Meaning |
|---|---|
| 200 | Renamed; includes `mktSecurityInfo_rows_updated` and `mktDayEndData_rows_updated` |
| 400 | Invalid or identical codes |
| 404 | `old_code` not found in either table |
| 409 | `new_code` already exists; the rename would break a UNIQUE key |
| 500 | Unexpected failure (details are logged server-side only) |

---

## Table columns

Rows from the `GET` data endpoints come back as arrays in this order.

**mktDayEndData:** `id, date, security_code, un_open, un_high, un_low, un_close, un_volume, total_value, ad_open, ad_high, ad_low, ad_close, ad_volume, adst_ratio_value, pct_change, pct_change_with_dsex, create_at, update_at`

**mktSecurityInfo:** `id, is_active, security_code, company_name, sector, marginable, un_close, ad_close, volitility, betas, outstanding_shares_mn, floating_percent, floating_shares_mn, create_at, update_at`
