# Architecture

The service hosts two modules: LVaR (`app/lvar`, `/lvar/*`) and Financial OCR (`app/ocr`, `/ocr/*`, documented in [ocr/ARCHITECTURE.md](ocr/ARCHITECTURE.md)). This doc covers the LVaR module: how it is structured, how data moves through it, and how the LVaR figure is calculated. Endpoint details are in [API.md](API.md). The module map below matches the knowledge graph in `graphify-out/` (open `graphify-out/graph.html` to explore it).

## Service layout

One FastAPI app (`app/main.py`, started with `uvicorn app.main:app`) hosts two independent
modules. Each module has its own package, router, database and URL prefix. They share only the
security dependency.

```
app/main.py  load_dotenv, CORS (CORS_ORIGINS), lifespan, GET / (open)
├── app/security.py         require_auth, admin creds      (shared)
├── app/lvar/router.py      /lvar/*  → app/lvar/...   SQLite app/lvar/storage/finStock.db
└── app/ocr/router.py       /ocr/*   → app/ocr/...    SQLite app/ocr/data/financial_ocr.db
                                                       files  app/ocr/storage/
```

The lifespan hook logs warnings about insecure configuration, creates the OCR folders and
creates the OCR schema. The LVaR schema is created on demand by
`POST /lvar/project_init/...`.

The rest of this document covers the LVaR module.

## Layers (LVaR)

```
            HTTP clients (Fintra frontend, sync.py)
                          │  X-API-Key: <API_KEY>
┌─────────────────────────▼──────────────────────────────┐
│ app/lvar/router.py  routes + pydantic bodies  (/lvar)   │
│ app/security.py     require_auth, admin creds           │
├────────────────────────────────────────────────────────┤
│ app/lvar/calculations/  lvarCal.py → orchestration, VaR │
│                         huihui.py → liquidity + LVaR    │
├────────────────────────────────────────────────────────┤
│ app/lvar/storage/   mkt_day_end.py, mkt_security.py     │
│ app/lvar/helper.py  freshness query                     │
├────────────────────────────────────────────────────────┤
│ app/lvar/database.py get_connection(), DDL, rename      │
└─────────────────────────┬──────────────────────────────┘
                          ▼
                 SQLite  (DATABASE_PATH, WAL mode)
```

| Module | Responsibility |
|---|---|
| [app/main.py](../app/main.py) | Loads `.env`, builds the FastAPI app (docs off when `ENV=production`), CORS, lifespan (security warnings, OCR dirs + schema), includes the `/lvar` and `/ocr` routers (plus the public `/ocr/health` router) |
| [app/lvar/router.py](../app/lvar/router.py) | `APIRouter(prefix="/lvar", dependencies=[Depends(require_auth)])`; validates bodies with `syncData` / `projectDataInit`, caps list sizes |
| [app/security.py](../app/security.py) | `require_auth`: checks the `X-API-Key` header against `API_KEY` (the only accepted credential; no user sessions or bearer tokens). `check_admin_credentials`: constant-time and fails closed. `project_delete_allowed`. Startup warnings for insecure config |
| [app/lvar/database.py](../app/lvar/database.py) | `get_connection()` opens SQLite with WAL, `synchronous=NORMAL` and a busy timeout. It is the most-connected node in the graph (22 edges): every storage and calculation path goes through it. Also table DDL, a table-name whitelist for f-string SQL, project init/delete, and the atomic `rename_security_code_in_all_tables` |
| [app/lvar/storage/mkt_day_end.py](../app/lvar/storage/mkt_day_end.py) | Upsert and read daily OHLCV. Price history for VaR. 30-day average traded value (ADTV). Splits `IN (...)` queries into chunks of 500 to stay under SQLite's variable limit |
| [app/lvar/storage/mkt_security.py](../app/lvar/storage/mkt_security.py) | Upsert and read security metadata. Floating shares feed the liquidity model |
| [app/lvar/helper.py](../app/lvar/helper.py) | `get_last_entry_from_sqlite`: latest dates and timestamps |
| [app/lvar/calculations/lvarCal.py](../app/lvar/calculations/lvarCal.py) | `lvrCal` entry point: `quntileCal` (VaR + ADTV) then `huihuiCal` (liquidity + LVaR + categories). Holds the policy parameters |
| [app/lvar/calculations/huihui.py](../app/lvar/calculations/huihui.py) | The maths: Hui-Heubel ratio, outlier cleaning, quadratic fit, Taylor-expanded cost, slippage, LVaR, worst-case EDR, categorisation |
| [sync.py](../sync.py) | Offline tool: copies MySQL `mkt_day_end_data` into the API in id-ordered batches, with retries and a checkpoint |

## Data model (SQLite)

**`mktDayEndData`**: one row per security per trading day. `UNIQUE(date, security_code)` is the upsert key. Holds unadjusted (`un_*`) and adjusted (`ad_*`) OHLCV, `total_value` (traded value in millions of BDT), percentage changes, and audit timestamps. The DSE broad index is stored as security `00DSEX`.

**`mktSecurityInfo`**: one row per security. `security_code` is unique; on older databases a unique index is added automatically before the upsert. Holds status flags, latest prices, volatility, beta, and the share counts in millions: outstanding shares, free-float shares and free-float %.

`/lvar/project_init` creates both tables. Writes use `INSERT ... ON CONFLICT DO UPDATE` with `executemany`, which keeps the write lock short when requests run concurrently on Cloud Run.

## Request flow: `POST /lvar/get-lvar-data`

```
syncData{mde, msi, csh}
  │
  ├─ post_mkt_day_end_data_to_sqlite(mde)   skip zero close/volume rows → upsert
  ├─ post_mkt_security_to_sqlite(msi)       fill defaults → upsert
  │
  └─ lvrCal(csh)
       ├─ quntileCal
       │    ├─ get_historical_market_price(codes + 00DSEX)   adjusted closes, all dates
       │    ├─ daily returns → per-client weighted portfolio returns → VaR_95
       │    └─ get_last_30_days_market_value_avg            ADTV (BDT)
       └─ huihuiCal
            ├─ get_outstanding_shares_security_codes        floating shares
            ├─ compute_all_hui_heubel → fit_quadratic_hh_zero_insignificant
            ├─ attach_cHH_taylor                            per-holding structural cost
            ├─ compute_LVaR_portfolio                       slippage + LVaR_95
            ├─ compute_worst_case_edr_from_existing_LVaR    stressed equity / debt
            └─ categorize_investors_by_edr                  10 risk categories
```

## LVaR methodology

### 1. Market VaR (`quntileCal`)
For each client, holding weights are `w_i = quantity_i × todaysPrice_i / total_market_value`. The portfolio's daily return is `Σ w_i · r_i,t`, using adjusted-close returns over the stored history. Then:

`VaR_95 = −quantile_5%(portfolio returns) × total_market_value`

If fewer than **250** daily return observations are available, `VaR_95 = 0`.

### 2. Hui-Heubel liquidity ratio (`hui_heubel_liquidity_ratio`)
History is split into non-overlapping 5-day windows. For each window:

`L_HH = ((P_max − P_min) / P_min) / (V / S)`

`P_max` / `P_min` are the window's adjusted high and low. `V` is the window volume. `S` is the free-float shares. The ratio measures how much the price moves per unit of turnover; higher means less liquid. Outliers beyond 6×IQR are removed per column.

### 3. Turnover-response curve (`fit_quadratic_hh_zero_insignificant`)
For each security, an OLS fit of `HH(q) = a + b·q + c·q²` over turnover `q`, with at least 8 observations. Any coefficient with p-value > 0.05 is set to zero.

### 4. Structural liquidation cost per holding (`attach_cHH_taylor`)
`f_i` is the share of free float the client holds. Starting from the security's latest observed ratio at turnover `q`, a second-order Taylor expansion estimates the ratio after selling that block:

`HH(q + f_i) ≈ HH(q) + HH'(q)·f_i + ½·HH''·f_i²`,  then  `cHH_i = HH(q + f_i) · f_i`

The result is capped at 1. Negative values are treated as missing.

### 5. Slippage and liquidity cost (`compute_slippage`, `compute_liquidity_cost`)
Square-root-law market impact for liquidating position value `N_i` over `T` days:

`Slip_i = k · (N_i / (ADTV_i · T))^γ`
`LC_i = min(1, α·Slip_i + (1 − α)·cHH_i)`, and `Liquidity_Loss_i = LC_i × market value_i`

### 6. LVaR (`compute_LVaR_portfolio`)
`LVaR_95 = VaR_95 + f · Σ Liquidity_Loss_i` when `VaR_95 > 1`; otherwise `0`.

### 7. Stress test and category (`compute_worst_case_edr_from_existing_LVaR`, `categorize_investors_by_edr`)
- `Debt` = first negative `liabilities` value, as a positive amount.
- `Current_EDR (CE) = Equity / Debt`.
- `WorstCase_EDR (SE) = (Equity − LVaR_95) / Debt`.
- `Default_Flag = (Equity − LVaR_95) ≤ 0`.

Clients are mapped into 10 categories by the SE/CE bands below. Clients with no debt get "No Debt / Insufficient Data".

| SE band | CE condition | Category |
|---|---|---|
| SE < 0 | CE < 0 / CE ≥ 0 | 1 Negative Equity (Critical) / 2 Turns Negative Under Stress |
| 0 – 0.25 | CE < 0.25 / ≥ 0.25 | 3 Forced-Sell Zone (Now) / 4 Forced-Sell If Market Falls |
| 0.25 – 0.50 | CE < 0.50 / ≥ 0.50 | 5 Margin Call Zone (Now) / 6 Margin Call If Market Falls |
| 0.50 – 0.75 | CE ≥ 0.50 | 7 Thin Cushion |
| 0.75 – 1.00 | CE < 1.00 / ≥ 1.00 | 8 Comfortable Cushion / 9 Strong |
| SE ≥ 1.00 | CE ≥ 1.00 | 10 Stress-Resilient |

The bands at SE 0.50–0.75 with CE < 0.50, and SE ≥ 1 with CE < 1, mean SE > CE. They are returned as `Exception: SE>CE (data/model flag)`.

### Policy parameters (`app/lvar/calculations/lvarCal.py`)

| Name | Value | Meaning |
|---|---|---|
| `k` | 0.015 | Slippage at 100% participation |
| `gamma` | 0.7 | Market-impact exponent |
| `T` | 5 | Liquidation horizon (days) |
| `alpha` | 0.7 | Weight of slippage vs. Hui-Heubel structural cost |
| `f` | 0.6 | Fraction of liquidity loss added to VaR |

These are module constants. Changing them changes every client's result, so treat any change as a risk-policy decision.

## Security model

- **Transport auth:** a router-level FastAPI dependency (`require_auth`) on every `/lvar` and `/ocr` route; only `/` and `/ocr/health` are open.
  - Every client (the frontend's server-side proxy, `sync.py`) sends the shared key as
    `X-API-Key: <API_KEY>`. That is the only accepted credential.
  - The backend does no user authentication: user login and sessions are handled entirely by the
    frontend. For attribution only, every `/ocr` route that changes data also takes the frontend's
    `X-Username` header (`400` without it, value trusted, not verified) and records the action in the
    `event_log` table (`GET /ocr/events`).
  - Comparisons are constant-time (`hmac.compare_digest`). Failures return
    `401 {"detail": "Not authenticated"}`.
- **Error responses:** JSON with a string `detail` everywhere (exception handlers in `app/main.py`).
- **Admin actions:** init and delete also need credentials from the environment. Delete is off unless `ALLOW_PROJECT_DELETE=true`.
- **SQL:** all values are bound parameters. Table names are only ever interpolated from a fixed whitelist. Rename input is validated against a regex.
- **Errors:** exceptions are logged server-side and clients get generic messages.
- **Docs:** hidden when `ENV=production`.
- **OCR module:** instrument codes and fiscal years are pattern-validated before they reach any file path; uploads are size-capped (`MAX_UPLOAD_MB`) and signature-checked (PDF / PNG / JPEG); stored files are served only through the authenticated `/ocr/files/{path}` route, confined to the OCR storage directory; responses carry no filesystem paths or exception text.

## Known limitations

- **Ephemeral storage on Cloud Run.** SQLite lives on the instance filesystem, so data is lost on restart unless `DATABASE_PATH` (and, for OCR, `OCR_DATABASE_PATH` / `OCR_STORAGE_DIR`) point at a mounted volume. Concurrent instances do not share data.
- **Short history gives zero risk.** A client whose holdings have fewer than 250 overlapping return days gets `VaR_95 = 0`, so `LVaR_95 = 0`. That makes SE equal CE, so the stress test has no effect for that client.
- **Missing liquidity inputs max out the cost.** A holding has no `cHH_i` when its security lacks `floating_shares_mn` or has fewer than 8 valid weekly windows. It has no `Slip_i` when there is no ADTV. In either case `alpha·Slip_i + (1 − alpha)·cHH_i` is NaN, and Python's `min(1, NaN)` returns `1`. The holding's liquidity cost then becomes **100% of its market value**, which can greatly inflate LVaR and push the client into a worse category. The skips are reported with `print()`, not the logger.
- **Unused code.** `get_today_mkt_day_end_data_from_sqlite` (it has a hard-coded date `2025-11-02`), `get_all_security_codes_last_valid_market_price`, `variance_ratio_series`, `average_monthly_volume` and `analyze_seasonality` are not called anywhere.
- **HTTP 200 on logical failures.** Callers must check `status` in the response body (see [API.md](API.md#conventions)).
