# Mutual Fund — NAV / profile scrape

**Controller:** `App\Http\Controllers\MfScrapeController`  
**Queue job:** `App\Jobs\ExecuteScrapeRun`  
**NAV table:** existing `mf_nav_history` (upsert on `fund_id` + `as_of_date`)  
**Returns:** existing `App\Services\MfReturnSeriesService` (equations 1–9)  
**Middleware:** `api` only (no JWT)

HTTP requests **never scrape inline**. They insert `scrape_logs` rows and dispatch queue jobs. Each successful NAV or profile job writes data, then calls `MfReturnSeriesService::rebuildQuietly()`.

---

## New vs reused tables

| Table | Action |
|-------|--------|
| `mf_amcs` | Reused. `slug` selects the scraper (`ekush`, `edge`, `shanta`, `vipb`, `cwt`, `ucb`, `idlc`). |
| `mf_funds` | Reused. `fund_url` is the scrape entry; live NAV snapshot columns are updated. |
| `mf_nav_history` | Reused. Scraped N(t)/P(t) upserted; K(t) still comes from the return-series rebuild. |
| `mf_dividend_history` | Reused. Profile scrapers upsert by `fund_id` + `financial_year`. |
| `mf_unit_adjustments` | Reused (s(t) in Eq. 5). Not written by scrape. |
| `mf_fund_holdings` / `mf_sector_allocations` | Reused when a profile page has holdings / mix. |
| `jobs` / `job_batches` / `failed_jobs` | Reused (already in this project). |
| `scrape_logs` | **New.** Run ledger (`queued` → `running` → `success`/`failed`). Uses `created_at`. |
| `mf_fund_published_returns` | **New.** Optional AMC-published % for cross-check only. Uses `create_at`/`update_at`. |

Existing `mf_*` columns were **not** altered.

---

## Column mapping (SQL pack → this project)

| Scrape / PDF field | Stored in |
|--------------------|-----------|
| N(t) | `mf_nav_history.nav` |
| P(t) | `mf_nav_history.composite_nav` (defaults to `nav` when the AMC has no composite) |
| K(t) | `mf_nav_history.adjusted_nav` (written by `MfReturnSeriesService`, not by the HTTP fetch) |
| buy / sell | `mf_nav_history.buy_price` / `sell_price` |
| as-of | `mf_nav_history.as_of_date` |
| source | `mf_nav_history.source_type` (`SCRAPE` / `API` / `CMS`) |
| live snapshot | `mf_funds.current_nav`, `current_buy_price`, `current_sell_price`, `composite_nav`, `latest_nav_date` |
| p / D / effective date | `mf_dividend_history.dividend_percentage` / `dividend_per_unit` / `effective_date` |
| fund page | `mf_funds.fund_url` |
| AMC scraper key | `mf_amcs.slug` |

Fund identity is the existing ticker in `mf_funds.slug` (e.g. `EFUF`). Page slugs are read from `fund_url` (last path segment) when an AMC site needs them.

---

## Endpoints

| Action | Method | Path | Route name |
|--------|--------|------|------------|
| Queue NAV + profiles | POST | `/api/mf-scrape/mf-scrape-all` | `mf-scrape-all` |
| Queue NAV only | POST | `/api/mf-scrape/mf-scrape-nav` | `mf-scrape-nav` |
| Queue profiles + dividends | POST | `/api/mf-scrape/mf-scrape-profiles` | `mf-scrape-profiles` |
| Force rebuild K(t) | POST | `/api/mf-scrape/mf-scrape-rebuild-returns` | `mf-scrape-rebuild-returns` |
| List runs | GET | `/api/mf-scrape/mf-scrape-index` | `mf-scrape-index` |
| Show one run | GET | `/api/mf-scrape/mf-scrape-show/{id}` | `mf-scrape-show/{id}` |
| Batch status | GET | `/api/mf-scrape/mf-scrape-batch/{batchId}` | `mf-scrape-batch/{batchId}` |
| Retry a run | POST | `/api/mf-scrape/mf-scrape-retry/{id}` | `mf-scrape-retry/{id}` |

Update is **POST**, not PUT.

---

## Queue body

```json
{
  "scope": "all",
  "include_profiles": true,
  "fund_ids": [1, 2],
  "amc_id": 1,
  "users_id": 1
}
```

| Field | Rules |
|-------|--------|
| `scope` | `all` (default), `stale` (latest NAV older than 2 days or null), `selected` (requires `fund_ids`) |
| `include_profiles` | scrape-all only; default `true` |
| `fund_ids` | required when `scope=selected` |
| `amc_id` | optional extra filter |
| `users_id` | stored on `scrape_logs.user_id` |

Only **active** funds (`mf_funds.status = A`) whose AMC `slug` is in `config/amc_scrapers.php` (NAV) or `config/amc_profile_scrapers.php` (profiles) are queued.

**Success (200):**

```json
{
  "status": true,
  "message": "Queued; workers will scrape then rebuild K(t).",
  "data": {
    "batch_id": "uuid…",
    "profile_batch_id": "uuid…",
    "nav_queued": 12,
    "nav_skipped": 1,
    "profile_queued": 12,
    "profile_skipped": 0
  }
}
```

`skipped` means that fund already has a queued/running run of the same type (last 30 minutes).

---

## How to run a scrape

1. Migrate the two new tables (`php artisan migrate` or the SQL files under `database/sql/`).
2. Confirm each AMC has the matching `mf_amcs.slug` and each fund has `fund_url`.
3. `QUEUE_CONNECTION=database` (already the project default). Prefer `DB_QUEUE_RETRY_AFTER=700` because each job may run up to 600s.
4. Start a worker:

```bash
php artisan queue:work --timeout=600
```

5. Trigger from API (`POST /api/mf-scrape/mf-scrape-all`) **or** artisan:

```bash
php artisan nav:dispatch
php artisan nav:scrape --fund=EFUF
php artisan nav:scrape --amc=ekush
php artisan nav:scrape --all
php artisan funds:scrape-profile EFUF
php artisan funds:scrape-profile --amc=edge
php artisan returns:rebuild
php artisan returns:rebuild EDGEAMCGF
```

`nav:scrape` and `funds:scrape-profile` run **inline** (no queue). `nav:dispatch` and the HTTP endpoints queue `ExecuteScrapeRun`.

Daily schedule (Asia/Dhaka 18:30): `php artisan nav:dispatch`.

Progress: `storage/logs/mfNavScrape.log` and `storage/logs/mfReturnSeries.log`.

---

## Related docs

- Frontend: `MF_Nav_Scrape_Frontend.md`
- Return series math: `MF_Return_Series.md`
- cURL: `curl/mutual_fund_api_curl.md` (NAV scrape section)
- Postman: `Mutual_Fund.postman_collection.json` folder **NAV Scrape**
