# Todays Market Data Ingest – Documentation This document describes the **Todays Mkt Data** feature: a secure API that allows an external server to POST market data (e.g. IMDS MKISTAT-style data) every minute into the `todays_mkt_data` table. --- ## 1. Overview - **Purpose:** Ingest live/periodic market statistics from an external system into the `todays_mkt_data` table. - **Auth:** API key only (no JWT). Key is configured in the app and sent by the client on each request. - **Endpoint:** `POST /api/todays-data/ingest` - **Logging:** Failed ingest attempts are written to a dedicated log file for debugging and auditing. --- ## 2. Database ### Table: `todays_mkt_data` | Column | Type | Nullable | Description | |------------------------|----------------|----------|--------------------------------------| | ID | bigint (PK) | No | Auto-increment primary key | | INSTRUMENT_CODE | string(50) | Yes | Instrument / security code | | OPEN_PRICE | decimal(10,2) | Yes | Open price | | HIGH_PRICE | decimal(10,2) | Yes | High price | | LOW_PRICE | decimal(10,2) | Yes | Low price | | CLOSE_PRICE | decimal(10,2) | Yes | Close price | | LAST_TRADED_PRICE | decimal(10,2) | Yes | Last traded price | | TOTAL_TRADES | double | Yes | Total number of trades | | TOTAL_VOLUME | double | Yes | Total volume | | TOTAL_VALUE | decimal(20,6) | Yes | Total value | | LM_DATE_TIME | datetime | Yes | Last modified date/time from source | | CHANGE_PCT_YDAY_CLOSE | float | Yes | Change % from previous day close | | SECTOR | string(64) | Yes | Sector (e.g. from security master) | | STORED_AT | datetime | Yes | When the row was stored by this API | **Migration:** `database/migrations/2026_03_01_000001_create_todays_mkt_data_table.php` Run migrations: ```bash php artisan migrate ``` --- ## 3. Configuration ### 3.1 API key The ingest endpoint is protected by a shared API key. Configure it as follows. **Option A – Config (recommended with `config:cache`):** In `config/services.php` the following entry is already present: ```php 'todays_data_ingest' => [ 'api_key' => env('TODAYS_DATA_INGEST_API_KEY'), ], ``` **Option B – .env only:** You can use `env('TODAYS_DATA_INGEST_API_KEY')` directly in code if you do **not** run `php artisan config:cache`. **Set the key in `.env`:** ```env TODAYS_DATA_INGEST_API_KEY=your_32_character_or_longer_secret_key ``` Generate a random key (32 hex characters): ```bash php -r "echo bin2hex(random_bytes(16));" ``` Or 32 alphanumeric characters: ```bash php artisan tinker >>> Str::random(32) ``` Use HTTPS in production so the key is never sent in clear text. --- ## 4. API Endpoint ### POST `/api/todays-data/ingest` - **Method:** POST - **Content-Type:** `application/json` - **Auth:** API key via header (see below). - **Body:** JSON with an array of rows under either `data` or `MKISTAT`. #### Authentication headers The client must send the API key in **one** of these ways: 1. **Custom header (recommended):** ```http X-API-Key: ``` 2. **Bearer token:** ```http Authorization: Bearer ``` If the key is missing or wrong, the response is `401 Unauthorized` with body: ```json { "status": false, "message": "Unauthorized. Invalid or missing API key." } ``` --- ## 5. Request format ### 5.1 Using `data` (direct column names) Send an array of objects whose keys match (or are mapped to) the table columns: ```json { "data": [ { "INSTRUMENT_CODE": "AB Bank", "OPEN_PRICE": 10.50, "HIGH_PRICE": 11.00, "LOW_PRICE": 10.20, "CLOSE_PRICE": 10.80, "LAST_TRADED_PRICE": 10.75, "TOTAL_TRADES": 1500, "TOTAL_VOLUME": 50000, "TOTAL_VALUE": 537500.00, "LM_DATE_TIME": "2026-03-01 10:30:00", "CHANGE_PCT_YDAY_CLOSE": 2.5, "sector": "Bank" } ] } ``` ### 5.2 Using `MKISTAT` (IMDS-style names) You can send the same payload using IMDS MKISTAT-style keys. The server maps them to the table columns as follows: | Request key (MKISTAT) | Table column | |--------------------------------|-------------------------| | MKISTAT_INSTRUMENT_CODE | INSTRUMENT_CODE | | MKISTAT_OPEN_PRICE | OPEN_PRICE | | MKISTAT_HIGH_PRICE | HIGH_PRICE | | MKISTAT_LOW_PRICE | LOW_PRICE | | MKISTAT_CLOSE_PRICE | CLOSE_PRICE | | MKISTAT_PUB_LAST_TRADED_PRICE | LAST_TRADED_PRICE | | MKISTAT_TOTAL_TRADES | TOTAL_TRADES | | MKISTAT_TOTAL_VOLUME | TOTAL_VOLUME | | MKISTAT_TOTAL_VALUE | TOTAL_VALUE | | MKISTAT_LM_DATE_TIME | LM_DATE_TIME | | MKISTAT_CHANGE_YDAY_CLOSE | CHANGE_PCT_YDAY_CLOSE | Example: ```json { "MKISTAT": [ { "MKISTAT_INSTRUMENT_CODE": "AB Bank", "MKISTAT_OPEN_PRICE": 10.50, "MKISTAT_HIGH_PRICE": 11.00, "MKISTAT_LOW_PRICE": 10.20, "MKISTAT_CLOSE_PRICE": 10.80, "MKISTAT_PUB_LAST_TRADED_PRICE": 10.75, "MKISTAT_TOTAL_TRADES": 1500, "MKISTAT_TOTAL_VOLUME": 50000, "MKISTAT_TOTAL_VALUE": 537500.00, "MKISTAT_LM_DATE_TIME": "2026-03-01 10:30:00", "MKISTAT_CHANGE_YDAY_CLOSE": 2.5 } ] } ``` - You must send **either** `data` **or** `MKISTAT` (one of them is required). - Each row can mix direct column names and MKISTAT keys; mapping is applied where applicable. - `stored_at` is set automatically by the server to the current time and should not be sent by the client. --- ## 6. Response format ### Success (201 Created) ```json { "status": true, "message": "5 record(s) stored successfully.", "count": 5 } ``` ### Validation error (422) ```json { "status": false, "message": "Validation failed.", "errors": { ... } } ``` ### No data (400) ```json { "status": false, "message": "No data to store." } ``` ### Server error (500) ```json { "status": false, "message": "Data storing failed.", "error": "Exception message here." } ``` On 500, the failure is also logged to the dedicated log file (see below). --- ## 7. Failure logging When an exception occurs during insert (e.g. DB error), the following happens: 1. The transaction is rolled back. 2. A log entry is written to the **todaysMktDataFailed** channel. 3. The client receives the 500 response above. **Log file:** `storage/logs/todaysMktDataFailed.log` **Logged fields:** - `message` – Exception message - `trace` – Full stack trace - `payload_keys` – Top-level keys of the request (e.g. `["data"]` or `["MKISTAT"]`) - `row_count` – Number of rows in the request - `ip` – Client IP - `timestamp` – Request time (ISO 8601) **Log channel:** Defined in `config/logging.php` as `todaysMktDataFailed`. --- ## 8. Files involved | File / location | Role | |----------------|------| | `database/migrations/2026_03_01_000001_create_todays_mkt_data_table.php` | Creates `todays_mkt_data` table | | `app/Models/TodaysMktData.php` | Eloquent model for `todays_mkt_data` | | `app/Http/Controllers/TodaysMktDataController.php` | Ingest logic, validation, mapping, logging | | `app/Http/Middleware/ValidateTodaysDataApiKey.php` | API key validation (X-API-Key or Bearer) | | `config/services.php` | `todays_data_ingest.api_key` from env | | `config/logging.php` | `todaysMktDataFailed` channel → `todaysMktDataFailed.log` | | `bootstrap/app.php` | Registers middleware alias `todays.data.api.key` | | `routes/api.php` | Route `POST api/todays-data/ingest` with `api` + `todays.data.api.key` | --- ## 9. External server integration (cron / scheduler) Example: call the ingest every minute with data from your market feed. **cURL (single placeholder):** ```bash curl -X POST "https://your-fi-adm-domain.com/api/todays-data/ingest" \ -H "Content-Type: application/json" \ -H "X-API-Key: YOUR_TODAYS_DATA_INGEST_API_KEY" \ -d '{"data": [ ... ]}' ``` **cURL – Multiple rows (localhost / Postman):** Use this to test the API locally or import into Postman (Import → Raw text → paste). ```bash curl --location 'http://localhost:8000/api/todays-data/ingest' \ --header 'Content-Type: application/json' \ --header 'X-API-Key: 5cyjQrtY89767890' \ --data '{ "data": [ { "INSTRUMENT_CODE": "POSTMAN-TEST-1", "OPEN_PRICE": 10.50, "HIGH_PRICE": 11.00, "LOW_PRICE": 10.20, "CLOSE_PRICE": 10.80, "LAST_TRADED_PRICE": 10.75, "TOTAL_TRADES": 100, "TOTAL_VOLUME": 5000, "TOTAL_VALUE": 53750.00, "LM_DATE_TIME": "2026-03-01 10:30:00", "CHANGE_PCT_YDAY_CLOSE": 2.5, "sector": "Bank" }, { "INSTRUMENT_CODE": "POSTMAN-TEST-2", "OPEN_PRICE": 25.00, "HIGH_PRICE": 26.50, "LOW_PRICE": 24.20, "CLOSE_PRICE": 25.80, "LAST_TRADED_PRICE": 25.60, "TOTAL_TRADES": 250, "TOTAL_VOLUME": 12000, "TOTAL_VALUE": 307200.00, "LM_DATE_TIME": "2026-03-01 10:31:00", "CHANGE_PCT_YDAY_CLOSE": 1.8, "sector": "Insurance" }, { "INSTRUMENT_CODE": "POSTMAN-TEST-3", "OPEN_PRICE": 85.00, "HIGH_PRICE": 87.50, "LOW_PRICE": 84.00, "CLOSE_PRICE": 86.25, "LAST_TRADED_PRICE": 86.00, "TOTAL_TRADES": 180, "TOTAL_VOLUME": 8500, "TOTAL_VALUE": 731000.00, "LM_DATE_TIME": "2026-03-01 10:32:00", "CHANGE_PCT_YDAY_CLOSE": -0.5, "sector": "Pharmaceuticals" }, { "INSTRUMENT_CODE": "POSTMAN-TEST-4", "OPEN_PRICE": 42.00, "HIGH_PRICE": 43.20, "LOW_PRICE": 41.50, "CLOSE_PRICE": 42.90, "LAST_TRADED_PRICE": 42.85, "TOTAL_TRADES": 320, "TOTAL_VOLUME": 15000, "TOTAL_VALUE": 642750.00, "LM_DATE_TIME": "2026-03-01 10:33:00", "CHANGE_PCT_YDAY_CLOSE": 3.2, "sector": "Fuel & Power" }, { "INSTRUMENT_CODE": "POSTMAN-TEST-5", "OPEN_PRICE": 120.00, "HIGH_PRICE": 122.00, "LOW_PRICE": 118.50, "CLOSE_PRICE": 121.00, "LAST_TRADED_PRICE": 120.75, "TOTAL_TRADES": 95, "TOTAL_VOLUME": 4200, "TOTAL_VALUE": 507150.00, "LM_DATE_TIME": "2026-03-01 10:34:00", "CHANGE_PCT_YDAY_CLOSE": 0.8, "sector": "Food & Allied" } ] }' ``` Expected success response: `{"status":true,"message":"5 record(s) stored successfully.","count":5}`. Change the port in the URL if your app uses something other than `8000`. **PHP (Guzzle-style):** ```php $client = new \GuzzleHttp\Client(); $response = $client->post('https://your-fi-adm-domain.com/api/todays-data/ingest', [ 'headers' => [ 'Content-Type' => 'application/json', 'X-API-Key' => getenv('TODAYS_DATA_INGEST_API_KEY'), ], 'json' => ['data' => $rows], ]); ``` **Cron (every minute):** ```cron * * * * * /usr/bin/php /path/to/your/script_that_posts_todays_mkt_data.php ``` Ensure the key is stored securely (e.g. env or secrets) on the external server and that the endpoint is only called over HTTPS in production. --- ## 10. Checklist - [ ] Run `php artisan migrate` to create `todays_mkt_data`. - [ ] Set `TODAYS_DATA_INGEST_API_KEY` in `.env` (and optionally use `config:cache` with config above). - [ ] Ensure `storage/logs` is writable so `todaysMktDataFailed.log` can be created. - [ ] On the external server: send `X-API-Key` or `Authorization: Bearer ` and JSON body with `data` or `MKISTAT`. - [ ] Use HTTPS in production. - [ ] Monitor `storage/logs/todaysMktDataFailed.log` when debugging failed ingests.