feat(datasets): periods (start_date/end_date), dg_id column, date filters; rename observation dates

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-05 13:16:07 +05:00
parent 37d4f3519b
commit 922812b765
10 changed files with 729 additions and 181 deletions

View File

@@ -72,10 +72,10 @@ the KATO column and the year columns:
`parse_error`).
3. The client polls `GET /datasets/{id}` until `awaiting_mapping`, then submits
`POST /datasets/{id}/mapping` with the chosen `kato_column` and a
`year_columns` map (each `{column, date}`). Validated against the detected
columns; `status` → `extracting`.
`year_columns` map (each `{column, start_date}`, optionally `end_date` for
a period). Validated against the detected columns; `status` → `extracting`.
4. A second worker job **unpivots** the attribute table into long-format
`dataset_observations` — one row per `(kato_code, date)` with a numeric
`dataset_observations` — one row per `(kato_code, start_date)` with a numeric
`value` (or `value_text` for non-numeric cells); `status` → `ready`. Read
them via `GET /datasets/{id}/observations` (paginated, optional
`?kato_code=`).
@@ -84,12 +84,25 @@ the KATO column and the year columns:
curl -X POST https://dssgis.dwh.kz/datasets/<id>/mapping -H 'Content-Type: application/json' -d '{
"kato_column": "като",
"year_columns": [
{"column": "F_2023", "date": "2023-01-01"},
{"column": "D_2025", "date": "2025-01-01"}
{"column": "F_2023", "start_date": "2023-01-01"},
{"column": "D_2025", "start_date": "2025-01-01"}
]
}'
```
### Dataset periods
A dataset carries an optional period — `start_date` and `end_date`
(`YYYY-MM-DD`, both nullable, `end_date` inclusive and never before
`start_date`) — plus an optional `dg_id` linking it to a *dataset group*: the
"одноимённые" datasets that describe the same thing for different periods. All
three are accepted on `POST /datasets` (form fields), `PUT /datasets/{id}` and
`PATCH /datasets/{id}` (an explicit `null`, or omission on PUT, clears them).
`GET /datasets` filters on them with `?dg_id=`, and with the inclusive bounds
`?start_date_from=`, `?start_date_to=`, `?end_date_from=` and `?end_date_to=`
(each `YYYY-MM-DD`).
## Getting started
```sh
@@ -117,13 +130,13 @@ server runs it is served at `/openapi.yaml`, with an interactive **Redoc** UI at
| GET | `/categories/{id}` | get |
| PUT | `/categories/{id}` | update |
| DELETE | `/categories/{id}` | delete |
| GET | `/datasets` | paginated list of summaries (`?page=`, `?page_size=`, `?category_id=`) |
| POST | `/datasets` | upload (multipart: `file`, `file_type`, `category_id`, `code`, `name`, `description?`, `unit?`, `meta?` (JSON), `automated?` (bool)) |
| GET | `/datasets` | paginated list of summaries (`?page=`, `?page_size=`, `?category_id=`, `?dg_id=`, `?start_date_from=`, `?start_date_to=`, `?end_date_from=`, `?end_date_to=`) |
| POST | `/datasets` | upload (multipart: `file`, `file_type`, `category_id`, `code`, `name`, `description?`, `unit?`, `meta?` (JSON), `automated?` (bool), `dg_id?`, `start_date?`, `end_date?`) |
| GET | `/datasets/{id}` | full dataset (geometry as GeoJSON, `bbox` for rasters) |
| GET | `/datasets/{id}.geojson` | GeoJSON `FeatureCollection`; plain `vector` returns its geometry as a single feature with the extracted attribute table as top-level properties; `vector_with_kato` always ignores its own geometry and joins the `districts` table by KATO, mapping observations onto each polygon |
| GET | `/datasets/{id}/status` | processing status; long-polls with `?current=<status>` (holds up to `?wait=` secs, default 25, max 60) |
| GET | `/datasets/{id}/download` | download the stored file |
| POST | `/datasets/{id}/mapping` | set KATO column + year→date map (vector_with_kato) |
| POST | `/datasets/{id}/mapping` | set KATO column + year→period map (vector_with_kato) |
| GET | `/datasets/{id}/observations` | paginated unpivoted values (`?kato_code=`, `?page=`, `?page_size=`) |
| DELETE | `/datasets/{id}` | delete (row + object) |