document CHPA warehouse refactor rules

2026-08-20 19:17:19 +08:00
parent b0d5f812ad
commit 8c3f45f178
2 changed files with 241 additions and 0 deletions
@@ -0,0 +1,146 @@
+# AGENTS.md
+
+## Repository purpose
+
+This repository contains the maintainable Databricks SQL replacement for the legacy CHPA warehouse jobs.
+
+- Treat the workspace-level `CHPA/` directory as read-only legacy input.
+- Put all rewritten code, documentation, and validation under `RE/`.
+- Preserve business behavior first. Separate logic refactoring, physical renaming, and business-rule changes.
+- Keep each persisted target in one clearly owned script whenever practical.
+
+## Warehouse layers
+
+The warehouse has exactly three persistent layers and data flows in one direction:
+
+```text
+DWD -> DWS -> DM
+```
+
+### DWD
+
+- Cleaned atomic details and foundational master data.
+- Retains source traceability and stable source grain.
+- Does not contain report-oriented hierarchy flattening or subject metrics.
+
+### DWS
+
+- Reusable dimensions, hierarchy-wide tables, and reusable facts built from DWD.
+- Standardizes shared codes, joins, and business definitions.
+- A job that reads DWD and produces a reusable derived table writes to DWS.
+
+### DM
+
+- Subject-area datasets, metrics, and report-facing outputs built from DWS.
+- Must not become an upstream dependency of DWD or DWS.
+
+DWD must not read DWS/DM. DWS must not read DM. DWS/DM derived results must not be written back to DWD, and DM derived results must not be written back to DWS.
+
+## Table naming
+
+Use the schema that matches the target layer and one of these fixed table prefixes:
+
+| Layer | Object type | Required form |
+| --- | --- | --- |
+| DWS | Dimension/master data | `dws.dws_ext_td_<business_entity>` |
+| DWS | Fact data | `dws.dws_ext_tf_<business_entity>` |
+| DM | Dimension/master data | `dm.dm_ext_td_<business_entity>` |
+| DM | Fact data | `dm.dm_ext_tf_<business_entity>` |
+
+Naming rules:
+
+- Use lowercase snake_case table names.
+- `td` means dimension/master data; `tf` means fact data.
+- Include the source/domain at the start of the business entity when it avoids ambiguity, for example `ims_atc_hierarchy`.
+- Do not write a newly derived DWS table with a `dwd_` name or into the `dwd` schema.
+- Keep legacy output column names during a logic-only migration. Rename columns only through an explicit coordinated contract change.
+
+Current hierarchy targets:
+
+- `dws.dws_ext_td_ims_atc_hierarchy`
+- `dws.dws_ext_td_ims_nfc_hierarchy`
+
+## File layout and order
+
+Use:
+
+```text
+sql/<domain>/<stage>/<sequence>_<target_table>.sql
+validation/<domain>/<stage>/<validation_name>.sql
+```
+
+Rules:
+
+- DWD jobs belong in `01_dwd`, DWS jobs in `02_dws`, and DM jobs in `03_dm`.
+- Use two-digit sequence numbers to make execution order explicit.
+- Use lowercase snake_case paths with no spaces.
+- Keep Databricks notebook markers and split executable statements with `-- COMMAND ----------`.
+- Split legacy multi-target scripts by target or responsibility when doing so does not alter transactional behavior.
+
+## SQL contracts
+
+Every persisted build script must include a concise header with:
+
+- purpose;
+- source tables;
+- target table;
+- output grain;
+- write mode;
+- legacy table replaced;
+- known consumers or migration dependency.
+
+SQL rules:
+
+- Use uppercase SQL keywords and descriptive lowercase snake_case CTE aliases.
+- Use four-space indentation and one selected column per line.
+- Use explicit target columns and explicit final projection columns.
+- Never use `SELECT *` in a persisted write.
+- Qualify columns whenever multiple relations are in scope.
+- Replace post-write null cleanup with projection expressions such as `COALESCE` when semantics are unchanged.
+- Replace update-driven staging with ordered CTEs only after proving the result is equivalent.
+- Keep one-time target DDL in a commented, separate Databricks cell above the write.
+- Prefer `CREATE TABLE IF NOT EXISTS <new_table> LIKE <verified_legacy_table>` when the legacy schema is the authoritative type contract.
+- Do not guess source field types when no DDL or `DESCRIBE` output is available.
+
+## Validation
+
+Before migrating downstream consumers:
+
+- compare old and new row counts on the same source snapshot;
+- run `EXCEPT ALL` in both directions with a fixed column order;
+- profile duplicate business keys or paths;
+- count null required keys;
+- count unmatched hierarchy/join levels;
+- make compatibility checks fail the notebook with `raise_error()` when any required check fails.
+
+Keep old physical tables as validation baselines until downstream migration is complete. Do not add new downstream dependencies to legacy derived tables.
+
+## Git workflow
+
+Repository:
+
+```text
+https://git.chenwuzhu.cn/chenwu/REFACTOR-MA.git
+```
+
+Commit identity for this checkout:
+
+```text
+chenwu <zhuchenwu@chenwuzhu.cn>
+```
+
+Workflow rules:
+
+1. Work on the local `main` branch unless the user requests a review branch.
+2. Check `git status` before editing and never discard unrelated user changes.
+3. Run whitespace/static checks and inspect staged diffs before committing.
+4. Use focused imperative commit messages that describe the warehouse change.
+5. Push to `origin/main` only after the requested batch is complete and locally verified.
+6. Never force-push or rewrite shared history without explicit approval.
+7. Report the pushed commit hash and any Databricks validation that could not be executed locally.
+
+Recent baseline commits:
+
+- `3f3dc44`: initial hierarchy SQL refactor.
+- `33e60d0`: move hierarchy outputs to DWS naming.
+- `b0d5f81`: add commented one-time DDL templates.