diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..d00d014 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,146 @@ +# AGENTS.md + +## Repository purpose + +This repository contains the maintainable Databricks SQL replacement for the legacy CHPA warehouse jobs. + +- Treat the workspace-level `CHPA/` directory as read-only legacy input. +- Put all rewritten code, documentation, and validation under `RE/`. +- Preserve business behavior first. Separate logic refactoring, physical renaming, and business-rule changes. +- Keep each persisted target in one clearly owned script whenever practical. + +## Warehouse layers + +The warehouse has exactly three persistent layers and data flows in one direction: + +```text +DWD -> DWS -> DM +``` + +### DWD + +- Cleaned atomic details and foundational master data. +- Retains source traceability and stable source grain. +- Does not contain report-oriented hierarchy flattening or subject metrics. + +### DWS + +- Reusable dimensions, hierarchy-wide tables, and reusable facts built from DWD. +- Standardizes shared codes, joins, and business definitions. +- A job that reads DWD and produces a reusable derived table writes to DWS. + +### DM + +- Subject-area datasets, metrics, and report-facing outputs built from DWS. +- Must not become an upstream dependency of DWD or DWS. + +DWD must not read DWS/DM. DWS must not read DM. DWS/DM derived results must not be written back to DWD, and DM derived results must not be written back to DWS. + +## Table naming + +Use the schema that matches the target layer and one of these fixed table prefixes: + +| Layer | Object type | Required form | +| --- | --- | --- | +| DWS | Dimension/master data | `dws.dws_ext_td_` | +| DWS | Fact data | `dws.dws_ext_tf_` | +| DM | Dimension/master data | `dm.dm_ext_td_` | +| DM | Fact data | `dm.dm_ext_tf_` | + +Naming rules: + +- Use lowercase snake_case table names. +- `td` means dimension/master data; `tf` means fact data. +- Include the source/domain at the start of the business entity when it avoids ambiguity, for example `ims_atc_hierarchy`. +- Do not write a newly derived DWS table with a `dwd_` name or into the `dwd` schema. +- Keep legacy output column names during a logic-only migration. Rename columns only through an explicit coordinated contract change. + +Current hierarchy targets: + +- `dws.dws_ext_td_ims_atc_hierarchy` +- `dws.dws_ext_td_ims_nfc_hierarchy` + +## File layout and order + +Use: + +```text +sql///_.sql +validation///.sql +``` + +Rules: + +- DWD jobs belong in `01_dwd`, DWS jobs in `02_dws`, and DM jobs in `03_dm`. +- Use two-digit sequence numbers to make execution order explicit. +- Use lowercase snake_case paths with no spaces. +- Keep Databricks notebook markers and split executable statements with `-- COMMAND ----------`. +- Split legacy multi-target scripts by target or responsibility when doing so does not alter transactional behavior. + +## SQL contracts + +Every persisted build script must include a concise header with: + +- purpose; +- source tables; +- target table; +- output grain; +- write mode; +- legacy table replaced; +- known consumers or migration dependency. + +SQL rules: + +- Use uppercase SQL keywords and descriptive lowercase snake_case CTE aliases. +- Use four-space indentation and one selected column per line. +- Use explicit target columns and explicit final projection columns. +- Never use `SELECT *` in a persisted write. +- Qualify columns whenever multiple relations are in scope. +- Replace post-write null cleanup with projection expressions such as `COALESCE` when semantics are unchanged. +- Replace update-driven staging with ordered CTEs only after proving the result is equivalent. +- Keep one-time target DDL in a commented, separate Databricks cell above the write. +- Prefer `CREATE TABLE IF NOT EXISTS LIKE ` when the legacy schema is the authoritative type contract. +- Do not guess source field types when no DDL or `DESCRIBE` output is available. + +## Validation + +Before migrating downstream consumers: + +- compare old and new row counts on the same source snapshot; +- run `EXCEPT ALL` in both directions with a fixed column order; +- profile duplicate business keys or paths; +- count null required keys; +- count unmatched hierarchy/join levels; +- make compatibility checks fail the notebook with `raise_error()` when any required check fails. + +Keep old physical tables as validation baselines until downstream migration is complete. Do not add new downstream dependencies to legacy derived tables. + +## Git workflow + +Repository: + +```text +https://git.chenwuzhu.cn/chenwu/REFACTOR-MA.git +``` + +Commit identity for this checkout: + +```text +chenwu +``` + +Workflow rules: + +1. Work on the local `main` branch unless the user requests a review branch. +2. Check `git status` before editing and never discard unrelated user changes. +3. Run whitespace/static checks and inspect staged diffs before committing. +4. Use focused imperative commit messages that describe the warehouse change. +5. Push to `origin/main` only after the requested batch is complete and locally verified. +6. Never force-push or rewrite shared history without explicit approval. +7. Report the pushed commit hash and any Databricks validation that could not be executed locally. + +Recent baseline commits: + +- `3f3dc44`: initial hierarchy SQL refactor. +- `33e60d0`: move hierarchy outputs to DWS naming. +- `b0d5f81`: add commented one-time DDL templates. diff --git a/docs/chpa_01_02_migration.md b/docs/chpa_01_02_migration.md new file mode 100644 index 0000000..e183a89 --- /dev/null +++ b/docs/chpa_01_02_migration.md @@ -0,0 +1,95 @@ +# CHPA 01/02 重构迁移清单 + +## 1. 范围 + +本批覆盖工作区 `CHPA/` 下全部 6 个 `01` 开头脚本和 11 个 `02` 开头脚本。原文件保持只读，新代码写入 `RE/sql/chpa/01_dwd` 和 `RE/sql/chpa/02_dws`。 + +重构遵循： + +```text +DWD -> DWS -> DM +``` + +本批不修改 `03` 开头脚本。所有 `03` 消费者的旧表引用必须在下一批统一迁移，旧表在此之前保留为验证基线。 + +## 2. 旧文件到新文件映射 + +| 旧文件 | 新文件 | 新目标或处理方式 | +| --- | --- | --- | +| `01 dwd_update.sql` | `01_dwd/01_standardize_gnd_codes.sql` | DWD 配置编码标准化 | +| `01 dwd_update.sql` | `01_dwd/02_refresh_ims_market_config.sql` | 保留 `dwd.dwd_gnd_ims_tblmarket` | +| `01 dwd_update.sql` | `01_dwd/03_refresh_retail_market_config.sql` | 保留 `dwd.dwd_gnd_ext_retail_tblmarket` | +| `01 dwd_update.sql` | `01_dwd/04_refresh_dtp_market_config.sql` | 保留 `dwd.dwd_gnd_dtp_tblmarket` | +| `01 dwd_update.sql` | `01_dwd/05_normalize_gnd_time_windows.sql` | DWD 时间窗口标准化，保留旧默认值差异 | +| `01 dwd_ims_td_manufacturer_corp.sql` | `01_dwd/06_fix_ims_manufacturer_type.sql` | DWD `ManufacturerType_ID=0 -> 2` 修复 | +| `01_FB_BLOB_TO_DWD.sql` | `01_dwd/07_ingest_pharbers_province_fact.sql` | 保留 `dwd.dwd_gnd_pharbers_prov_fact` | +| `01 dwd_ims_atc_hierarchy.sql` | `02_dws/01_dws_ext_td_ims_atc_hierarchy.sql` | `dws.dws_ext_td_ims_atc_hierarchy` | +| `01 dwd_ims_nfc_hierarchy.sql` | `02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql` | `dws.dws_ext_td_ims_nfc_hierarchy` | +| `01 dwd_ims_td_manufacturer_corp.sql` | `02_dws/03_dws_ext_td_ims_manufacturer_corporation.sql` | `dws.dws_ext_td_ims_manufacturer_corporation` | +| `01 dwd_ims_td_pack_property.sql` | `02_dws/04_dws_ext_td_ims_pack_property.sql` | `dws.dws_ext_td_ims_pack_property` | +| `02 DWS_IMS_TD_GEO.sql` | `02_dws/05_dws_ext_td_ims_geo.sql` | `dws.dws_ext_td_ims_geo` | +| `02 dws_ims_td_corp_cn.sql` | `02_dws/06_dws_ext_td_ims_corporation_cn.sql` | `dws.dws_ext_td_ims_corporation_cn` | +| `02 dws_ims_td_manu_cn.sql` | `02_dws/07_dws_ext_td_ims_manufacturer_cn.sql` | `dws.dws_ext_td_ims_manufacturer_cn` | +| `02 tmp_ims_td_prod_tmp.sql` | `02_dws/08_dws_ext_td_ims_product_multi_manufacturer.sql` | 将临时名单升级为声明式 DWS 标记维表 | +| `02 dws_ims_td_prod_cn.sql` | `02_dws/09_dws_ext_td_ims_product_cn.sql` | `dws.dws_ext_td_ims_product_cn`，声明式生成 `RANK_TYPE` | +| `02 dws_ims_td_atc_cn.sql` | `02_dws/10_dws_ext_td_ims_atc_cn.sql` | `dws.dws_ext_td_ims_atc_cn` | +| `02 dws_ims_td_nfc_cn.sql` | `02_dws/11_dws_ext_td_ims_nfc_cn.sql` | `dws.dws_ext_td_ims_nfc_cn` | +| `02 dws_ims_td_market_ta.sql` | `02_dws/12_dws_ext_td_ims_market_ta.sql` | `dws.dws_ext_td_ims_market_ta` | +| `02 tmp_ims_tf_fact_sales.sql` | `02_dws/13_dws_ext_td_ims_pack_ym.sql` | `dws.dws_ext_td_ims_pack_ym` | +| `02 tmp_ims_tf_fact_sales.sql` | `02_dws/14_dws_ext_tf_ims_chpa_sales.sql` | `dws.dws_ext_tf_ims_chpa_sales` | +| `02 dws_ims_td_date.sql` | `02_dws/15_dws_ext_td_ims_date.sql` | `dws.dws_ext_td_ims_date` | +| `02 dws_ims_td_market.sql` | `02_dws/16_dws_ext_td_ims_market.sql` | `dws.dws_ext_td_ims_market` | + +## 3. 执行顺序 + +### DWD + +1. 标准化 GND 配置编码。 +2. 刷新 IMS、Retail、DTP 市场配置快照。 +3. 按旧业务口径补齐各配置表时间窗口。 +4. 修复 manufacturer 类型。 +5. Pharbers 省级事实入湖任务独立运行；无匹配文件时必须失败，不能复用旧 staging 数据。 + +### DWS + +1. ATC/NFC hierarchy。 +2. Manufacturer-corporation mapping。 +3. Pack property。 +4. Geo、corporation/manufacturer 中文维度。 +5. 多厂家产品标记、产品中文维度。 +6. ATC/NFC 中文维度、market-TA mapping。 +7. Pack 月快照、CHPA sales 事实、日期维度。 +8. Market 维度。 + +## 4. 明确保留的兼容行为 + +- ATC/NFC 父级编码截取长度保持不变。 +- Pack、product、molecule 编码补零规则保持不变。 +- A5Z/A5ZD corporation/manufacturer 覆盖规则保持不变。 +- GND 时间默认值中 `190001/209901` 与 `200001/299912` 的历史差异暂不统一。 +- CHPA sales 全国数据从 `202201` 开始，省级数据不增加该下限。 +- Market 扩展市场 ratio 的旧脚本净结果保持为 `1`；缺失 `ELSE` 的疑点只记录，不在本批修复。 +- 日期维度保留 `max(YM)-900` 的旧窗口算法，尽管旧注释称“最近五年”。 + +## 5. 本批修复的工程问题 + +- DWD 派生宽表迁移到 DWS，并使用 `dws_ext_td_` / `dws_ext_tf_` 命名。 +- 持久化写入使用显式目标字段和最终投影。 +- 一次性建表语句以注释形式放在独立 Databricks cell。 +- 产品排序从“临时表 + UPDATE”改为声明式构建。 +- CHPA sales 从共享 `tmp` 表提升为正式 DWS 事实表。 +- DWS sales 不再读取 DM geography，改为读取本批 DWS geo。 +- Market 的共享临时表、MERGE DELETE 和 UPDATE 改为 notebook 内 CTE、anti join、rank 和投影。 +- Pharbers 入湖在无文件时失败，避免旧 staging 被再次覆盖到 DWD。 + +## 6. 下一批必须迁移的 03 消费者 + +本批新表上线后，以下类型的旧引用仍存在于 `03` 脚本中： + +- `dws.dws_ims_td_geo`、`dws.dws_ims_td_market`； +- `dws.dws_ims_td_*_cn`； +- `dwd.dwd_ims_td_pack_property`； +- `tmp.tmp_ims_tf_fact_sales`； +- `dws.dws_ims_td_pack_ym`。 + +在 `03` 消费者全部迁移并完成新旧双向差集验证之前，不删除旧表。