From 33e60d02e3b77615e6dd72ceb0fb01fb3e10d178 Mon Sep 17 00:00:00 2001 From: chenwu Date: Thu, 20 Aug 2026 19:01:46 +0800 Subject: [PATCH] align hierarchy outputs with DWS naming --- README.md | 60 ++- docs/sql_refactoring_standard.md | 147 ++++--- .../01_dws_ext_td_ims_atc_hierarchy.sql} | 7 +- .../02_dws_ext_td_ims_nfc_hierarchy.sql} | 7 +- .../01_dwd/validate_hierarchy_refactor.sql | 396 ------------------ .../02_dws/validate_hierarchy_refactor.sql | 250 +++++++++++ 6 files changed, 380 insertions(+), 487 deletions(-) rename sql/chpa/{01_dwd/01_dwd_ims_atc_hierarchy.sql => 02_dws/01_dws_ext_td_ims_atc_hierarchy.sql} (90%) rename sql/chpa/{01_dwd/02_dwd_ims_nfc_hierarchy.sql => 02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql} (88%) delete mode 100644 validation/chpa/01_dwd/validate_hierarchy_refactor.sql create mode 100644 validation/chpa/02_dws/validate_hierarchy_refactor.sql diff --git a/README.md b/README.md index e3208c6..7dec781 100644 --- a/README.md +++ b/README.md @@ -2,23 +2,40 @@ Databricks 数仓 SQL 重构仓库。 +## 数仓分层 + +仓库采用三层模型，数据依赖保持单向流动： + +```text +DWD -> DWS -> DM +``` + +- `DWD`：清洗后的原子明细和基础主数据。 +- `DWS`：引用 DWD 构建的公共维度、宽表和可复用事实表。 +- `DM`：面向具体分析主题、指标和报表的数据集市。 + ## 首批 review 范围 -本批次只重构原 `CHPA/01` 中两个层级维表脚本： +本批次重构原 `CHPA/01` 中两个层级维表脚本： -- `sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql` -- `sql/chpa/01_dwd/02_dwd_ims_nfc_hierarchy.sql` +- `sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql` +- `sql/chpa/02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql` -工作区 `CHPA/` 下的原脚本保持不变，便于逐项对照。 +两个脚本读取 DWD，并写入新的 DWS 维度表： -## 兼容性决策 +- `dws.dws_ext_td_ims_atc_hierarchy` +- `dws.dws_ext_td_ims_nfc_hierarchy` -首批代码继续写入现有物理表： +工作区 `CHPA/` 下的原脚本保持不变。旧表 `dwd.dwd_ims_atc_hierarchy` 和 `dwd.dwd_ims_nfc_hierarchy` 仅作为本批新旧结果校验基线。 -- `dwd.dwd_ims_atc_hierarchy` -- `dwd.dwd_ims_nfc_hierarchy` +## 表命名 -这样不会立即影响 pack property 和 DWS 等现有下游任务。推荐的新表名及迁移规则记录在 `docs/sql_refactoring_standard.md`；是否执行物理改名，待本批 review 后决定。 +- DWS 维度表：`dws_ext_td_<业务实体>` +- DWS 事实表：`dws_ext_tf_<业务实体>` +- DM 维度表：`dm_ext_td_<业务实体>` +- DM 事实表：`dm_ext_tf_<业务实体>` + +完整规范及首批表名映射见 `docs/sql_refactoring_standard.md`。 ## 目录结构 @@ -29,26 +46,25 @@ RE/ | `-- sql_refactoring_standard.md |-- sql/ | `-- chpa/ -| `-- 01_dwd/ -| |-- 01_dwd_ims_atc_hierarchy.sql -| `-- 02_dwd_ims_nfc_hierarchy.sql +| `-- 02_dws/ +| |-- 01_dws_ext_td_ims_atc_hierarchy.sql +| `-- 02_dws_ext_td_ims_nfc_hierarchy.sql `-- validation/ `-- chpa/ - `-- 01_dwd/ + `-- 02_dws/ `-- validate_hierarchy_refactor.sql ``` ## 执行顺序 -1. 在覆盖目标表前运行 `validation/chpa/01_dwd/validate_hierarchy_refactor.sql`。 -2. 核对两个 `DESCRIBE TABLE` 结果；兼容性检查必须全部返回 `passed = true`。 -3. review 重复路径和未匹配层级指标，确认它们符合现有业务口径。 -4. 通过后，按文件名前缀顺序执行 `sql/chpa/01_dwd/` 中的重构脚本。 - -两个重构脚本都要求现有 DWD 源表、目标表已经创建。 +1. 确认两个 DWS 目标表已按脚本中的显式字段契约创建。 +2. 按文件名前缀顺序执行 `sql/chpa/02_dws/` 中的脚本。 +3. 运行 `validation/chpa/02_dws/validate_hierarchy_refactor.sql`，直接比较新 DWS 输出与旧 DWD 基线。 +4. 兼容性检查必须全部返回 `passed = true`；同时 review 重复路径和未匹配层级指标。 +5. 下游迁移完成前保留旧 DWD 表；下游不得继续新增对旧表的依赖。 ## 本次 review 重点 -1. 表命名模型是否适用于整个数仓。 -2. 物理表改名采用兼容视图还是一次性联动下游。 -3. 替换调度任务前，在 Databricks 中对比新旧结果的行数、业务键重复数和空键数。 +1. `DWD -> DWS -> DM` 的职责边界是否符合现有调度设计。 +2. `dws_ext_td_ims_*` 中是否需要保留来源系统 `ims`。 +3. DWS 目标表的 catalog、字段类型、表属性和权限是否需要统一 DDL 模板。 diff --git a/docs/sql_refactoring_standard.md b/docs/sql_refactoring_standard.md index c841df9..c5ef6a2 100644 --- a/docs/sql_refactoring_standard.md +++ b/docs/sql_refactoring_standard.md @@ -2,16 +2,77 @@ ## 1. 范围和原则 -本规范适用于 CHPA Databricks 数仓脚本。第一阶段目标是在不改变业务结果的前提下提高可维护性。 +本规范适用于 CHPA Databricks 数仓脚本。第一阶段目标是在不改变业务口径的前提下提高可维护性。 - 先保持业务口径，再做性能优化。 - 每个脚本只负责一个主要目标表。 -- 文件头声明源表、目标表、数据粒度、写入方式和依赖。 +- 文件头声明源表、目标表、数据粒度、写入方式和迁移关系。 - 持久化写入必须显式列出目标列和查询列，禁止使用 `SELECT *`。 -- 逻辑重构与生产表改名分开实施。 -- 为行数、唯一性、非空约束和未匹配记录建立验证查询或任务检查。 +- 为行数、唯一性、非空约束和未匹配记录建立验证检查。 -## 2. 文件和目录命名 +## 2. 数仓分层 + +数仓固定为三层，依赖方向为： + +```text +DWD -> DWS -> DM +``` + +### DWD + +- 保存清洗、标准化后的原子明细和基础主数据。 +- 保持源数据可追溯性和稳定粒度。 +- 不承载面向报表的层级拉宽、跨主题指标聚合。 + +### DWS + +- 引用 DWD 构建公共维度、层级宽表和可复用事实表。 +- 统一编码、公共口径和跨明细关联结果。 +- 本批 ATC/NFC 层级宽维表属于 DWS，不再写回 DWD。 + +### DM + +- 引用 DWS 构建具体业务主题、指标和报表数据集。 +- 允许面向使用场景组织字段，但不得反向成为 DWD/DWS 的依赖。 + +读取关系保持清晰：DWS 读取 DWD，DM 读取 DWS。DWD 不读取 DWS/DM，DWS 不读取 DM；DWS/DM 的派生结果不得写回 DWD，DM 的派生结果不得写回 DWS。 + +## 3. 表命名 + +表名由“层级 + ext + 对象类型 + 业务实体”组成： + +```text +[.]._ext__ +``` + +固定前缀如下： + +| 层级 | 对象 | 表名前缀 | +| --- | --- | --- | +| DWS | 维度/主数据 | `dws_ext_td_` | +| DWS | 事实数据 | `dws_ext_tf_` | +| DM | 维度/主数据 | `dm_ext_td_` | +| DM | 事实数据 | `dm_ext_tf_` | + +命名规则： + +- `` 与 `` 保持一致，例如 `dws.dws_ext_td_xxx`。 +- `td` 表示维度或主数据，`tf` 表示事实数据。 +- `` 使用小写 snake_case。 +- 需要区分来源系统时，将来源放在业务实体开头，例如 `ims_atc_hierarchy`。 +- `atc`、`nfc`、`ims` 等已形成业务共识的缩写可以保留。 + +首批及后续建议映射： + +| 旧表 | 新表 | 类型 | +| --- | --- | --- | +| `dwd.dwd_ims_atc_hierarchy` | `dws.dws_ext_td_ims_atc_hierarchy` | DWS 维度 | +| `dwd.dwd_ims_nfc_hierarchy` | `dws.dws_ext_td_ims_nfc_hierarchy` | DWS 维度 | +| `dwd.dwd_ims_td_manufacturer_corp` | `dws.dws_ext_td_ims_manufacturer_corporation` | DWS 维度，后续处理 | +| `dwd.dwd_ims_td_pack_property` | `dws.dws_ext_td_ims_pack_property` | DWS 维度，后续处理 | +| `dwd.dwd_gnd_pharbers_prov_fact` | `dws.dws_ext_tf_pharbers_province_sales` | DWS 事实，需先确认粒度 | + +## 4. 文件和目录命名 目录结构统一为： @@ -21,63 +82,31 @@ sql/<业务域>/<阶段>/<序号>_<目标表>.sql 规则： -- 目录和文件名使用小写 snake_case。 -- 路径和文件名不使用空格。 -- 使用两位序号明确 notebook/job 的执行顺序。 -- 一个脚本只写入一个主要持久化目标。像 `01 dwd_update.sql` 这种多目标脚本，应按目标表和职责拆分。 +- 目录和文件名使用小写 snake_case，不使用空格。 +- 目录阶段必须和输出层一致，例如 DWS 脚本放在 `02_dws`。 +- 使用两位序号明确 notebook/job 执行顺序。 +- 一个脚本只写入一个主要持久化目标。 +- 多目标脚本按目标表和职责拆分。 示例： ```text -sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql +sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql ``` -## 3. 物理表命名 - -新表使用以下格式： - -```text -[.]<分层_schema>.<来源系统>_<对象类型>_<业务实体>[_<限定词>] -``` - -未启用 Unity Catalog 或依赖默认 catalog 时可以省略 catalog。分层已经由 schema 表达，表名不再重复 `dwd`、`dws` 或 `dm`。 - -- 来源系统：如 `ims`、`gnd`、`pharbers`。 -- 对象类型：`td` 表示维度/主数据，`tf` 表示事实数据。 -- 业务实体：使用稳定的业务名词，如 `atc_hierarchy`、`pack_property`。 -- 限定词：按需表达粒度或变体，如 `monthly`、`province`。 - -`01` 组表名建议： - -| 现有物理表名 | 推荐规范表名 | 首批处理方式 | -| --- | --- | --- | -| `dwd.dwd_ims_atc_hierarchy` | `dwd.ims_td_atc_hierarchy` | 保留现名 | -| `dwd.dwd_ims_nfc_hierarchy` | `dwd.ims_td_nfc_hierarchy` | 保留现名 | -| `dwd.dwd_ims_td_manufacturer_corp` | `dwd.ims_td_manufacturer_corporation` | 后续 review | -| `dwd.dwd_ims_td_pack_property` | `dwd.ims_td_pack_property` | 后续 review | -| `dwd.dwd_gnd_pharbers_prov_fact` | `dwd.pharbers_tf_province_sales` | 先确认业务粒度 | - -推荐迁移顺序： - -1. 创建规范名称的新表，并用兼容视图暴露旧表名。 -2. 新旧表并行运行，对比数据结果。 -3. 在一次受控发布中修改全部下游引用。 -4. 所有消费者迁移后再移除旧名称兼容视图。 - -## 4. 字段命名 +## 5. 字段命名 - 新字段使用小写 snake_case。 - 标识使用 `_id`，业务编码使用 `_code`，名称或描述使用 `_name`、`_description`。 - 日期时间后缀按真实类型使用 `_date`、`_timestamp` 或 `_at`。 -- `atc`、`nfc`、`prod`、`pack` 等已形成业务共识或受输出契约约束的缩写可以保留。 -- 纯逻辑重构中不直接修改持久化输出字段名。 +- 纯逻辑重构中不直接修改已有输出字段契约。 -## 5. SQL 结构和格式 +## 6. SQL 结构和格式 脚本统一按以下顺序组织： 1. Databricks notebook 标记和脚本契约头。 -2. 带显式目标列的 `INSERT OVERWRITE`。 +2. 带显式目标列的写入语句。 3. 源数据标准化 CTE。 4. 业务规则 CTE。 5. 按目标表字段顺序显式编写最终 `SELECT`。 @@ -86,27 +115,21 @@ sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql - SQL 关键字使用大写。 - CTE 和别名使用小写 snake_case。 -- 使用四个空格缩进。 -- `SELECT` 每行一个字段。 -- 每个 JOIN 条件单独成行。 -- 使用 `atc_level_1` 等有业务意义的别名，维护代码中不使用 `t1`、`a` 等无语义别名。 +- 使用四个空格缩进，`SELECT` 每行一个字段。 +- 使用有业务意义的别名，不使用 `t1`、`a` 等无语义别名。 - 同一作用域出现多个关系时，所有字段都带关系限定符。 -## 6. 写入和数据质量规则 +## 7. 写入和数据质量 - 只有可确定性重跑的全量任务可以使用 `INSERT OVERWRITE`。 - 不在一个脚本中混合无关的 `UPDATE` 和目标表构建逻辑。 -- 编码补零等标准化逻辑集中到一个明确的处理阶段，避免在多个任务中重复表达式。 +- 编码补零等标准化逻辑集中到一个明确阶段。 - 每个脚本必须声明预期目标粒度。 -- 最少验证以下指标： - - 新结果与旧结果的总行数； - - 声明业务键的重复数； - - 必填键的空值数； - - 层级关联新增的未匹配记录数。 +- 最少验证新旧行数、双向差集、业务键重复、必填键空值和层级未匹配数。 -## 7. 首批兼容说明 +## 8. 首批兼容说明 -ATC 和 NFC 脚本原样保留以下父级编码规则： +ATC 和 NFC 脚本保留旧逻辑中的父级编码规则： - ATC 2 级关联 1 级：取前 1 位。 - ATC 3 级关联 2 级：取前 3 位。 @@ -114,6 +137,4 @@ ATC 和 NFC 脚本原样保留以下父级编码规则： - NFC 2 级关联 1 级：取前 1 位。 - NFC 3 级关联 2 级：取前 2 位。 -任何口径简化都应先用生产样本验证这些规则。 - -首批上线前运行 `validation/chpa/01_dwd/validate_hierarchy_refactor.sql`。新旧双向差集必须为 0，行数必须一致；重复路径和未匹配层级作为业务 review 指标记录，不在未确认阈值前自动判定失败。 +两个 DWS 脚本执行后运行 `validation/chpa/02_dws/validate_hierarchy_refactor.sql`。新 DWS 输出与旧 DWD 基线的行数必须一致，双向差集必须为 0；重复路径和未匹配层级作为业务 review 指标记录。 diff --git a/sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql b/sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql similarity index 90% rename from sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql rename to sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql index 558fe43..8ce1836 100644 --- a/sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql +++ b/sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql @@ -2,14 +2,15 @@ -- ============================================================================= -- Purpose : Build the flattened IMS ATC level 1-4 hierarchy. -- Source : dwd.dwd_ims_td_therapeutic_class --- Target : dwd.dwd_ims_atc_hierarchy +-- Target : dws.dws_ext_td_ims_atc_hierarchy -- Grain : One row per hierarchy path rooted at ATC1_CODE; lower levels may be null. -- Write mode : Full refresh (INSERT OVERWRITE). --- Downstream : dwd.dwd_ims_td_pack_property, dws.dws_ims_td_atc_cn +-- Replaces : dwd.dwd_ims_atc_hierarchy (legacy output). +-- Consumers : Pack-property and ATC-CN jobs must migrate to the DWS target. -- Notes : Parent-code matching is preserved from the legacy script. -- ============================================================================= -INSERT OVERWRITE TABLE dwd.dwd_ims_atc_hierarchy ( +INSERT OVERWRITE TABLE dws.dws_ext_td_ims_atc_hierarchy ( ATC1_ID, ATC1_CODE, ATC1_DES, diff --git a/sql/chpa/01_dwd/02_dwd_ims_nfc_hierarchy.sql b/sql/chpa/02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql similarity index 88% rename from sql/chpa/01_dwd/02_dwd_ims_nfc_hierarchy.sql rename to sql/chpa/02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql index cbc81ce..a3f12b2 100644 --- a/sql/chpa/01_dwd/02_dwd_ims_nfc_hierarchy.sql +++ b/sql/chpa/02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql @@ -2,14 +2,15 @@ -- ============================================================================= -- Purpose : Build the flattened IMS NFC level 1-3 hierarchy. -- Source : dwd.dwd_ims_td_new_form_class --- Target : dwd.dwd_ims_nfc_hierarchy +-- Target : dws.dws_ext_td_ims_nfc_hierarchy -- Grain : One row per hierarchy path rooted at NFC1_CODE; lower levels may be null. -- Write mode : Full refresh (INSERT OVERWRITE). --- Downstream : dwd.dwd_ims_td_pack_property, dws.dws_ims_td_nfc_cn +-- Replaces : dwd.dwd_ims_nfc_hierarchy (legacy output). +-- Consumers : Pack-property and NFC-CN jobs must migrate to the DWS target. -- Notes : Parent-code matching is preserved from the legacy script. -- ============================================================================= -INSERT OVERWRITE TABLE dwd.dwd_ims_nfc_hierarchy ( +INSERT OVERWRITE TABLE dws.dws_ext_td_ims_nfc_hierarchy ( NFC1_ID, NFC1_CODE, NFC1_DES, diff --git a/validation/chpa/01_dwd/validate_hierarchy_refactor.sql b/validation/chpa/01_dwd/validate_hierarchy_refactor.sql deleted file mode 100644 index 23152c0..0000000 --- a/validation/chpa/01_dwd/validate_hierarchy_refactor.sql +++ /dev/null @@ -1,396 +0,0 @@ --- Databricks notebook source --- MAGIC %md --- MAGIC # ATC/NFC hierarchy refactor validation --- MAGIC Run this notebook before executing the refactored overwrite scripts. --- MAGIC The legacy targets and source tables must represent the same data snapshot. - --- COMMAND ---------- - --- Confirm the ATC target contains the expected 12 non-partition output columns. -DESCRIBE TABLE dwd.dwd_ims_atc_hierarchy; - --- COMMAND ---------- - --- Confirm the NFC target contains the expected 9 non-partition output columns. -DESCRIBE TABLE dwd.dwd_ims_nfc_hierarchy; - --- COMMAND ---------- - -CREATE OR REPLACE TEMP VIEW validation_atc_hierarchy_candidate AS -WITH therapeutic_class AS ( - SELECT - therapeutic_id, - therapeutic_code, - therapeutic_name, - therapeutic_level - FROM dwd.dwd_ims_td_therapeutic_class -), -atc_level_1 AS ( - SELECT - therapeutic_id AS atc1_id, - therapeutic_code AS atc1_code, - therapeutic_name AS atc1_des - FROM therapeutic_class - WHERE therapeutic_level = '1' -), -atc_level_2 AS ( - SELECT - therapeutic_id AS atc2_id, - therapeutic_code AS atc2_code, - therapeutic_name AS atc2_des - FROM therapeutic_class - WHERE therapeutic_level = '2' -), -atc_level_3 AS ( - SELECT - therapeutic_id AS atc3_id, - therapeutic_code AS atc3_code, - therapeutic_name AS atc3_des - FROM therapeutic_class - WHERE therapeutic_level = '3' -), -atc_level_4 AS ( - SELECT - therapeutic_id AS atc4_id, - therapeutic_code AS atc4_code, - therapeutic_name AS atc4_des - FROM therapeutic_class - WHERE therapeutic_level = '4' -) -SELECT - atc_level_1.atc1_id AS ATC1_ID, - atc_level_1.atc1_code AS ATC1_CODE, - atc_level_1.atc1_des AS ATC1_DES, - atc_level_2.atc2_id AS ATC2_ID, - atc_level_2.atc2_code AS ATC2_CODE, - atc_level_2.atc2_des AS ATC2_DES, - atc_level_3.atc3_id AS ATC3_ID, - atc_level_3.atc3_code AS ATC3_CODE, - atc_level_3.atc3_des AS ATC3_DES, - atc_level_4.atc4_id AS ATC4_ID, - atc_level_4.atc4_code AS ATC4_CODE, - atc_level_4.atc4_des AS ATC4_DES -FROM atc_level_1 -LEFT JOIN atc_level_2 - ON atc_level_1.atc1_code = LEFT(atc_level_2.atc2_code, 1) -LEFT JOIN atc_level_3 - ON atc_level_2.atc2_code = LEFT(atc_level_3.atc3_code, 3) -LEFT JOIN atc_level_4 - ON atc_level_3.atc3_code = LEFT(atc_level_4.atc4_code, 4) -; - --- COMMAND ---------- - -CREATE OR REPLACE TEMP VIEW validation_nfc_hierarchy_candidate AS -WITH new_form_class AS ( - SELECT - newformclass_id, - newformclass_code, - newformclass_name, - newformclass_level - FROM dwd.dwd_ims_td_new_form_class -), -nfc_level_1 AS ( - SELECT - newformclass_id AS nfc1_id, - newformclass_code AS nfc1_code, - newformclass_name AS nfc1_des - FROM new_form_class - WHERE newformclass_level = '1' -), -nfc_level_2 AS ( - SELECT - newformclass_id AS nfc2_id, - newformclass_code AS nfc2_code, - newformclass_name AS nfc2_des - FROM new_form_class - WHERE newformclass_level = '2' -), -nfc_level_3 AS ( - SELECT - newformclass_id AS nfc3_id, - newformclass_code AS nfc3_code, - newformclass_name AS nfc3_des - FROM new_form_class - WHERE newformclass_level = '3' -) -SELECT - nfc_level_1.nfc1_id AS NFC1_ID, - nfc_level_1.nfc1_code AS NFC1_CODE, - nfc_level_1.nfc1_des AS NFC1_DES, - nfc_level_2.nfc2_id AS NFC2_ID, - nfc_level_2.nfc2_code AS NFC2_CODE, - nfc_level_2.nfc2_des AS NFC2_DES, - nfc_level_3.nfc3_id AS NFC3_ID, - nfc_level_3.nfc3_code AS NFC3_CODE, - nfc_level_3.nfc3_des AS NFC3_DES -FROM nfc_level_1 -LEFT JOIN nfc_level_2 - ON nfc_level_1.nfc1_code = LEFT(nfc_level_2.nfc2_code, 1) -LEFT JOIN nfc_level_3 - ON nfc_level_2.nfc2_code = LEFT(nfc_level_3.nfc3_code, 2) -; - --- COMMAND ---------- - --- Every check must return passed = true before replacing the scheduled jobs. -CREATE OR REPLACE TEMP VIEW validation_hierarchy_compatibility_checks AS -WITH compatibility_checks AS ( - SELECT - 'atc_row_count' AS check_name, - (SELECT COUNT(*) FROM validation_atc_hierarchy_candidate) AS actual_value, - (SELECT COUNT(*) FROM dwd.dwd_ims_atc_hierarchy) AS expected_value - UNION ALL - SELECT - 'atc_legacy_minus_candidate' AS check_name, - ( - SELECT COUNT(*) - FROM ( - SELECT - ATC1_ID, - ATC1_CODE, - ATC1_DES, - ATC2_ID, - ATC2_CODE, - ATC2_DES, - ATC3_ID, - ATC3_CODE, - ATC3_DES, - ATC4_ID, - ATC4_CODE, - ATC4_DES - FROM dwd.dwd_ims_atc_hierarchy - EXCEPT ALL - SELECT - ATC1_ID, - ATC1_CODE, - ATC1_DES, - ATC2_ID, - ATC2_CODE, - ATC2_DES, - ATC3_ID, - ATC3_CODE, - ATC3_DES, - ATC4_ID, - ATC4_CODE, - ATC4_DES - FROM validation_atc_hierarchy_candidate - ) AS differences - ) AS actual_value, - 0 AS expected_value - UNION ALL - SELECT - 'atc_candidate_minus_legacy' AS check_name, - ( - SELECT COUNT(*) - FROM ( - SELECT - ATC1_ID, - ATC1_CODE, - ATC1_DES, - ATC2_ID, - ATC2_CODE, - ATC2_DES, - ATC3_ID, - ATC3_CODE, - ATC3_DES, - ATC4_ID, - ATC4_CODE, - ATC4_DES - FROM validation_atc_hierarchy_candidate - EXCEPT ALL - SELECT - ATC1_ID, - ATC1_CODE, - ATC1_DES, - ATC2_ID, - ATC2_CODE, - ATC2_DES, - ATC3_ID, - ATC3_CODE, - ATC3_DES, - ATC4_ID, - ATC4_CODE, - ATC4_DES - FROM dwd.dwd_ims_atc_hierarchy - ) AS differences - ) AS actual_value, - 0 AS expected_value - UNION ALL - SELECT - 'nfc_row_count' AS check_name, - (SELECT COUNT(*) FROM validation_nfc_hierarchy_candidate) AS actual_value, - (SELECT COUNT(*) FROM dwd.dwd_ims_nfc_hierarchy) AS expected_value - UNION ALL - SELECT - 'nfc_legacy_minus_candidate' AS check_name, - ( - SELECT COUNT(*) - FROM ( - SELECT - NFC1_ID, - NFC1_CODE, - NFC1_DES, - NFC2_ID, - NFC2_CODE, - NFC2_DES, - NFC3_ID, - NFC3_CODE, - NFC3_DES - FROM dwd.dwd_ims_nfc_hierarchy - EXCEPT ALL - SELECT - NFC1_ID, - NFC1_CODE, - NFC1_DES, - NFC2_ID, - NFC2_CODE, - NFC2_DES, - NFC3_ID, - NFC3_CODE, - NFC3_DES - FROM validation_nfc_hierarchy_candidate - ) AS differences - ) AS actual_value, - 0 AS expected_value - UNION ALL - SELECT - 'nfc_candidate_minus_legacy' AS check_name, - ( - SELECT COUNT(*) - FROM ( - SELECT - NFC1_ID, - NFC1_CODE, - NFC1_DES, - NFC2_ID, - NFC2_CODE, - NFC2_DES, - NFC3_ID, - NFC3_CODE, - NFC3_DES - FROM validation_nfc_hierarchy_candidate - EXCEPT ALL - SELECT - NFC1_ID, - NFC1_CODE, - NFC1_DES, - NFC2_ID, - NFC2_CODE, - NFC2_DES, - NFC3_ID, - NFC3_CODE, - NFC3_DES - FROM dwd.dwd_ims_nfc_hierarchy - ) AS differences - ) AS actual_value, - 0 AS expected_value - UNION ALL - SELECT - 'except_all_null_semantics' AS check_name, - ( - SELECT COUNT(*) - FROM ( - SELECT CAST(NULL AS STRING) AS nullable_value - EXCEPT ALL - SELECT CAST(NULL AS STRING) AS nullable_value - ) AS differences - ) AS actual_value, - 0 AS expected_value -) -SELECT - check_name, - actual_value, - expected_value, - actual_value = expected_value AS passed -FROM compatibility_checks -; - --- COMMAND ---------- - -SELECT - check_name, - actual_value, - expected_value, - passed -FROM validation_hierarchy_compatibility_checks -ORDER BY check_name; - --- COMMAND ---------- - --- Profile duplicates and unmatched levels. These are review metrics, not automatic failures. -WITH atc_duplicate_paths AS ( - SELECT COUNT(*) AS path_count - FROM validation_atc_hierarchy_candidate - GROUP BY ATC1_CODE, ATC2_CODE, ATC3_CODE, ATC4_CODE - HAVING COUNT(*) > 1 -), -nfc_duplicate_paths AS ( - SELECT COUNT(*) AS path_count - FROM validation_nfc_hierarchy_candidate - GROUP BY NFC1_CODE, NFC2_CODE, NFC3_CODE - HAVING COUNT(*) > 1 -) -SELECT - 'atc_duplicate_path_rows' AS metric_name, - COALESCE(SUM(path_count - 1), 0) AS metric_value -FROM atc_duplicate_paths -UNION ALL -SELECT - 'atc_source_level_1_required_key_null' AS metric_name, - COUNT_IF(therapeutic_id IS NULL OR therapeutic_code IS NULL) AS metric_value -FROM dwd.dwd_ims_td_therapeutic_class -WHERE therapeutic_level = '1' -UNION ALL -SELECT - 'atc_unmatched_level_2' AS metric_name, - COUNT_IF(ATC2_CODE IS NULL) AS metric_value -FROM validation_atc_hierarchy_candidate -UNION ALL -SELECT - 'atc_unmatched_level_3' AS metric_name, - COUNT_IF(ATC3_CODE IS NULL) AS metric_value -FROM validation_atc_hierarchy_candidate -UNION ALL -SELECT - 'atc_unmatched_level_4' AS metric_name, - COUNT_IF(ATC4_CODE IS NULL) AS metric_value -FROM validation_atc_hierarchy_candidate -UNION ALL -SELECT - 'nfc_duplicate_path_rows' AS metric_name, - COALESCE(SUM(path_count - 1), 0) AS metric_value -FROM nfc_duplicate_paths -UNION ALL -SELECT - 'nfc_source_level_1_required_key_null' AS metric_name, - COUNT_IF(newformclass_id IS NULL OR newformclass_code IS NULL) AS metric_value -FROM dwd.dwd_ims_td_new_form_class -WHERE newformclass_level = '1' -UNION ALL -SELECT - 'nfc_unmatched_level_2' AS metric_name, - COUNT_IF(NFC2_CODE IS NULL) AS metric_value -FROM validation_nfc_hierarchy_candidate -UNION ALL -SELECT - 'nfc_unmatched_level_3' AS metric_name, - COUNT_IF(NFC3_CODE IS NULL) AS metric_value -FROM validation_nfc_hierarchy_candidate -ORDER BY metric_name; - --- COMMAND ---------- - --- Abort the notebook when any compatibility check fails. -SELECT IF( - COUNT_IF(NOT passed) = 0, - 'All hierarchy compatibility checks passed.', - raise_error( - CONCAT( - 'Hierarchy compatibility validation failed: ', - CAST(COUNT_IF(NOT passed) AS STRING), - ' check(s) did not pass.' - ) - ) -) AS validation_result -FROM validation_hierarchy_compatibility_checks; diff --git a/validation/chpa/02_dws/validate_hierarchy_refactor.sql b/validation/chpa/02_dws/validate_hierarchy_refactor.sql new file mode 100644 index 0000000..fb74a36 --- /dev/null +++ b/validation/chpa/02_dws/validate_hierarchy_refactor.sql @@ -0,0 +1,250 @@ +-- Databricks notebook source +-- MAGIC %md +-- MAGIC # ATC/NFC DWS hierarchy validation +-- MAGIC Run this notebook after both DWS hierarchy scripts finish. +-- MAGIC The DWS outputs and legacy DWD baselines must represent the same source snapshot. + +-- COMMAND ---------- + +DESCRIBE TABLE dws.dws_ext_td_ims_atc_hierarchy; + +-- COMMAND ---------- + +DESCRIBE TABLE dws.dws_ext_td_ims_nfc_hierarchy; + +-- COMMAND ---------- + +-- Fix the comparison column set and order at the view boundary. +CREATE OR REPLACE TEMP VIEW validation_atc_legacy AS +SELECT + ATC1_ID, + ATC1_CODE, + ATC1_DES, + ATC2_ID, + ATC2_CODE, + ATC2_DES, + ATC3_ID, + ATC3_CODE, + ATC3_DES, + ATC4_ID, + ATC4_CODE, + ATC4_DES +FROM dwd.dwd_ims_atc_hierarchy; + +-- COMMAND ---------- + +CREATE OR REPLACE TEMP VIEW validation_atc_dws AS +SELECT + ATC1_ID, + ATC1_CODE, + ATC1_DES, + ATC2_ID, + ATC2_CODE, + ATC2_DES, + ATC3_ID, + ATC3_CODE, + ATC3_DES, + ATC4_ID, + ATC4_CODE, + ATC4_DES +FROM dws.dws_ext_td_ims_atc_hierarchy; + +-- COMMAND ---------- + +CREATE OR REPLACE TEMP VIEW validation_nfc_legacy AS +SELECT + NFC1_ID, + NFC1_CODE, + NFC1_DES, + NFC2_ID, + NFC2_CODE, + NFC2_DES, + NFC3_ID, + NFC3_CODE, + NFC3_DES +FROM dwd.dwd_ims_nfc_hierarchy; + +-- COMMAND ---------- + +CREATE OR REPLACE TEMP VIEW validation_nfc_dws AS +SELECT + NFC1_ID, + NFC1_CODE, + NFC1_DES, + NFC2_ID, + NFC2_CODE, + NFC2_DES, + NFC3_ID, + NFC3_CODE, + NFC3_DES +FROM dws.dws_ext_td_ims_nfc_hierarchy; + +-- COMMAND ---------- + +CREATE OR REPLACE TEMP VIEW validation_hierarchy_compatibility_checks AS +WITH compatibility_checks AS ( + SELECT + 'atc_row_count' AS check_name, + (SELECT COUNT(*) FROM validation_atc_dws) AS actual_value, + (SELECT COUNT(*) FROM validation_atc_legacy) AS expected_value + UNION ALL + SELECT + 'atc_legacy_minus_dws' AS check_name, + ( + SELECT COUNT(*) + FROM ( + SELECT * FROM validation_atc_legacy + EXCEPT ALL + SELECT * FROM validation_atc_dws + ) AS differences + ) AS actual_value, + 0 AS expected_value + UNION ALL + SELECT + 'atc_dws_minus_legacy' AS check_name, + ( + SELECT COUNT(*) + FROM ( + SELECT * FROM validation_atc_dws + EXCEPT ALL + SELECT * FROM validation_atc_legacy + ) AS differences + ) AS actual_value, + 0 AS expected_value + UNION ALL + SELECT + 'nfc_row_count' AS check_name, + (SELECT COUNT(*) FROM validation_nfc_dws) AS actual_value, + (SELECT COUNT(*) FROM validation_nfc_legacy) AS expected_value + UNION ALL + SELECT + 'nfc_legacy_minus_dws' AS check_name, + ( + SELECT COUNT(*) + FROM ( + SELECT * FROM validation_nfc_legacy + EXCEPT ALL + SELECT * FROM validation_nfc_dws + ) AS differences + ) AS actual_value, + 0 AS expected_value + UNION ALL + SELECT + 'nfc_dws_minus_legacy' AS check_name, + ( + SELECT COUNT(*) + FROM ( + SELECT * FROM validation_nfc_dws + EXCEPT ALL + SELECT * FROM validation_nfc_legacy + ) AS differences + ) AS actual_value, + 0 AS expected_value + UNION ALL + SELECT + 'except_all_null_semantics' AS check_name, + ( + SELECT COUNT(*) + FROM ( + SELECT CAST(NULL AS STRING) AS nullable_value + EXCEPT ALL + SELECT CAST(NULL AS STRING) AS nullable_value + ) AS differences + ) AS actual_value, + 0 AS expected_value +) +SELECT + check_name, + actual_value, + expected_value, + actual_value = expected_value AS passed +FROM compatibility_checks; + +-- COMMAND ---------- + +SELECT + check_name, + actual_value, + expected_value, + passed +FROM validation_hierarchy_compatibility_checks +ORDER BY check_name; + +-- COMMAND ---------- + +-- These quality metrics require business review; they do not fail automatically. +WITH atc_duplicate_paths AS ( + SELECT COUNT(*) AS path_count + FROM validation_atc_dws + GROUP BY ATC1_CODE, ATC2_CODE, ATC3_CODE, ATC4_CODE + HAVING COUNT(*) > 1 +), +nfc_duplicate_paths AS ( + SELECT COUNT(*) AS path_count + FROM validation_nfc_dws + GROUP BY NFC1_CODE, NFC2_CODE, NFC3_CODE + HAVING COUNT(*) > 1 +) +SELECT + 'atc_duplicate_path_rows' AS metric_name, + COALESCE(SUM(path_count - 1), 0) AS metric_value +FROM atc_duplicate_paths +UNION ALL +SELECT + 'atc_source_level_1_required_key_null' AS metric_name, + COUNT_IF(therapeutic_id IS NULL OR therapeutic_code IS NULL) AS metric_value +FROM dwd.dwd_ims_td_therapeutic_class +WHERE therapeutic_level = '1' +UNION ALL +SELECT + 'atc_unmatched_level_2' AS metric_name, + COUNT_IF(ATC2_CODE IS NULL) AS metric_value +FROM validation_atc_dws +UNION ALL +SELECT + 'atc_unmatched_level_3' AS metric_name, + COUNT_IF(ATC3_CODE IS NULL) AS metric_value +FROM validation_atc_dws +UNION ALL +SELECT + 'atc_unmatched_level_4' AS metric_name, + COUNT_IF(ATC4_CODE IS NULL) AS metric_value +FROM validation_atc_dws +UNION ALL +SELECT + 'nfc_duplicate_path_rows' AS metric_name, + COALESCE(SUM(path_count - 1), 0) AS metric_value +FROM nfc_duplicate_paths +UNION ALL +SELECT + 'nfc_source_level_1_required_key_null' AS metric_name, + COUNT_IF(newformclass_id IS NULL OR newformclass_code IS NULL) AS metric_value +FROM dwd.dwd_ims_td_new_form_class +WHERE newformclass_level = '1' +UNION ALL +SELECT + 'nfc_unmatched_level_2' AS metric_name, + COUNT_IF(NFC2_CODE IS NULL) AS metric_value +FROM validation_nfc_dws +UNION ALL +SELECT + 'nfc_unmatched_level_3' AS metric_name, + COUNT_IF(NFC3_CODE IS NULL) AS metric_value +FROM validation_nfc_dws +ORDER BY metric_name; + +-- COMMAND ---------- + +-- Block downstream migration when any DWS-to-DWD compatibility check fails. +SELECT IF( + COUNT_IF(NOT passed) = 0, + 'All DWS hierarchy compatibility checks passed.', + raise_error( + CONCAT( + 'DWS hierarchy compatibility validation failed: ', + CAST(COUNT_IF(NOT passed) AS STRING), + ' check(s) did not pass.' + ) + ) +) AS validation_result +FROM validation_hierarchy_compatibility_checks;