align hierarchy outputs with DWS naming

README.md

@@ -2,23 +2,40 @@
 
 Databricks 数仓 SQL 重构仓库。
 
+## 数仓分层
+
+仓库采用三层模型，数据依赖保持单向流动：
+
+```text
+DWD -> DWS -> DM
+```
+
+- `DWD`：清洗后的原子明细和基础主数据。
+- `DWS`：引用 DWD 构建的公共维度、宽表和可复用事实表。
+- `DM`：面向具体分析主题、指标和报表的数据集市。
+
 ## 首批 review 范围
 
-本批次只重构原 `CHPA/01` 中两个层级维表脚本：
+本批次重构原 `CHPA/01` 中两个层级维表脚本：
 
-- `sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql`
-- `sql/chpa/01_dwd/02_dwd_ims_nfc_hierarchy.sql`
+- `sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql`
+- `sql/chpa/02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql`
 
-工作区 `CHPA/` 下的原脚本保持不变，便于逐项对照。
+两个脚本读取 DWD，并写入新的 DWS 维度表：
 
-## 兼容性决策
+- `dws.dws_ext_td_ims_atc_hierarchy`
+- `dws.dws_ext_td_ims_nfc_hierarchy`
 
-首批代码继续写入现有物理表：
+工作区 `CHPA/` 下的原脚本保持不变。旧表 `dwd.dwd_ims_atc_hierarchy` 和 `dwd.dwd_ims_nfc_hierarchy` 仅作为本批新旧结果校验基线。
 
-- `dwd.dwd_ims_atc_hierarchy`
-- `dwd.dwd_ims_nfc_hierarchy`
+## 表命名
 
-这样不会立即影响 pack property 和 DWS 等现有下游任务。推荐的新表名及迁移规则记录在 `docs/sql_refactoring_standard.md`；是否执行物理改名，待本批 review 后决定。
+- DWS 维度表：`dws_ext_td_<业务实体>`
+- DWS 事实表：`dws_ext_tf_<业务实体>`
+- DM 维度表：`dm_ext_td_<业务实体>`
+- DM 事实表：`dm_ext_tf_<业务实体>`
+
+完整规范及首批表名映射见 `docs/sql_refactoring_standard.md`。
 
 ## 目录结构
 
@@ -29,26 +46,25 @@ RE/
 |   `-- sql_refactoring_standard.md
 |-- sql/
 |   `-- chpa/
-|       `-- 01_dwd/
-|           |-- 01_dwd_ims_atc_hierarchy.sql
-|           `-- 02_dwd_ims_nfc_hierarchy.sql
+|       `-- 02_dws/
+|           |-- 01_dws_ext_td_ims_atc_hierarchy.sql
+|           `-- 02_dws_ext_td_ims_nfc_hierarchy.sql
 `-- validation/
     `-- chpa/
-        `-- 01_dwd/
+        `-- 02_dws/
             `-- validate_hierarchy_refactor.sql
 ```
 
 ## 执行顺序
 
-1. 在覆盖目标表前运行 `validation/chpa/01_dwd/validate_hierarchy_refactor.sql`。
-2. 核对两个 `DESCRIBE TABLE` 结果；兼容性检查必须全部返回 `passed = true`。
-3. review 重复路径和未匹配层级指标，确认它们符合现有业务口径。
-4. 通过后，按文件名前缀顺序执行 `sql/chpa/01_dwd/` 中的重构脚本。
-
-两个重构脚本都要求现有 DWD 源表、目标表已经创建。
+1. 确认两个 DWS 目标表已按脚本中的显式字段契约创建。
+2. 按文件名前缀顺序执行 `sql/chpa/02_dws/` 中的脚本。
+3. 运行 `validation/chpa/02_dws/validate_hierarchy_refactor.sql`，直接比较新 DWS 输出与旧 DWD 基线。
+4. 兼容性检查必须全部返回 `passed = true`；同时 review 重复路径和未匹配层级指标。
+5. 下游迁移完成前保留旧 DWD 表；下游不得继续新增对旧表的依赖。
 
 ## 本次 review 重点
 
-1. 表命名模型是否适用于整个数仓。
-2. 物理表改名采用兼容视图还是一次性联动下游。
-3. 替换调度任务前，在 Databricks 中对比新旧结果的行数、业务键重复数和空键数。
+1. `DWD -> DWS -> DM` 的职责边界是否符合现有调度设计。
+2. `dws_ext_td_ims_*` 中是否需要保留来源系统 `ims`。
+3. DWS 目标表的 catalog、字段类型、表属性和权限是否需要统一 DDL 模板。

docs/sql_refactoring_standard.md

@@ -2,16 +2,77 @@
 
 ## 1. 范围和原则
 
-本规范适用于 CHPA Databricks 数仓脚本。第一阶段目标是在不改变业务结果的前提下提高可维护性。
+本规范适用于 CHPA Databricks 数仓脚本。第一阶段目标是在不改变业务口径的前提下提高可维护性。
 
 - 先保持业务口径，再做性能优化。
 - 每个脚本只负责一个主要目标表。
-- 文件头声明源表、目标表、数据粒度、写入方式和依赖。
+- 文件头声明源表、目标表、数据粒度、写入方式和迁移关系。
 - 持久化写入必须显式列出目标列和查询列，禁止使用 `SELECT *`。
-- 逻辑重构与生产表改名分开实施。
-- 为行数、唯一性、非空约束和未匹配记录建立验证查询或任务检查。
+- 为行数、唯一性、非空约束和未匹配记录建立验证检查。
 
-## 2. 文件和目录命名
+## 2. 数仓分层
+
+数仓固定为三层，依赖方向为：
+
+```text
+DWD -> DWS -> DM
+```
+
+### DWD
+
+- 保存清洗、标准化后的原子明细和基础主数据。
+- 保持源数据可追溯性和稳定粒度。
+- 不承载面向报表的层级拉宽、跨主题指标聚合。
+
+### DWS
+
+- 引用 DWD 构建公共维度、层级宽表和可复用事实表。
+- 统一编码、公共口径和跨明细关联结果。
+- 本批 ATC/NFC 层级宽维表属于 DWS，不再写回 DWD。
+
+### DM
+
+- 引用 DWS 构建具体业务主题、指标和报表数据集。
+- 允许面向使用场景组织字段，但不得反向成为 DWD/DWS 的依赖。
+
+读取关系保持清晰：DWS 读取 DWD，DM 读取 DWS。DWD 不读取 DWS/DM，DWS 不读取 DM；DWS/DM 的派生结果不得写回 DWD，DM 的派生结果不得写回 DWS。
+
+## 3. 表命名
+
+表名由“层级 + ext + 对象类型 + 业务实体”组成：
+
+```text
+[<catalog>.]<schema>.<layer>_ext_<object_type>_<business_entity>
+```
+
+固定前缀如下：
+
+| 层级 | 对象 | 表名前缀 |
+| --- | --- | --- |
+| DWS | 维度/主数据 | `dws_ext_td_` |
+| DWS | 事实数据 | `dws_ext_tf_` |
+| DM | 维度/主数据 | `dm_ext_td_` |
+| DM | 事实数据 | `dm_ext_tf_` |
+
+命名规则：
+
+- `<schema>` 与 `<layer>` 保持一致，例如 `dws.dws_ext_td_xxx`。
+- `td` 表示维度或主数据，`tf` 表示事实数据。
+- `<business_entity>` 使用小写 snake_case。
+- 需要区分来源系统时，将来源放在业务实体开头，例如 `ims_atc_hierarchy`。
+- `atc`、`nfc`、`ims` 等已形成业务共识的缩写可以保留。
+
+首批及后续建议映射：
+
+| 旧表 | 新表 | 类型 |
+| --- | --- | --- |
+| `dwd.dwd_ims_atc_hierarchy` | `dws.dws_ext_td_ims_atc_hierarchy` | DWS 维度 |
+| `dwd.dwd_ims_nfc_hierarchy` | `dws.dws_ext_td_ims_nfc_hierarchy` | DWS 维度 |
+| `dwd.dwd_ims_td_manufacturer_corp` | `dws.dws_ext_td_ims_manufacturer_corporation` | DWS 维度，后续处理 |
+| `dwd.dwd_ims_td_pack_property` | `dws.dws_ext_td_ims_pack_property` | DWS 维度，后续处理 |
+| `dwd.dwd_gnd_pharbers_prov_fact` | `dws.dws_ext_tf_pharbers_province_sales` | DWS 事实，需先确认粒度 |
+
+## 4. 文件和目录命名
 
 目录结构统一为：
 
@@ -21,63 +82,31 @@ sql/<业务域>/<阶段>/<序号>_<目标表>.sql
 
 规则：
 
-- 目录和文件名使用小写 snake_case。
-- 路径和文件名不使用空格。
-- 使用两位序号明确 notebook/job 的执行顺序。
-- 一个脚本只写入一个主要持久化目标。像 `01 dwd_update.sql` 这种多目标脚本，应按目标表和职责拆分。
+- 目录和文件名使用小写 snake_case，不使用空格。
+- 目录阶段必须和输出层一致，例如 DWS 脚本放在 `02_dws`。
+- 使用两位序号明确 notebook/job 执行顺序。
+- 一个脚本只写入一个主要持久化目标。
+- 多目标脚本按目标表和职责拆分。
 
 示例：
 
 ```text
-sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql
+sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql
 ```
 
-## 3. 物理表命名
-
-新表使用以下格式：
-
-```text
-[<catalog>.]<分层_schema>.<来源系统>_<对象类型>_<业务实体>[_<限定词>]
-```
-
-未启用 Unity Catalog 或依赖默认 catalog 时可以省略 catalog。分层已经由 schema 表达，表名不再重复 `dwd`、`dws` 或 `dm`。
-
-- 来源系统：如 `ims`、`gnd`、`pharbers`。
-- 对象类型：`td` 表示维度/主数据，`tf` 表示事实数据。
-- 业务实体：使用稳定的业务名词，如 `atc_hierarchy`、`pack_property`。
-- 限定词：按需表达粒度或变体，如 `monthly`、`province`。
-
-`01` 组表名建议：
-
-| 现有物理表名 | 推荐规范表名 | 首批处理方式 |
-| --- | --- | --- |
-| `dwd.dwd_ims_atc_hierarchy` | `dwd.ims_td_atc_hierarchy` | 保留现名 |
-| `dwd.dwd_ims_nfc_hierarchy` | `dwd.ims_td_nfc_hierarchy` | 保留现名 |
-| `dwd.dwd_ims_td_manufacturer_corp` | `dwd.ims_td_manufacturer_corporation` | 后续 review |
-| `dwd.dwd_ims_td_pack_property` | `dwd.ims_td_pack_property` | 后续 review |
-| `dwd.dwd_gnd_pharbers_prov_fact` | `dwd.pharbers_tf_province_sales` | 先确认业务粒度 |
-
-推荐迁移顺序：
-
-1. 创建规范名称的新表，并用兼容视图暴露旧表名。
-2. 新旧表并行运行，对比数据结果。
-3. 在一次受控发布中修改全部下游引用。
-4. 所有消费者迁移后再移除旧名称兼容视图。
-
-## 4. 字段命名
+## 5. 字段命名
 
 - 新字段使用小写 snake_case。
 - 标识使用 `_id`，业务编码使用 `_code`，名称或描述使用 `_name`、`_description`。
 - 日期时间后缀按真实类型使用 `_date`、`_timestamp` 或 `_at`。
-- `atc`、`nfc`、`prod`、`pack` 等已形成业务共识或受输出契约约束的缩写可以保留。
-- 纯逻辑重构中不直接修改持久化输出字段名。
+- 纯逻辑重构中不直接修改已有输出字段契约。
 
-## 5. SQL 结构和格式
+## 6. SQL 结构和格式
 
 脚本统一按以下顺序组织：
 
 1. Databricks notebook 标记和脚本契约头。
-2. 带显式目标列的 `INSERT OVERWRITE`。
+2. 带显式目标列的写入语句。
 3. 源数据标准化 CTE。
 4. 业务规则 CTE。
 5. 按目标表字段顺序显式编写最终 `SELECT`。
@@ -86,27 +115,21 @@ sql/chpa/01_dwd/01_dwd_ims_atc_hierarchy.sql
 
 - SQL 关键字使用大写。
 - CTE 和别名使用小写 snake_case。
-- 使用四个空格缩进。
-- `SELECT` 每行一个字段。
-- 每个 JOIN 条件单独成行。
-- 使用 `atc_level_1` 等有业务意义的别名，维护代码中不使用 `t1`、`a` 等无语义别名。
+- 使用四个空格缩进，`SELECT` 每行一个字段。
+- 使用有业务意义的别名，不使用 `t1`、`a` 等无语义别名。
 - 同一作用域出现多个关系时，所有字段都带关系限定符。
 
-## 6. 写入和数据质量规则
+## 7. 写入和数据质量
 
 - 只有可确定性重跑的全量任务可以使用 `INSERT OVERWRITE`。
 - 不在一个脚本中混合无关的 `UPDATE` 和目标表构建逻辑。
-- 编码补零等标准化逻辑集中到一个明确的处理阶段，避免在多个任务中重复表达式。
+- 编码补零等标准化逻辑集中到一个明确阶段。
 - 每个脚本必须声明预期目标粒度。
-- 最少验证以下指标：
-  - 新结果与旧结果的总行数；
-  - 声明业务键的重复数；
-  - 必填键的空值数；
-  - 层级关联新增的未匹配记录数。
+- 最少验证新旧行数、双向差集、业务键重复、必填键空值和层级未匹配数。
 
-## 7. 首批兼容说明
+## 8. 首批兼容说明
 
-ATC 和 NFC 脚本原样保留以下父级编码规则：
+ATC 和 NFC 脚本保留旧逻辑中的父级编码规则：
 
 - ATC 2 级关联 1 级：取前 1 位。
 - ATC 3 级关联 2 级：取前 3 位。
@@ -114,6 +137,4 @@ ATC 和 NFC 脚本原样保留以下父级编码规则：
 - NFC 2 级关联 1 级：取前 1 位。
 - NFC 3 级关联 2 级：取前 2 位。
 
-任何口径简化都应先用生产样本验证这些规则。
-
-首批上线前运行 `validation/chpa/01_dwd/validate_hierarchy_refactor.sql`。新旧双向差集必须为 0，行数必须一致；重复路径和未匹配层级作为业务 review 指标记录，不在未确认阈值前自动判定失败。
+两个 DWS 脚本执行后运行 `validation/chpa/02_dws/validate_hierarchy_refactor.sql`。新 DWS 输出与旧 DWD 基线的行数必须一致，双向差集必须为 0；重复路径和未匹配层级作为业务 review 指标记录。

sql/chpa/02_dws/01_dws_ext_td_ims_atc_hierarchy.sql

@@ -2,14 +2,15 @@
 -- =============================================================================
 -- Purpose    : Build the flattened IMS ATC level 1-4 hierarchy.
 -- Source     : dwd.dwd_ims_td_therapeutic_class
--- Target     : dwd.dwd_ims_atc_hierarchy
+-- Target     : dws.dws_ext_td_ims_atc_hierarchy
 -- Grain      : One row per hierarchy path rooted at ATC1_CODE; lower levels may be null.
 -- Write mode : Full refresh (INSERT OVERWRITE).
--- Downstream : dwd.dwd_ims_td_pack_property, dws.dws_ims_td_atc_cn
+-- Replaces   : dwd.dwd_ims_atc_hierarchy (legacy output).
+-- Consumers  : Pack-property and ATC-CN jobs must migrate to the DWS target.
 -- Notes      : Parent-code matching is preserved from the legacy script.
 -- =============================================================================
 
-INSERT OVERWRITE TABLE dwd.dwd_ims_atc_hierarchy (
+INSERT OVERWRITE TABLE dws.dws_ext_td_ims_atc_hierarchy (
     ATC1_ID,
     ATC1_CODE,
     ATC1_DES,

sql/chpa/02_dws/02_dws_ext_td_ims_nfc_hierarchy.sql

@@ -2,14 +2,15 @@
 -- =============================================================================
 -- Purpose    : Build the flattened IMS NFC level 1-3 hierarchy.
 -- Source     : dwd.dwd_ims_td_new_form_class
--- Target     : dwd.dwd_ims_nfc_hierarchy
+-- Target     : dws.dws_ext_td_ims_nfc_hierarchy
 -- Grain      : One row per hierarchy path rooted at NFC1_CODE; lower levels may be null.
 -- Write mode : Full refresh (INSERT OVERWRITE).
--- Downstream : dwd.dwd_ims_td_pack_property, dws.dws_ims_td_nfc_cn
+-- Replaces   : dwd.dwd_ims_nfc_hierarchy (legacy output).
+-- Consumers  : Pack-property and NFC-CN jobs must migrate to the DWS target.
 -- Notes      : Parent-code matching is preserved from the legacy script.
 -- =============================================================================
 
-INSERT OVERWRITE TABLE dwd.dwd_ims_nfc_hierarchy (
+INSERT OVERWRITE TABLE dws.dws_ext_td_ims_nfc_hierarchy (
     NFC1_ID,
     NFC1_CODE,
     NFC1_DES,

validation/chpa/01_dwd/validate_hierarchy_refactor.sql

@@ -1,396 +0,0 @@
--- Databricks notebook source
--- MAGIC %md
--- MAGIC # ATC/NFC hierarchy refactor validation
--- MAGIC Run this notebook before executing the refactored overwrite scripts.
--- MAGIC The legacy targets and source tables must represent the same data snapshot.
-
--- COMMAND ----------
-
--- Confirm the ATC target contains the expected 12 non-partition output columns.
-DESCRIBE TABLE dwd.dwd_ims_atc_hierarchy;
-
--- COMMAND ----------
-
--- Confirm the NFC target contains the expected 9 non-partition output columns.
-DESCRIBE TABLE dwd.dwd_ims_nfc_hierarchy;
-
--- COMMAND ----------
-
-CREATE OR REPLACE TEMP VIEW validation_atc_hierarchy_candidate AS
-WITH therapeutic_class AS (
-    SELECT
-        therapeutic_id,
-        therapeutic_code,
-        therapeutic_name,
-        therapeutic_level
-    FROM dwd.dwd_ims_td_therapeutic_class
-),
-atc_level_1 AS (
-    SELECT
-        therapeutic_id AS atc1_id,
-        therapeutic_code AS atc1_code,
-        therapeutic_name AS atc1_des
-    FROM therapeutic_class
-    WHERE therapeutic_level = '1'
-),
-atc_level_2 AS (
-    SELECT
-        therapeutic_id AS atc2_id,
-        therapeutic_code AS atc2_code,
-        therapeutic_name AS atc2_des
-    FROM therapeutic_class
-    WHERE therapeutic_level = '2'
-),
-atc_level_3 AS (
-    SELECT
-        therapeutic_id AS atc3_id,
-        therapeutic_code AS atc3_code,
-        therapeutic_name AS atc3_des
-    FROM therapeutic_class
-    WHERE therapeutic_level = '3'
-),
-atc_level_4 AS (
-    SELECT
-        therapeutic_id AS atc4_id,
-        therapeutic_code AS atc4_code,
-        therapeutic_name AS atc4_des
-    FROM therapeutic_class
-    WHERE therapeutic_level = '4'
-)
-SELECT
-    atc_level_1.atc1_id AS ATC1_ID,
-    atc_level_1.atc1_code AS ATC1_CODE,
-    atc_level_1.atc1_des AS ATC1_DES,
-    atc_level_2.atc2_id AS ATC2_ID,
-    atc_level_2.atc2_code AS ATC2_CODE,
-    atc_level_2.atc2_des AS ATC2_DES,
-    atc_level_3.atc3_id AS ATC3_ID,
-    atc_level_3.atc3_code AS ATC3_CODE,
-    atc_level_3.atc3_des AS ATC3_DES,
-    atc_level_4.atc4_id AS ATC4_ID,
-    atc_level_4.atc4_code AS ATC4_CODE,
-    atc_level_4.atc4_des AS ATC4_DES
-FROM atc_level_1
-LEFT JOIN atc_level_2
-    ON atc_level_1.atc1_code = LEFT(atc_level_2.atc2_code, 1)
-LEFT JOIN atc_level_3
-    ON atc_level_2.atc2_code = LEFT(atc_level_3.atc3_code, 3)
-LEFT JOIN atc_level_4
-    ON atc_level_3.atc3_code = LEFT(atc_level_4.atc4_code, 4)
-;
-
--- COMMAND ----------
-
-CREATE OR REPLACE TEMP VIEW validation_nfc_hierarchy_candidate AS
-WITH new_form_class AS (
-    SELECT
-        newformclass_id,
-        newformclass_code,
-        newformclass_name,
-        newformclass_level
-    FROM dwd.dwd_ims_td_new_form_class
-),
-nfc_level_1 AS (
-    SELECT
-        newformclass_id AS nfc1_id,
-        newformclass_code AS nfc1_code,
-        newformclass_name AS nfc1_des
-    FROM new_form_class
-    WHERE newformclass_level = '1'
-),
-nfc_level_2 AS (
-    SELECT
-        newformclass_id AS nfc2_id,
-        newformclass_code AS nfc2_code,
-        newformclass_name AS nfc2_des
-    FROM new_form_class
-    WHERE newformclass_level = '2'
-),
-nfc_level_3 AS (
-    SELECT
-        newformclass_id AS nfc3_id,
-        newformclass_code AS nfc3_code,
-        newformclass_name AS nfc3_des
-    FROM new_form_class
-    WHERE newformclass_level = '3'
-)
-SELECT
-    nfc_level_1.nfc1_id AS NFC1_ID,
-    nfc_level_1.nfc1_code AS NFC1_CODE,
-    nfc_level_1.nfc1_des AS NFC1_DES,
-    nfc_level_2.nfc2_id AS NFC2_ID,
-    nfc_level_2.nfc2_code AS NFC2_CODE,
-    nfc_level_2.nfc2_des AS NFC2_DES,
-    nfc_level_3.nfc3_id AS NFC3_ID,
-    nfc_level_3.nfc3_code AS NFC3_CODE,
-    nfc_level_3.nfc3_des AS NFC3_DES
-FROM nfc_level_1
-LEFT JOIN nfc_level_2
-    ON nfc_level_1.nfc1_code = LEFT(nfc_level_2.nfc2_code, 1)
-LEFT JOIN nfc_level_3
-    ON nfc_level_2.nfc2_code = LEFT(nfc_level_3.nfc3_code, 2)
-;
-
--- COMMAND ----------
-
--- Every check must return passed = true before replacing the scheduled jobs.
-CREATE OR REPLACE TEMP VIEW validation_hierarchy_compatibility_checks AS
-WITH compatibility_checks AS (
-    SELECT
-        'atc_row_count' AS check_name,
-        (SELECT COUNT(*) FROM validation_atc_hierarchy_candidate) AS actual_value,
-        (SELECT COUNT(*) FROM dwd.dwd_ims_atc_hierarchy) AS expected_value
-    UNION ALL
-    SELECT
-        'atc_legacy_minus_candidate' AS check_name,
-        (
-            SELECT COUNT(*)
-            FROM (
-                SELECT
-                    ATC1_ID,
-                    ATC1_CODE,
-                    ATC1_DES,
-                    ATC2_ID,
-                    ATC2_CODE,
-                    ATC2_DES,
-                    ATC3_ID,
-                    ATC3_CODE,
-                    ATC3_DES,
-                    ATC4_ID,
-                    ATC4_CODE,
-                    ATC4_DES
-                FROM dwd.dwd_ims_atc_hierarchy
-                EXCEPT ALL
-                SELECT
-                    ATC1_ID,
-                    ATC1_CODE,
-                    ATC1_DES,
-                    ATC2_ID,
-                    ATC2_CODE,
-                    ATC2_DES,
-                    ATC3_ID,
-                    ATC3_CODE,
-                    ATC3_DES,
-                    ATC4_ID,
-                    ATC4_CODE,
-                    ATC4_DES
-                FROM validation_atc_hierarchy_candidate
-            ) AS differences
-        ) AS actual_value,
-        0 AS expected_value
-    UNION ALL
-    SELECT
-        'atc_candidate_minus_legacy' AS check_name,
-        (
-            SELECT COUNT(*)
-            FROM (
-                SELECT
-                    ATC1_ID,
-                    ATC1_CODE,
-                    ATC1_DES,
-                    ATC2_ID,
-                    ATC2_CODE,
-                    ATC2_DES,
-                    ATC3_ID,
-                    ATC3_CODE,
-                    ATC3_DES,
-                    ATC4_ID,
-                    ATC4_CODE,
-                    ATC4_DES
-                FROM validation_atc_hierarchy_candidate
-                EXCEPT ALL
-                SELECT
-                    ATC1_ID,
-                    ATC1_CODE,
-                    ATC1_DES,
-                    ATC2_ID,
-                    ATC2_CODE,
-                    ATC2_DES,
-                    ATC3_ID,
-                    ATC3_CODE,
-                    ATC3_DES,
-                    ATC4_ID,
-                    ATC4_CODE,
-                    ATC4_DES
-                FROM dwd.dwd_ims_atc_hierarchy
-            ) AS differences
-        ) AS actual_value,
-        0 AS expected_value
-    UNION ALL
-    SELECT
-        'nfc_row_count' AS check_name,
-        (SELECT COUNT(*) FROM validation_nfc_hierarchy_candidate) AS actual_value,
-        (SELECT COUNT(*) FROM dwd.dwd_ims_nfc_hierarchy) AS expected_value
-    UNION ALL
-    SELECT
-        'nfc_legacy_minus_candidate' AS check_name,
-        (
-            SELECT COUNT(*)
-            FROM (
-                SELECT
-                    NFC1_ID,
-                    NFC1_CODE,
-                    NFC1_DES,
-                    NFC2_ID,
-                    NFC2_CODE,
-                    NFC2_DES,
-                    NFC3_ID,
-                    NFC3_CODE,
-                    NFC3_DES
-                FROM dwd.dwd_ims_nfc_hierarchy
-                EXCEPT ALL
-                SELECT
-                    NFC1_ID,
-                    NFC1_CODE,
-                    NFC1_DES,
-                    NFC2_ID,
-                    NFC2_CODE,
-                    NFC2_DES,
-                    NFC3_ID,
-                    NFC3_CODE,
-                    NFC3_DES
-                FROM validation_nfc_hierarchy_candidate
-            ) AS differences
-        ) AS actual_value,
-        0 AS expected_value
-    UNION ALL
-    SELECT
-        'nfc_candidate_minus_legacy' AS check_name,
-        (
-            SELECT COUNT(*)
-            FROM (
-                SELECT
-                    NFC1_ID,
-                    NFC1_CODE,
-                    NFC1_DES,
-                    NFC2_ID,
-                    NFC2_CODE,
-                    NFC2_DES,
-                    NFC3_ID,
-                    NFC3_CODE,
-                    NFC3_DES
-                FROM validation_nfc_hierarchy_candidate
-                EXCEPT ALL
-                SELECT
-                    NFC1_ID,
-                    NFC1_CODE,
-                    NFC1_DES,
-                    NFC2_ID,
-                    NFC2_CODE,
-                    NFC2_DES,
-                    NFC3_ID,
-                    NFC3_CODE,
-                    NFC3_DES
-                FROM dwd.dwd_ims_nfc_hierarchy
-            ) AS differences
-        ) AS actual_value,
-        0 AS expected_value
-    UNION ALL
-    SELECT
-        'except_all_null_semantics' AS check_name,
-        (
-            SELECT COUNT(*)
-            FROM (
-                SELECT CAST(NULL AS STRING) AS nullable_value
-                EXCEPT ALL
-                SELECT CAST(NULL AS STRING) AS nullable_value
-            ) AS differences
-        ) AS actual_value,
-        0 AS expected_value
-)
-SELECT
-    check_name,
-    actual_value,
-    expected_value,
-    actual_value = expected_value AS passed
-FROM compatibility_checks
-;
-
--- COMMAND ----------
-
-SELECT
-    check_name,
-    actual_value,
-    expected_value,
-    passed
-FROM validation_hierarchy_compatibility_checks
-ORDER BY check_name;
-
--- COMMAND ----------
-
--- Profile duplicates and unmatched levels. These are review metrics, not automatic failures.
-WITH atc_duplicate_paths AS (
-    SELECT COUNT(*) AS path_count
-    FROM validation_atc_hierarchy_candidate
-    GROUP BY ATC1_CODE, ATC2_CODE, ATC3_CODE, ATC4_CODE
-    HAVING COUNT(*) > 1
-),
-nfc_duplicate_paths AS (
-    SELECT COUNT(*) AS path_count
-    FROM validation_nfc_hierarchy_candidate
-    GROUP BY NFC1_CODE, NFC2_CODE, NFC3_CODE
-    HAVING COUNT(*) > 1
-)
-SELECT
-    'atc_duplicate_path_rows' AS metric_name,
-    COALESCE(SUM(path_count - 1), 0) AS metric_value
-FROM atc_duplicate_paths
-UNION ALL
-SELECT
-    'atc_source_level_1_required_key_null' AS metric_name,
-    COUNT_IF(therapeutic_id IS NULL OR therapeutic_code IS NULL) AS metric_value
-FROM dwd.dwd_ims_td_therapeutic_class
-WHERE therapeutic_level = '1'
-UNION ALL
-SELECT
-    'atc_unmatched_level_2' AS metric_name,
-    COUNT_IF(ATC2_CODE IS NULL) AS metric_value
-FROM validation_atc_hierarchy_candidate
-UNION ALL
-SELECT
-    'atc_unmatched_level_3' AS metric_name,
-    COUNT_IF(ATC3_CODE IS NULL) AS metric_value
-FROM validation_atc_hierarchy_candidate
-UNION ALL
-SELECT
-    'atc_unmatched_level_4' AS metric_name,
-    COUNT_IF(ATC4_CODE IS NULL) AS metric_value
-FROM validation_atc_hierarchy_candidate
-UNION ALL
-SELECT
-    'nfc_duplicate_path_rows' AS metric_name,
-    COALESCE(SUM(path_count - 1), 0) AS metric_value
-FROM nfc_duplicate_paths
-UNION ALL
-SELECT
-    'nfc_source_level_1_required_key_null' AS metric_name,
-    COUNT_IF(newformclass_id IS NULL OR newformclass_code IS NULL) AS metric_value
-FROM dwd.dwd_ims_td_new_form_class
-WHERE newformclass_level = '1'
-UNION ALL
-SELECT
-    'nfc_unmatched_level_2' AS metric_name,
-    COUNT_IF(NFC2_CODE IS NULL) AS metric_value
-FROM validation_nfc_hierarchy_candidate
-UNION ALL
-SELECT
-    'nfc_unmatched_level_3' AS metric_name,
-    COUNT_IF(NFC3_CODE IS NULL) AS metric_value
-FROM validation_nfc_hierarchy_candidate
-ORDER BY metric_name;
-
--- COMMAND ----------
-
--- Abort the notebook when any compatibility check fails.
-SELECT IF(
-    COUNT_IF(NOT passed) = 0,
-    'All hierarchy compatibility checks passed.',
-    raise_error(
-        CONCAT(
-            'Hierarchy compatibility validation failed: ',
-            CAST(COUNT_IF(NOT passed) AS STRING),
-            ' check(s) did not pass.'
-        )
-    )
-) AS validation_result
-FROM validation_hierarchy_compatibility_checks;

validation/chpa/02_dws/validate_hierarchy_refactor.sql

@@ -0,0 +1,250 @@
+-- Databricks notebook source
+-- MAGIC %md
+-- MAGIC # ATC/NFC DWS hierarchy validation
+-- MAGIC Run this notebook after both DWS hierarchy scripts finish.
+-- MAGIC The DWS outputs and legacy DWD baselines must represent the same source snapshot.
+
+-- COMMAND ----------
+
+DESCRIBE TABLE dws.dws_ext_td_ims_atc_hierarchy;
+
+-- COMMAND ----------
+
+DESCRIBE TABLE dws.dws_ext_td_ims_nfc_hierarchy;
+
+-- COMMAND ----------
+
+-- Fix the comparison column set and order at the view boundary.
+CREATE OR REPLACE TEMP VIEW validation_atc_legacy AS
+SELECT
+    ATC1_ID,
+    ATC1_CODE,
+    ATC1_DES,
+    ATC2_ID,
+    ATC2_CODE,
+    ATC2_DES,
+    ATC3_ID,
+    ATC3_CODE,
+    ATC3_DES,
+    ATC4_ID,
+    ATC4_CODE,
+    ATC4_DES
+FROM dwd.dwd_ims_atc_hierarchy;
+
+-- COMMAND ----------
+
+CREATE OR REPLACE TEMP VIEW validation_atc_dws AS
+SELECT
+    ATC1_ID,
+    ATC1_CODE,
+    ATC1_DES,
+    ATC2_ID,
+    ATC2_CODE,
+    ATC2_DES,
+    ATC3_ID,
+    ATC3_CODE,
+    ATC3_DES,
+    ATC4_ID,
+    ATC4_CODE,
+    ATC4_DES
+FROM dws.dws_ext_td_ims_atc_hierarchy;
+
+-- COMMAND ----------
+
+CREATE OR REPLACE TEMP VIEW validation_nfc_legacy AS
+SELECT
+    NFC1_ID,
+    NFC1_CODE,
+    NFC1_DES,
+    NFC2_ID,
+    NFC2_CODE,
+    NFC2_DES,
+    NFC3_ID,
+    NFC3_CODE,
+    NFC3_DES
+FROM dwd.dwd_ims_nfc_hierarchy;
+
+-- COMMAND ----------
+
+CREATE OR REPLACE TEMP VIEW validation_nfc_dws AS
+SELECT
+    NFC1_ID,
+    NFC1_CODE,
+    NFC1_DES,
+    NFC2_ID,
+    NFC2_CODE,
+    NFC2_DES,
+    NFC3_ID,
+    NFC3_CODE,
+    NFC3_DES
+FROM dws.dws_ext_td_ims_nfc_hierarchy;
+
+-- COMMAND ----------
+
+CREATE OR REPLACE TEMP VIEW validation_hierarchy_compatibility_checks AS
+WITH compatibility_checks AS (
+    SELECT
+        'atc_row_count' AS check_name,
+        (SELECT COUNT(*) FROM validation_atc_dws) AS actual_value,
+        (SELECT COUNT(*) FROM validation_atc_legacy) AS expected_value
+    UNION ALL
+    SELECT
+        'atc_legacy_minus_dws' AS check_name,
+        (
+            SELECT COUNT(*)
+            FROM (
+                SELECT * FROM validation_atc_legacy
+                EXCEPT ALL
+                SELECT * FROM validation_atc_dws
+            ) AS differences
+        ) AS actual_value,
+        0 AS expected_value
+    UNION ALL
+    SELECT
+        'atc_dws_minus_legacy' AS check_name,
+        (
+            SELECT COUNT(*)
+            FROM (
+                SELECT * FROM validation_atc_dws
+                EXCEPT ALL
+                SELECT * FROM validation_atc_legacy
+            ) AS differences
+        ) AS actual_value,
+        0 AS expected_value
+    UNION ALL
+    SELECT
+        'nfc_row_count' AS check_name,
+        (SELECT COUNT(*) FROM validation_nfc_dws) AS actual_value,
+        (SELECT COUNT(*) FROM validation_nfc_legacy) AS expected_value
+    UNION ALL
+    SELECT
+        'nfc_legacy_minus_dws' AS check_name,
+        (
+            SELECT COUNT(*)
+            FROM (
+                SELECT * FROM validation_nfc_legacy
+                EXCEPT ALL
+                SELECT * FROM validation_nfc_dws
+            ) AS differences
+        ) AS actual_value,
+        0 AS expected_value
+    UNION ALL
+    SELECT
+        'nfc_dws_minus_legacy' AS check_name,
+        (
+            SELECT COUNT(*)
+            FROM (
+                SELECT * FROM validation_nfc_dws
+                EXCEPT ALL
+                SELECT * FROM validation_nfc_legacy
+            ) AS differences
+        ) AS actual_value,
+        0 AS expected_value
+    UNION ALL
+    SELECT
+        'except_all_null_semantics' AS check_name,
+        (
+            SELECT COUNT(*)
+            FROM (
+                SELECT CAST(NULL AS STRING) AS nullable_value
+                EXCEPT ALL
+                SELECT CAST(NULL AS STRING) AS nullable_value
+            ) AS differences
+        ) AS actual_value,
+        0 AS expected_value
+)
+SELECT
+    check_name,
+    actual_value,
+    expected_value,
+    actual_value = expected_value AS passed
+FROM compatibility_checks;
+
+-- COMMAND ----------
+
+SELECT
+    check_name,
+    actual_value,
+    expected_value,
+    passed
+FROM validation_hierarchy_compatibility_checks
+ORDER BY check_name;
+
+-- COMMAND ----------
+
+-- These quality metrics require business review; they do not fail automatically.
+WITH atc_duplicate_paths AS (
+    SELECT COUNT(*) AS path_count
+    FROM validation_atc_dws
+    GROUP BY ATC1_CODE, ATC2_CODE, ATC3_CODE, ATC4_CODE
+    HAVING COUNT(*) > 1
+),
+nfc_duplicate_paths AS (
+    SELECT COUNT(*) AS path_count
+    FROM validation_nfc_dws
+    GROUP BY NFC1_CODE, NFC2_CODE, NFC3_CODE
+    HAVING COUNT(*) > 1
+)
+SELECT
+    'atc_duplicate_path_rows' AS metric_name,
+    COALESCE(SUM(path_count - 1), 0) AS metric_value
+FROM atc_duplicate_paths
+UNION ALL
+SELECT
+    'atc_source_level_1_required_key_null' AS metric_name,
+    COUNT_IF(therapeutic_id IS NULL OR therapeutic_code IS NULL) AS metric_value
+FROM dwd.dwd_ims_td_therapeutic_class
+WHERE therapeutic_level = '1'
+UNION ALL
+SELECT
+    'atc_unmatched_level_2' AS metric_name,
+    COUNT_IF(ATC2_CODE IS NULL) AS metric_value
+FROM validation_atc_dws
+UNION ALL
+SELECT
+    'atc_unmatched_level_3' AS metric_name,
+    COUNT_IF(ATC3_CODE IS NULL) AS metric_value
+FROM validation_atc_dws
+UNION ALL
+SELECT
+    'atc_unmatched_level_4' AS metric_name,
+    COUNT_IF(ATC4_CODE IS NULL) AS metric_value
+FROM validation_atc_dws
+UNION ALL
+SELECT
+    'nfc_duplicate_path_rows' AS metric_name,
+    COALESCE(SUM(path_count - 1), 0) AS metric_value
+FROM nfc_duplicate_paths
+UNION ALL
+SELECT
+    'nfc_source_level_1_required_key_null' AS metric_name,
+    COUNT_IF(newformclass_id IS NULL OR newformclass_code IS NULL) AS metric_value
+FROM dwd.dwd_ims_td_new_form_class
+WHERE newformclass_level = '1'
+UNION ALL
+SELECT
+    'nfc_unmatched_level_2' AS metric_name,
+    COUNT_IF(NFC2_CODE IS NULL) AS metric_value
+FROM validation_nfc_dws
+UNION ALL
+SELECT
+    'nfc_unmatched_level_3' AS metric_name,
+    COUNT_IF(NFC3_CODE IS NULL) AS metric_value
+FROM validation_nfc_dws
+ORDER BY metric_name;
+
+-- COMMAND ----------
+
+-- Block downstream migration when any DWS-to-DWD compatibility check fails.
+SELECT IF(
+    COUNT_IF(NOT passed) = 0,
+    'All DWS hierarchy compatibility checks passed.',
+    raise_error(
+        CONCAT(
+            'DWS hierarchy compatibility validation failed: ',
+            CAST(COUNT_IF(NOT passed) AS STRING),
+            ' check(s) did not pass.'
+        )
+    )
+) AS validation_result
+FROM validation_hierarchy_compatibility_checks;