REFACTOR-MA/AGENTS.md

AGENTS.md

Repository purpose

This repository contains the maintainable Databricks SQL replacement for the legacy CHPA warehouse jobs.

• Treat the workspace-level CHPA/ directory as read-only legacy input.
• Put all rewritten code, documentation, and validation under RE/.
• Preserve business behavior first. Separate logic refactoring, physical renaming, and business-rule changes.
• Keep each persisted target in one clearly owned script whenever practical.

Warehouse layers

The warehouse has exactly three persistent layers and data flows in one direction:

DWD -> DWS -> DM

DWD

• Cleaned atomic details and foundational master data.
• Retains source traceability and stable source grain.
• Does not contain report-oriented hierarchy flattening or subject metrics.

DWS

• Reusable dimensions, hierarchy-wide tables, and reusable facts built from DWD.
• Standardizes shared codes, joins, and business definitions.
• A job that reads DWD and produces a reusable derived table writes to DWS.

DM

• Subject-area datasets, metrics, and report-facing outputs built from DWS.
• Must not become an upstream dependency of DWD or DWS.

DWD must not read DWS/DM. DWS must not read DM. DWS/DM derived results must not be written back to DWD, and DM derived results must not be written back to DWS.

Table naming

Use the schema that matches the target layer and one of these fixed table prefixes:

Layer	Object type	Required form
DWS	Dimension/master data	dws.dws_ext_td_<business_entity>
DWS	Fact data	dws.dws_ext_tf_<business_entity>
DM	Dimension/master data	dm.dm_ext_td_<business_entity>
DM	Fact data	dm.dm_ext_tf_<business_entity>

Naming rules:

• Use lowercase snake_case table names.
• td means dimension/master data; tf means fact data.
• Include the source/domain at the start of the business entity when it avoids ambiguity, for example ims_atc_hierarchy.
• Do not write a newly derived DWS table with a dwd_ name or into the dwd schema.
• Keep legacy output column names during a logic-only migration. Rename columns only through an explicit coordinated contract change.

Current hierarchy targets:

• dws.dws_ext_td_ims_atc_hierarchy
• dws.dws_ext_td_ims_nfc_hierarchy

File layout and order

Use:

sql/<domain>/<stage>/<sequence>_<target_table>.sql
validation/<domain>/<stage>/<validation_name>.sql

Rules:

• DWD jobs belong in 01_dwd, DWS jobs in 02_dws, and DM jobs in 03_dm.
• Use two-digit sequence numbers to make execution order explicit.
• Use lowercase snake_case paths with no spaces.
• Keep Databricks notebook markers and split executable statements with -- COMMAND ----------.
• Split legacy multi-target scripts by target or responsibility when doing so does not alter transactional behavior.

SQL contracts

Every persisted build script must include a concise header with:

• purpose;
• source tables;
• target table;
• output grain;
• write mode;
• legacy table replaced;
• known consumers or migration dependency.

SQL rules:

• Use uppercase SQL keywords and descriptive lowercase snake_case CTE aliases.
• Use four-space indentation and one selected column per line.
• Use explicit target columns and explicit final projection columns.
• Never use SELECT * in a persisted write.
• Qualify columns whenever multiple relations are in scope.
• Replace post-write null cleanup with projection expressions such as COALESCE when semantics are unchanged.
• Replace update-driven staging with ordered CTEs only after proving the result is equivalent.
• Keep one-time target DDL in a commented, separate Databricks cell above the write.
• Prefer CREATE TABLE IF NOT EXISTS <new_table> LIKE <verified_legacy_table> when the legacy schema is the authoritative type contract.
• For a stateful rolling/incremental target, seed retained state with an approved clone or backfill before the first run; an empty LIKE table is not sufficient.
• When a legacy SELECT * contract is unknown, define and document a deliberate narrowed contract with zero-row CTAS type inference instead of creating unknown columns that remain NULL.
• Do not guess source field types when no DDL or DESCRIBE output is available.

Validation

Before migrating downstream consumers:

• compare old and new row counts on the same source snapshot;
• run EXCEPT ALL in both directions with a fixed column order;
• profile duplicate business keys or paths;
• count null required keys;
• count unmatched hierarchy/join levels;
• make compatibility checks fail the notebook with raise_error() when any required check fails.

Keep old physical tables as validation baselines until downstream migration is complete. Do not add new downstream dependencies to legacy derived tables.

Git workflow

Repository:

https://git.chenwuzhu.cn/chenwu/REFACTOR-MA.git

Commit identity for this checkout:

chenwu <zhuchenwu@chenwuzhu.cn>

Workflow rules:

1. Work on the local main branch unless the user requests a review branch.
2. Check git status before editing and never discard unrelated user changes.
3. Run whitespace/static checks and inspect staged diffs before committing.
4. Use focused imperative commit messages that describe the warehouse change.
5. Push to origin/main only after the requested batch is complete and locally verified.
6. Never force-push or rewrite shared history without explicit approval.
7. Report the pushed commit hash and any Databricks validation that could not be executed locally.

Recent baseline commits:

• 3f3dc44: initial hierarchy SQL refactor.
• 33e60d0: move hierarchy outputs to DWS naming.
• b0d5f81: add commented one-time DDL templates.