REFACTOR-MA/AGENTS.md

# AGENTS.md

## Repository purpose

This repository contains the maintainable Databricks SQL replacement for the legacy CHPA warehouse jobs.

- Treat the workspace-level `CHPA/` directory as read-only legacy input.
- Put all rewritten code, documentation, and validation under `RE/`.
- Preserve business behavior first. Separate logic refactoring, physical renaming, and business-rule changes.
- Keep each persisted target in one clearly owned script whenever practical.

## Warehouse layers

The warehouse has exactly three persistent layers and data flows in one direction:

```text
DWD -> DWS -> DM
```

### DWD

- Cleaned atomic details and foundational master data.
- Retains source traceability and stable source grain.
- Does not contain report-oriented hierarchy flattening or subject metrics.

### DWS

- Reusable dimensions, hierarchy-wide tables, and reusable facts built from DWD.
- Standardizes shared codes, joins, and business definitions.
- A job that reads DWD and produces a reusable derived table writes to DWS.

### DM

- Subject-area datasets, metrics, and report-facing outputs built from DWS.
- Must not become an upstream dependency of DWD or DWS.

DWD must not read DWS/DM. DWS must not read DM. DWS/DM derived results must not be written back to DWD, and DM derived results must not be written back to DWS.

## Table naming

Use the schema that matches the target layer and one of these fixed table prefixes:

| Layer | Object type | Required form |
| --- | --- | --- |
| DWS | Dimension/master data | `dws.dws_ext_td_<business_entity>` |
| DWS | Fact data | `dws.dws_ext_tf_<business_entity>` |
| DM | Dimension/master data | `dm.dm_ext_td_<business_entity>` |
| DM | Fact data | `dm.dm_ext_tf_<business_entity>` |

Naming rules:

- Use lowercase snake_case table names.
- `td` means dimension/master data; `tf` means fact data.
- Include the source/domain at the start of the business entity when it avoids ambiguity, for example `ims_atc_hierarchy`.
- Do not write a newly derived DWS table with a `dwd_` name or into the `dwd` schema.
- Keep legacy output column names during a logic-only migration. Rename columns only through an explicit coordinated contract change.

Current hierarchy targets:

- `dws.dws_ext_td_ims_atc_hierarchy`
- `dws.dws_ext_td_ims_nfc_hierarchy`

## File layout and order

Use:

```text
sql/<domain>/<stage>/<sequence>_<target_table>.sql
validation/<domain>/<stage>/<validation_name>.sql
```

Rules:

- DWD jobs belong in `01_dwd`, DWS jobs in `02_dws`, and DM jobs in `03_dm`.
- Use two-digit sequence numbers to make execution order explicit.
- Use lowercase snake_case paths with no spaces.
- Keep Databricks notebook markers and split executable statements with `-- COMMAND ----------`.
- Split legacy multi-target scripts by target or responsibility when doing so does not alter transactional behavior.

## SQL contracts

Every persisted build script must include a concise header with:

- purpose;
- source tables;
- target table;
- output grain;
- write mode;
- legacy table replaced;
- known consumers or migration dependency.

SQL rules:

- Use uppercase SQL keywords and descriptive lowercase snake_case CTE aliases.
- Use four-space indentation and one selected column per line.
- Use explicit target columns and explicit final projection columns.
- Never use `SELECT *` in a persisted write.
- Qualify columns whenever multiple relations are in scope.
- Replace post-write null cleanup with projection expressions such as `COALESCE` when semantics are unchanged.
- Replace update-driven staging with ordered CTEs only after proving the result is equivalent.
- Keep one-time target DDL in a commented, separate Databricks cell above the write.
- Prefer `CREATE TABLE IF NOT EXISTS <new_table> LIKE <verified_legacy_table>` when the legacy schema is the authoritative type contract.
- For a stateful rolling/incremental target, seed retained state with an approved clone or backfill before the first run; an empty `LIKE` table is not sufficient.
- When a legacy `SELECT *` contract is unknown, define and document a deliberate narrowed contract with zero-row CTAS type inference instead of creating unknown columns that remain NULL.
- Do not guess source field types when no DDL or `DESCRIBE` output is available.

## Validation

Before migrating downstream consumers:

- compare old and new row counts on the same source snapshot;
- run `EXCEPT ALL` in both directions with a fixed column order;
- profile duplicate business keys or paths;
- count null required keys;
- count unmatched hierarchy/join levels;
- make compatibility checks fail the notebook with `raise_error()` when any required check fails.

Keep old physical tables as validation baselines until downstream migration is complete. Do not add new downstream dependencies to legacy derived tables.

## Git workflow

Repository:

```text
https://git.chenwuzhu.cn/chenwu/REFACTOR-MA.git
```

Commit identity for this checkout:

```text
chenwu <zhuchenwu@chenwuzhu.cn>
```

Workflow rules:

1. Work on the local `main` branch unless the user requests a review branch.
2. Check `git status` before editing and never discard unrelated user changes.
3. Run whitespace/static checks and inspect staged diffs before committing.
4. Use focused imperative commit messages that describe the warehouse change.
5. Push to `origin/main` only after the requested batch is complete and locally verified.
6. Never force-push or rewrite shared history without explicit approval.
7. Report the pushed commit hash and any Databricks validation that could not be executed locally.

Recent baseline commits:

- `3f3dc44`: initial hierarchy SQL refactor.
- `33e60d0`: move hierarchy outputs to DWS naming.
- `b0d5f81`: add commented one-time DDL templates.