Validate and Enhance Snapshot Data Pipeline Behavior

[sambodeme]

Description:

Ensure the pipeline in fac-snapshot-db is stateless and reproducible, functioning as intended when regenerated nightly via a GitHub Action. Include checks to create necessary tables if they do not already exist. Validate the pipeline’s overall behavior and output integrity.

Tasks:

• Table Checks: Ensure the pipeline includes checks to create required tables in fac-snapshot-db only if they do not exist.
• Set Up Nightly Pipeline: Configure a stateless/reproducible data pipeline in DB2 that regenerates nightly via a GitHub Action.
• Pipeline Validation: Validate the entire pipeline to ensure it behaves as expected under various conditions (e.g., pre-existing tables, newly created tables).
• Testing: Test pipeline regeneration and data output integrity.
• Documentation: Document the pipeline behavior, table-handling logic, and GitHub Action configuration for team reference.