Generate Manifold Merge

[claytongentry]

This PR adds support for generating BigQuery merge routines for Manifold tables. The merge routines are responsible for aggregating metrics across different contexts and dimensions.

For now, we support COUNTIF and SUM(IF...) style aggregations.

Configuration Format

• Added support for a simplified source configuration format in manifold.yml:

  source: my_project.my_dataset.my_table
  filter: timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
  timestamp:
    field: created_at
    interval: DAY

SQL Generation

• Added MetricsBuilder class to handle complex metric aggregations:

  • Supports COUNTIF and SUMIF metrics
  • Handles nested context conditions with logical operators (AND, OR, NOT, etc.)
  • Generates properly structured metric fields in the output

• Added SQLBuilder class to generate merge routine SQL:

  • Generates CTE for metrics calculation
  • Properly handles timestamp truncation and filtering
  • Joins with dimensions table for complete manifold records
  • Implements MERGE semantics for upserting records

Schema Validation

• Added validation for required manifold configuration fields:
  • Source table
  • Timestamp field
  • Contexts
  • Metrics

Example

timestamp:
  field: created_at
  interval: DAY

contexts:
  paid: IS_PAID(context.location)

metrics:
  countif: tapCount

source: analytics.events

will generate a merge

1. Aggregates tap counts for paid contexts
2. Groups by day using the created_at field
3. Merges with existing dimensions
4. Updates or inserts records in the manifold table