DV360 & Ads Data Hub Data Pipeline

Project Overview

Built a production multi-pipeline data platform that synchronizes Google DV360 advertising data from GCP to AWS. The system consists of three event-driven pipelines -- metadata sync with parent-before-child ordering across 6 dependency tiers, performance report auto-detection and field normalization, and Ads Data Hub match-rate analysis with async state machine orchestration. All three pipelines use a transactional outbox pattern with receipt-based reconciliation for reliable cross-cloud delivery from BigQuery to AWS SQS.

Key Challenges

• DV360 campaign entities needed dependency-aware delivery so parent entities reached downstream systems before child entities
• DV360 report tables arrived asynchronously in BigQuery and needed automatic detection and normalization
• Ads Data Hub workflows required long-running async orchestration with privacy-aware failure handling
• Cross-cloud delivery to AWS SQS needed receipt tracking and reconciliation to prevent silent message loss
• System required multi-project CI/CD with environment isolation between dev and production

Results & Impact

• Three independent event-driven pipelines running in production with no manual cron jobs or polling
• 8 DV360 entity types synchronized with enforced parent-before-child ordering across 6 tiers
• 3 report formats auto-detected, normalized, and delivered via BigQuery audit log triggers
• 6-state async state machine managing ADH API interactions with retry and privacy cooldown
• Per-message receipt tracking and expected-vs-actual reconciliation for every cross-cloud delivery