Marketing Data Engineering
DV360 & Ads Data Hub Data Pipeline
Production event-driven data platform for DV360 metadata, DV360 report normalization, Ads Data Hub match-rate workflows, BigQuery processing, and reliable AWS SQS delivery.
Advertising Operations Agency · Solo
My role & contribution
Built the data pipelines for this engagement and handed them over to the client's internal team.
Production pipeline delivery
The problem
Built a production multi-pipeline data platform that synchronizes Google DV360 advertising data from Google Cloud to AWS. The system includes metadata sync with parent-before-child ordering, report table auto-detection and normalization, and Ads Data Hub match-rate orchestration. Delivery uses a BigQuery outbox pattern, SQS batch sends, receipt tracking, and reconciliation so cross-cloud movement can be monitored instead of treated as a black box.
The existing workflow
DV360 entities, asynchronously delivered reports, and long-running Ads Data Hub jobs needed different collection paths but a dependable downstream delivery contract.
System flow
01
Source triggers
DV360 metadata, DV360 report tables, and Ads Data Hub jobs trigger separate Cloud Run and workflow paths.
02
BigQuery processing
Landing/report tables are detected, normalized, and prepared for downstream delivery from BigQuery.
03
Outbox delivery
Expected messages are staged through a BigQuery outbox before batched delivery to AWS SQS.
04
Reconciliation
SQS receipts are compared with expected delivery counts so missing or failed sends are visible.
Key engineering decisions
Separate source trigger patterns
Metadata sync, report landing-table detection, and Ads Data Hub orchestration operate independently instead of forcing every source through one synchronous request.
Make delivery observable
A BigQuery outbox records expected sends. Batched SQS sends, receipt tracking, and reconciliation distinguish processed data from acknowledged downstream delivery.
Difficult problems
- DV360 campaign entities needed dependency-aware delivery so parent objects reached downstream systems before child entities
- DV360 report tables arrived asynchronously in BigQuery and needed automatic detection and normalization
- Ads Data Hub workflows required long-running async orchestration with privacy-aware failure handling
- Cross-cloud delivery to AWS SQS needed receipt tracking and reconciliation to prevent silent message loss
Work contributed
- Built three independent event-driven pipelines with source-specific trigger patterns
- Implemented tiered DV360 entity delivery for parent-before-child dependencies
- Normalized multiple DV360 report formats from BigQuery landing tables
- Added receipt tracking and expected-vs-actual reconciliation for cross-cloud delivery
Validation
- Checked dependency-aware parent/child ordering for entity delivery.
- Validated normalized report shape against landing tables.
- Compared expected outbox sends with delivery receipts and investigated mismatches.
Supported outcome
- Built three independent event-driven pipelines with source-specific trigger patterns
- Implemented tiered DV360 entity delivery for parent-before-child dependencies
- Normalized multiple DV360 report formats from BigQuery landing tables
- Added receipt tracking and expected-vs-actual reconciliation for cross-cloud delivery
Technologies in context
Python · BigQuery · Cloud Run · Cloud Tasks · Cloud Workflows · Eventarc · Ads Data Hub · DV360 · AWS SQS · Docker
Related services & work
Working through a similar problem?
Share the workflow and the output your team needs. We can identify a focused starting point.
Discuss your workflow