Senior Data Platform Engineer

Government-backed Abu Dhabi organization focused on advanced technology R&D (est. 2020), defining strategy, funding, and policies across AI, robotics, and emerging technologies. Oversees the full innovation lifecycle - from research and programs to commercialization - through dedicated applied research, innovation, and venture entities.

The first production system is an AI-enabled operational platform that gives a senior leadership team a shared situational picture, an AI-classified signal feed, a daily AI-generated briefing, and an action accountability tracker. MVP target: operational within two weeks of team formation. The platform is also the technical foundation for all subsequent Data & AI systems across the organization.

Responsibilities

Build and operate the data platform that powers the DAIO's (Data & AI Office) production systems and the long-term data estate. In the immediate term: the signal ingestion pipeline, data quality layer, and observability for all data flows. In the medium term: the enterprise data warehouse on Azure and sovereign compute, the metadata catalog, and the governed data access layer for AI agents.

WHAT THIS ROLE BUILDS & OWNS

• Signal ingestion pipeline — 30-minute polling job across all defined open-source feeds (news wires, maritime AIS, financial feeds, social/keyword feeds)
• Deduplication and normalization layer — common signal schema across all sources
• Ingestion observability — every item logged with source, timestamp, processing status, and failure reason; no silent drops
• PostgreSQL schema deployment and migration scripts (Alembic)
• Azure Redis Cache — session management and ingestion queue configuration
• Phase 2 data warehouse: ADLS + Synapse/Fabric, data ingestion from SAP, M365, and ATRC enterprise systems
• Data quality monitoring — automated checks on signal completeness, classification coverage, and freshness

KEY DECISIONS THIS ROLE OWNS

• Polling frequency, retry logic, and backoff strategy for each signal source
• Deduplication key design — what makes a signal unique across sources
• Whether a data quality failure is a warning (flag it) or a stop (pause ingestion)
• Schema migration approach — blue-green, Alembic auto-migrate, or manual rollout
• Data retention schedule — what is archived, what is purged, and when

WHAT THIS ROLE DOES NOT DO

• Define the data model or classification schema — that is the Head of Data Architecture
• Build the application API endpoints — that is the Backend/Systems Engineers
• Write AI prompts or tune classification outputs
• Manage cloud infrastructure provisioning — that is a DevOps/infra function

Skills Required

• Microsoft data stack - Fabric or Synapse;
• Strong Python(custom integration code, async processing);
• Building and supporting data pipelines(Airflow or similar);
• Postgres/Alembic
• Observability and monitoring(prometheus/grafana/etc)

Engagement Model: Direct Independent Contractor (Please read carefully)

This is an independent contractor opportunity based on a direct contractual relationship between Zoolatech and the individual service provider.

To facilitate this direct partnership, we engage with professionals who are registered and operate as a sole proprietorship, private entrepreneur, or an equivalent self-employment status in your country.

Please note, our model does not accommodate contracts through third-party intermediaries such as agencies, incubators, or umbrella companies. The essential requirement is your ability to enter into a service agreement and invoice Zoolatech directly. This is not an offer of direct employment

Please note that only candidates whose profiles closely match our requirements will be contacted.