Lead Data Engineer
co.brick•Gliwice•Zdalnie
📝 Opis główny / Wstęp
Join a global tourism leader as a Lead Data Architect to spearhead an extensive cloud data modernization program. The primary focus of this long-term mission is a complex migration to Databricks Unity Catalog, involving the transition of all data layers (landing, raw, prepared, reporting, services) from Hive Metastore. You will be responsible for re-architecting the Lakehouse structure, standardizing governance, and modernizing ETL/ELT pipelines in a high-stakes, professional environment.
Szczegóły wspólpracy
-
Start Date: ASAP
-
Duration: Long-term.
-
Working Hours: 9:00 – 17:00 CET/CEST.
-
English Level: Minimum B2+.
Architectural Leadership: Designing and developing data architecture within Azure and Databricks environments, serving as the technical lead for the project team.
Unity Catalog Migration: Leading the migration of infrastructure, pipelines (DLT, Python/SQL jobs, Synapse/ADF), and metadata frameworks to Unity Catalog.
Lakehouse Design: Implementing Medallion Architecture and Metadata-Driven ETL standards across the organization.
Optimization & Standards: Fine-tuning data processing workflows (Python, SQL, Spark) and implementing robust logging, monitoring, and data quality frameworks.
DevOps Mastery: Creating and implementing CI/CD processes and branching strategies using Azure DevOps and Git.
Stakeholder Collaboration: Partnering with business and technical stakeholders to deliver top-tier solutions and mentoring junior team members.
Cloud Infrastructure: Advanced knowledge of Microsoft Azure (networking, authorization, data environment design).
Databricks Expert: Deep proficiency in Unity Catalog, Delta Live Tables (DLT), Workflows, and workspace administration.
Synapse Stack: Strong experience with Azure Synapse (Serverless and Pipelines).
Development: Excellent Python skills and experience with Apache Spark (workflow design and code optimization).
Data Modeling: Advanced SQL (query optimization, SQL Server) and experience designing scalable data warehouses.
Modern Practices: Proven track record with Medallion Architecture, Metadata-Driven ETL, and Lakehouse design.
Leadership: Previous experience in a Lead Engineer role, including technical decision-making and mentoring.
Nice-to-Have
-
Familiarity with Power BI for reporting and analytics integration.
-
Strong documentation skills and a passion for knowledge sharing.