Key Responsibilities

Design and Build Data Pipelines: Develop and implement efficient data pipelines for collecting, transforming, and storing data across different platforms.
Data Integration: Integrate data from a variety of sources, including cloud platforms, Google Cloud), databases (SQL/NoSQL), APIs, and external services.
Optimize Data Flow: Ensure optimal performance of data flows by troubleshooting and fine-tuning existing pipelines.
Data Transformation: Implement ETL (Extract, Transform, Load) processes, transforming raw data into usable formats for analytics and reporting.
Collaboration: Work with cross-functional teams (data engineering, operations) to understand data needs and implement solutions; support the applications on need basis over the weekend or non-office hours.
Automation & Scalability: Build scalable, automated workflows to handle large volumes of data with high reliability and low latency.
Monitoring & Maintenance: Set up monitoring and alerting systems for data pipelines to ensure minimal downtime and maximum performance.
Documentation: Document data flows, pipeline configurations, and processing logic to ensure maintainability and transparency.

Required Qualifications

Technical Skills:

Proficient in programming languages such as Python, Java.
Hands-on experience with data pipeline orchestration tools (e.g. Google Dataflow).
Experience with cloud-based data platforms like Google Cloud (BigQuery, Dataflow).
Strong experience with ETL frameworks and tools; real-time data streaming and processing.
Familiarity with data formats like JSON, Parquet, etc.
Knowledge of SQL and NoSQL databases; data governance, data quality, and security best practices.

Dataflow Engineer