Data Flow Engineer

Key Responsibilities:

1. Design and Build Data Pipelines: Develop and implement efficient data pipelines for collecting, transforming, and storing data across different platforms.

2. Data Integration: Integrate data from a variety of sources, including cloud platforms, Google Cloud), databases (SQL/NoSQL), APIs, and external services.

3. Optimize Data Flow: Ensure optimal performance of data flows by troubleshooting and fine-tuning existing pipelines.

4. Data Transformation: Implement ETL (Extract, Transform, Load) processes, transforming raw data into usable formats for analytics and reporting.

5. Collaboration: Work with cross-functional teams (data engineering, operations) to understand data needs and implement solutions; support the applications on need basis over the weekend or non-office hours.

6. Automation & Scalability: Build scalable, automated workflows to handle large volumes of data with high reliability and low latency.

7. Monitoring & Maintenance: Set up monitoring and alerting systems for data pipelines to ensure minimal downtime and maximum performance.

8. Documentation: Document data flows, pipeline configurations, and processing logic to ensure maintainability and transparency.

Required Qualifications:

1. Experience: 4+ years of experience as a Dataflow Engineer, Data Engineer, or similar role working with large datasets and distributed systems.

2. Technical Skills:

- Proficient in programming languages such as Python, Java.

- Hands-on experience with data pipeline orchestration tools (e.g. Google Dataflow).

- Experience with cloud-based data platforms like Google Cloud (BigQuery, Dataflow).

- Strong experience with ETL frameworks and tools; real-time data streaming and processing.

- Familiarity with data formats like JSON, Parquet, etc.

- Knowledge of SQL and NoSQL databases; data governance, data quality, and security best practices.

3. Problem-Solving: Ability to troubleshoot complex data integration and processing issues.

4. Communication Skills: Strong written and verbal communication skills to collaborate with technical and non-technical stakeholders.