Senior Data Engineer

Job Description

We are seeking a skilled and dedicated hands-on Data Engineer to join our team. The successful candidate will have a strong background in data architecture, database management, and data pipeline development, working closely with our business stakeholders, clients, vendors, analysts and data scientists to ensure the smooth flow of data across the organization. You will play a key role in designing, developing, and maintaining robust and scalable data solutions that support critical business insights.

Responsibilities:

• Data Infrastructure Management: Design, implement, and maintain efficient and scalable data infrastructure, including data lakes, data warehouses, and relational and NoSQL databases.
• Reporting: Maintain existing reports and data feeds. Optimize existing ones for resilience and spend. Build new reports and feeds into our internal and external customers.
• Data Pipeline Development: Develop, test, and maintain ETL (Extract, Transform, Load) pipelines to collect, process, and transfer data across different sources and destinations.
• Data Quality and Governance: Ensure data quality, consistency, and accuracy through data validation, monitoring, and management of data governance standards.
• Optimization: Optimize database queries, data storage, and retrieval for performance improvements.
• Collaboration: Work closely with data scientists, analysts, and product teams to understand data requirements and translate them into efficient technical solutions.
• Data Security: Implement and enforce data security best practices to ensure compliance with regulatory standards.
• Documentation: Document data architecture, processes, and infrastructure changes.

Qualifications:

• Education: Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field.
• Experience: 3+ years of experience in a data engineering role or similar.
• Technical Skills:
  • Proficiency in SQL and experience with relational databases (e.g., MySQL, PostgreSQL).
  • Strong programming skills in Java, Python and Spring Boot.
  • Hands-on experience with big data processing tools (e.g., Apache Spark, Hadoop).
  • Experience with cloud platforms (e.g., AWS, Google Cloud, Azure) and their data services.
  • Familiarity with ETL tools (e.g., Apache NiFi, Airflow, Talend).
  • Knowledge of data warehousing concepts (e.g., Redshift, Snowflake).
• Soft Skills: Strong analytical skills, problem-solving abilities, and excellent communication and teamwork skills.

Preferred Skills:

• Experience with data modeling and database schema design.
• Familiarity with real-time data streaming (e.g., Kafka).
• Knowledge of machine learning infrastructure (e.g., MLflow, TensorFlow Extended).
• Experience working in the P&C Insurance industry.

Mandatory Competencies