Software Engineer - Snowflake

We are looking for a talented and motivated Software Engineer with experience in Confluent, Snowflake and QLIK to join our team. This role involves designing, developing, and maintaining real-time data streaming and cloud-based data warehousing solutions using Confluent Kafka, Snowflake. The ideal candidate will work closely with cross-functional teams to build scalable, efficient, and secure data pipelines that drive business intelligence and analytics.

Responsibilities:

• Data Pipeline Development: Design, develop, and optimize real-time and batch data pipelines using Confluent Kafka and Snowflake.

• Streaming Data Processing: Implement event-driven architectures leveraging Kafka Streams, ksqlDB, and Kafka Connect.

• Data Integration: Integrate structured and semi-structured data from multiple sources into Snowflake for analytics and reporting.

• Database Management: Develop, optimize, and maintain Snowflake-based data warehouses, ensuring high availability and performance.

• Security & Compliance: Implement data security best practices, including encryption, role-based access control (RBAC), and compliance with GDPR, HIPAA, or other relevant regulations.

• Optimization: Optimize query performance and storage efficiency in Snowflake using clustering, partitioning, and caching techniques.

• Testing: Conduct unit testing and data validation to ensure accuracy and reliability of data pipelines.

• Troubleshooting: Diagnose and resolve issues related to Kafka messaging, Snowflake storage, and data transformations.

• Documentation: Prepare and maintain technical documentation, including architecture diagrams, data flow processes, and best practices.

• Collaboration: Work with Data Scientists, BI Teams, and Cloud Engineers to provide real-time and batch data solutions.

• Maintenance: Monitor and maintain the stability of streaming pipelines and data warehouse operations.

Qualifications:

Must have

• Strong understanding of real-time data streaming and event-driven architectures.

• Hands-on experience with Confluent Kafka (Kafka Streams, Kafka Connect, ksqlDB).

• Experience in Snowflake data warehouse development and administration.

• Proficiency in SQL, Python, or Scala for data processing and transformation.

• Familiarity with cloud platforms (AWS, Azure, GCP) for data pipeline orchestration.

• Experience with ETL/ELT processes and data pipeline automation.

• Knowledge of CI/CD for data pipelines, version control (Git), and DevOps tools.

• Strong problem-solving and analytical skills.

Nice to have

• Confluent Kafka Certification or Snowflake Certification.

• Experience with Apache Airflow, dbt, or other data orchestration tools.

• Knowledge of Big Data Technologies (Spark, Flink, or Druid).

• Understanding of GraphQL, REST APIs, and WebSockets for data integration.

• Experience working in finance, healthcare, or other regulated industries.