Senior Data Engineer Needed for Stream Processing Setup

Ridgeline Software

Contract · Remote · Flexible hours

About the role

Ridgeline Software is revamping its legacy data pipeline to support continuous, zero‑downtime releases. As a Senior Data Engineer you will drive the migration of the core "Atlas" ingestion module to a cloud‑native architecture while preserving live traffic.

Key responsibilities

• Design and implement a streaming ETL layer using Apache Flink on AWS Kinesis to replace batch jobs.
• Migrate existing PostgreSQL and MySQL schemas to Amazon Aurora with zero‑downtime cut‑over strategy.
• Build automated data validation tests in Python with Great Expectations for each migration milestone.
• Set up CI/CD pipelines in GitHub Actions to deploy Terraform‑provisioned infrastructure for data workloads.
• Monitor latency and throughput via CloudWatch dashboards, tuning resources to meet SLA targets.
• Document migration steps and handover procedures for the operations team.

Must-have skills

• Advanced Python for data pipelines and orchestration.
• Deep experience with PostgreSQL, MySQL, and Amazon Aurora.
• Proficiency in AWS services: Kinesis, Lambda, CloudFormation/Terraform.
• Strong knowledge of streaming frameworks such as Flink or Spark Structured Streaming.
• Hands‑on CI/CD implementation using GitHub Actions or Jenkins.

Nice to have

• Familiarity with data catalog tools like AWS Glue.
• Experience implementing data observability platforms.