Senior Data Engineer – Pipeline Reliability Initiative

Northbay Systems

Contract · Remote · Flexible hours

About the role

Northbay Systems is accelerating its release cadence by building a CI/CD and platform automation sprint for its flagship data platform, codenamed "PulseStream". As a Senior Data Engineer you will design, implement, and tune data pipelines that integrate directly into the automated deployment workflow, ensuring faster, reliable releases of analytics services.

Key responsibilities

• Architect end‑to‑end data pipelines using Python and Airflow that trigger on GitHub Actions events.
• Containerize ETL jobs with Docker and push to AWS ECR for seamless rollout.
• Implement schema migration automation for PostgreSQL and MySQL using Flyway within the CI pipeline.
• Set up automated data quality checks with Great Expectations and embed results into the release dashboard.
• Collaborate with DevOps to integrate Terraform‑managed AWS resources into the CI/CD flow.
• Document pipeline versioning strategy and create runbooks for rollback procedures.

Must-have skills

• Advanced Python programming for data engineering.
• Deep experience with PostgreSQL and MySQL schema design.
• Proficiency in AWS services (S3, Redshift, Lambda, ECS).
• Hands‑on with CI/CD tools: GitHub Actions, Docker, Terraform.
• Strong understanding of data pipeline orchestration (Airflow, Prefect).

Nice to have

• Familiarity with Great Expectations or similar data testing frameworks.
• Experience building real‑time streaming pipelines with Kafka.