Lead Big Data Architect Needed for Warehouse Migration

Silverline Labs

Contract · Remote · Flexible hours

About the role

Silverline Labs is building a next‑gen data platform to accelerate product releases. As a Big Data Architect you will drive a CI/CD and platform automation sprint, creating reusable pipelines that push data workloads to production faster and more reliably.

Key responsibilities

• Design end‑to‑end data pipelines using Apache Spark and Airflow for the "Velocity Hub" automation module.
• Implement Terraform‑based infrastructure as code for Hadoop clusters on AWS.
• Create Docker images for Spark jobs and integrate them into Jenkins‑X pipelines.
• Define schema evolution policies in PostgreSQL and enforce data quality checks with Great Expectations.
• Collaborate with DevOps to embed automated testing and rollback mechanisms for data releases.
• Document architecture decisions and produce runbooks for the CI/CD workflow.

Must-have skills

• Deep expertise in designing scalable big‑data architectures (Spark, Hadoop).
• Proven experience building CI/CD pipelines for data workloads.
• Strong command of AWS services (EMR, S3, Redshift) and Terraform.
• Advanced Python programming for ETL and automation.
• Solid understanding of PostgreSQL performance tuning and data modeling.

Nice to have

• Familiarity with Kubernetes operators for Spark.
• Experience with data catalog tools such as AWS Glue.