Lead Big Data Architect Needed for Database Optimization

LumenWorks

Contract · Remote · Flexible hours

About the role

LumenWorks is modernizing its data platform to support rapid product releases. As a Big Data Architect you will lead a test automation uplift aimed at reducing regression risk before each launch, leveraging the newly coined Nimbus Test Suite to validate data pipelines and analytics workloads.

Key responsibilities

• Design and implement a scalable test automation framework for end‑to‑end data pipeline validation using Python and PyTest.
• Integrate the Nimbus Test Suite with existing CI/CD pipelines on AWS CodePipeline.
• Develop automated regression test cases for Spark jobs, Kafka streams, and Redshift ETL processes.
• Collaborate with data engineers to embed contract‑based testing in Airflow DAGs.
• Monitor test coverage metrics and produce weekly risk reports for product releases.
• Guide the migration of legacy batch jobs to containerized environments with Docker for consistent test execution.

Must-have skills

• Deep expertise in big data architecture, data modeling, and schema design.
• Extensive experience with Python, Spark, and SQL on PostgreSQL/AWS Redshift.
• Proven ability to build CI/CD pipelines and automated testing frameworks.
• Strong knowledge of Docker containerization and orchestration.
• Hands‑on experience with AWS services (S3, EMR, Glue, CodePipeline).

Nice to have

• Familiarity with Terraform for infrastructure as code.
• Experience with data quality tools such as Great Expectations.