Senior Generative AI Specialist – Eval Harness Design

TechFlow Inc

Contract · Remote · Flexible hours

About the role

TechFlow Inc is launching a high‑velocity CI/CD and platform automation sprint to accelerate release cycles for its new Generative AI suite, codenamed "Nebula Forge". As a Generative AI Specialist you will architect prompt pipelines, integrate model serving into the deployment workflow, and ensure reproducible builds across environments.

Key responsibilities

• Design and implement a GitOps‑based CI pipeline using GitHub Actions and Terraform for model versioning.
• Containerize LLM inference services with Docker and orchestrate them on Kubernetes, applying blue‑green deployments.
• Develop automated testing harnesses for prompt validation and output quality using pytest and custom evaluation scripts.
• Integrate Hugging Face Transformers models into the CI flow, automating model artifact storage in S3.
• Collaborate with DevOps to set up monitoring dashboards in Grafana for latency and cost metrics.
• Deliver a sprint‑end demo showcasing end‑to‑end model deployment from code commit to production endpoint.

Must-have skills

• Deep experience with LLM frameworks (e.g., Hugging Face, LangChain).
• Proficiency in Python for model scripting and automation.
• Strong knowledge of CI/CD tools: GitHub Actions, Jenkins, or GitLab CI.
• Containerization expertise with Docker and Kubernetes.
• Understanding of prompt engineering and evaluation metrics.

Nice to have

• Familiarity with Terraform or Pulumi for infrastructure as code.
• Experience with monitoring stacks like Prometheus/Grafana.