Member of Technical Staff - ML Infrastructure Engineer
Published: 2025-10-23At Black Forest Labs, we’re on a mission to advance the state of the art in generative deep learning for media, building powerful, creative, and open models that push what’s possible. Born from foundational research, we continuously create advanced infrastructure to transform ideas into images and videos. Our team pioneered Latent Diffusion, Stable Diffusion, and FLUX.1 – milestones in the ...
Job details
Black Forest Labs is a cutting-edge startup pioneering generative image and video models. Our team, which invented Stable Diffusion, Stable Video Diffusion, and FLUX.1, is currently looking for a strong candidate to join us in developing and maintaining our ML infra including large GPU training and inference clusters.
Role:
- Design, deploy, and maintain cloud-based ML training (Slurm) and inference (Kubernetes) clusters
- Implement and manage network-based cloud file systems and blob/S3 storage solutions
- Develop and maintain Infrastructure as Code (IaC) for resource provisioning
- Implement and optimize CI/CD pipelines for ML workflows
- Design and implement custom autoscaling solutions for ML workloads
- Ensure security best practices across the ML infrastructure
- Provide developer-friendly tools and practices for efficient ML operations
Ideal Experience:
- Strong proficiency in cloud platforms (AWS, Azure, or GCP) with focus on ML/AI services
- Extensive experience with Kubernetes and Slurm cluster management
- Expertise in Infrastructure as Code tools (e.g., Terraform, Ansible)
- Proven track record in managing and optimizing network-based cloud file systems and object storage
- Experience with CI/CD tools and practices (e.g., CircleCI, GitHub Actions, ArgoCD)
- Strong understanding of security principles and best practices in cloud environments
- Experience with monitoring and observability tools (e.g., Prometheus, Grafana, Loki)
- Familiarity with ML workflows and GPU infrastructure management
- Demonstrated ability to handle complex migrations and breaking changes in production environments
Nice to have:
- Experience with custom autoscaling solutions for ML workloads
- Knowledge of cost optimization strategies for cloud-based ML infrastructure
- Familiarity with MLOps practices and tools
- Experience with high-performance computing (HPC) environments
- Understanding of data versioning and experiment tracking for ML
- Knowledge of network optimization for distributed ML training
- Experience with multi-cloud or hybrid cloud architectures
- Familiarity with container security and vulnerability scanning tools