Senior Principal DevOps Engineer – Orchestration

About the role

Own the reliability, deployment, and operational infrastructure behind Orchestrator and the AI Fabric environments it manages.

You will build and maintain Kubernetes clusters across on-prem and cloud, design CI/CD pipelines for continuous delivery, manage Terraform-driven infrastructure-as-code, and handle secret and certificate rotations.

You will stand up and operate the full observability stack — Prometheus, Grafana, Loki, Splunk, Datadog, and Timestream — ensuring end-to-end visibility across customer deployments. When things break, you are the person who troubleshoots and debugs infrastructure issues at the platform and customer site level.

Beyond keeping things running, you will build internal tooling and analytics that improve operational efficiency, reduce incident response time, and scale our infrastructure as deployments grow. We are looking for someone who has been through production pain, knows what good SRE looks like, and can bring that discipline to a fast-moving team.

• Kubernetes cluster lifecycle across hybrid environments: provision, upgrade, scale, and harden clusters running on bare-metal (on-prem customer datacenters) and cloud (AWS/GCP) using kubeadm, Rancher, or equivalent tooling
• CI/CD pipeline design and ownership: build and maintain pipelines (GitHub Actions or equivalent) that deliver Go microservices, React UI, Helm charts, and edge appliance images from commit to production with automated testing gates
• Infrastructure-as-code: manage all cloud and on-prem infrastructure through Terraform modules with proper state management, drift detection, and PR-based review workflows
• Observability stack operations: deploy, tune, and maintain Prometheus (metrics), Grafana (dashboards), Loki (logs), Splunk and Datadog (enterprise monitoring), and Timestream (time-series analytics) — build the dashboards and alerts that give the team real-time visibility into platform and customer-site health
• Secret and certificate management: automate mTLS certificate rotation across hub-to-edge communication channels, manage Vault or equivalent secret stores, and handle credential lifecycle for multi-tenant deployments
• Incident response and debugging: own the runbooks, triage production issues across the distributed hub/edge architecture, perform root cause analysis, and drive post-incident reviews that result in real fixes — not just documents
• Customer site operations: support deployment, upgrade, and troubleshooting of edge appliances running in customer datacenters with varying network constraints and access patterns
• Internal tooling: build CLI tools, deployment automation, environment provisioners, and operational dashboards that reduce toil and make the engineering team faster
• Capacity planning and cost optimization: monitor resource utilization across clusters and cloud accounts, right-size workloads, and forecast infrastructure needs as site count grows

• 8–14 years in SRE, DevOps, or infrastructure engineering roles supporting production distributed systems
• Deep Kubernetes expertise: cluster administration, networking (CNI, ingress, service mesh), storage (PV/PVC, CSI drivers), RBAC, and troubleshooting pod/node-level issues in both cloud and bare-metal environments
• Strong Terraform skills with experience managing multi-environment, multi-provider infrastructure at scale
• Hands-on experience building and operating CI/CD pipelines end-to-end — not just configuring someone else's templates
• Production experience with at least three of: Prometheus, Grafana, Loki, Splunk, Datadog, Timestream, or comparable observability tools
• Solid scripting and automation skills in Python, Bash, or Go
• Working knowledge of Linux systems internals: networking (iptables, DNS, TCP debugging), storage, process management, and performance analysis
• Experience managing TLS/mTLS certificates, secret rotation, and Vault or equivalent in production
• Comfort working across cloud (AWS/GCP) and on-prem environments with different constraints and access models
• Strong debugging instincts — you can follow a problem from a user report through load balancers, ingress, service mesh, application logs, and database queries to root cause

• Experience supporting network infrastructure or datacenter automation platforms
• Helm chart authoring and management for complex multi-service applications
• Bare-metal Kubernetes provisioning (not just managed EKS/GKE)
• eBPF-based observability tools (Cilium, Pixie, Hubble)
• Experience operating Kafka, ClickHouse, ArangoDB, or Redis in production
• Familiarity with SONiC, network switch management, or ZTP workflows
• On-call experience with a structured incident management process (PagerDuty, Opsgenie)
• SOC 2, FedRAMP, or equivalent compliance experience for infrastructure