Senior Principal Engineer — Orchestration

The role

You will define the cross-cutting architecture that the engineering team builds within. You own platform-scale initiatives from vision through delivery — authoring architecture decision documents, building POCs to validate direction, and partnering with Product to shape the roadmap. You write code, you ship, and you are accountable for the quality of what reaches stakeholders.

• Distributed control plane spanning cloud services and on-prem edge appliances connected via mTLS gRPC streams
• Observability and telemetry at scale: OpenTelemetry collection pipelines, stream processing (Kafka), time-series storage (ClickHouse/VictoriaMetrics), real-time fabric state views, packet-event analysis, and fleet-wide health aggregation
• Agentic AI operations: design and build autonomous infrastructure agents that evaluate prerequisites, orchestrate multi-step workflows (onboarding, upgrades, drift remediation), handle failure recovery, and interact with the control plane through tool-use patterns (LangGraph, MCP)
• Fleet orchestration: aggregate health/compliance/drift APIs, cross-site campaign execution, template promotion workflows, parallel site onboarding
• Data architecture across Postgres, ArangoDB, ClickHouse, Redis, Kafka, and Git-backed content stores
• Multi-tenant SaaS with site-scoped RBAC, session lifecycle, and enterprise IdP integration
• Device lifecycle management: zero-touch provisioning, enrollment protocols, config push via edge relay, drift detection and remediation
• Intent compilation engine: workspace management, merge request lifecycle, change request execution with gate-based verification and rollback

• 15–18 years building and operating distributed systems serving enterprise customers across cloud and on-prem environments
• Deep proficiency in Go (or comparable systems language) with strong distributed systems fundamentals
• Significant experience designing observability and telemetry platforms: collection agents, stream processing, time-series databases, alerting pipelines, and real-time dashboards at scale
• Production experience with microservices architecture: gRPC, Protocol Buffers, spec-first REST APIs (OpenAPI)
• Hands-on with multiple storage paradigms: relational, graph, time-series, key-value, and streaming
• Track record building multi-tenant platforms with tenant isolation, RBAC, and identity federation
• Experience with Kubernetes, Helm, and hybrid cloud/on-prem deployment models
• Strong API design sense: versioning, backward compatibility, contract-first development
• Demonstrated ability to lead platform-scale technical initiatives across multiple teams and deliver on time
• Ability to communicate architectural decisions clearly through writing and diagrams 
  
  

How we expect you to work

• Cross-functional by default — you partner with Product to shape the roadmap, work with QA to define quality gates, and collaborate with Customer Engineering to ground decisions in real deployment reality
• Solution-oriented — you don't just identify architectural problems, you build POCs to prove the path forward and help the team move
• Accountable for delivery — you own outcomes across the initiatives you lead, not just the designs
• Hands-on always — you write code, review the most consequential PRs, and prove your architecture works by building it

 

Nice to have

• AI/ML agent architectures for infrastructure operations: LangGraph, AutoGen, MCP tool-use, human-in-the-loop gating, and autonomous workflow orchestration
• Network automation or infrastructure management platforms
• Datacenter networking experience: OpenConfig, gNMI, or fabric management at scale
• Hub-and-spoke / edge computing / control-plane-data-plane separation architectures
• OpenTelemetry contributor experience or deep familiarity with the collector ecosystem
• Device enrollment or zero-touch provisioning systems
• Open-source contributions or published work in distributed systems