Staff Software Development Test Engineer

About the Role : 

We are looking for a highly motivated AI Software Development Engineer in Test to join Tekion’s AI Platform team. Our team builds the infrastructure and tooling that powers every AI capability at Tekion — model training and serving infrastructure, feature and data pipelines, model registry and deployment, LLM gateways, vector stores, and the observability that ML Engineers and Data Scientists build on top of.

In this role, you will be responsible for building confidence in the quality, reliability, correctness, and performance of the AI Platform itself. You will work closely with Platform and Infrastructure Engineers, ML Engineers, and Product Management to define robust validation strategies, automate testing across platform services and SDKs, and ensure the platform behaves correctlyand scales under the demands of a growing fleet of AI agents.
As the platform expands to serve 100+ AI agents, you will help define quality and evaluation frameworks for platform-level AI services — validating model-serving correctness, feature integrity, and the reliability of the tooling teams depend on. You will also use AI and LLMs to accelerate test generation and automation across the platform.


What You’ll Do : 

• Develop a deep understanding of Tekion’s AI Platform architecture, services, data/feature flows, and the ML lifecycle it supports.
• Own end-to-end quality for platform capabilities — feature pipelines, training and serving infrastructure, model registry, deployment pipelines, LLM gateways, and vector/data stores.
• Design and implement automated testing frameworks for platform APIs, SDKs, services, and infrastructure components.
• Validate the correctness, completeness, and consistency of features and model inputs/outputs as they move through the platform.
• Test model-serving and inference services for correctness, latency, throughput, scalability, and reliability under load.
• Build regression and reconciliation suites for large-scale platform changes, migrations, and 
  version upgrades.
• Validate platform capabilities such as ingestion, feature computation, model deployment/rollback, multi-tenancy, and governance.
• Define quality metrics and SLO validation, with automated monitoring to detect platform anomalies before they impact ML teams or customers.
• Contribute to evaluation frameworks for platform-level AI services (e.g., serving correctness, feature integrity, eval-infrastructure reliability).
• Use AI/LLMs to accelerate quality engineering — generating tests, synthetic data, and automation scaffolding.
• Drive root cause analysis for production platform issues and partner with engineering teams on preventive improvements.
• Champion quality engineering and reliability best practices across the AI Platform organization.

What We’re Looking For

• 8+ years in SDET, quality engineering, or software engineering, with strong experience building automation frameworks for platform, infrastructure, or backend services.
• Strong programming skills in Python and/or Java/Go, with the ability to write productionquality automation and tooling.
• Experience testing distributed systems, APIs, data pipelines, or ML/data platforms.
• Solid understanding of ML lifecycle components — feature stores, training/serving, model registry, deployment — or a strong platform-testing background with ML fluency.
• Experience with performance, load, and scalability testing.
• Familiarity with CI/CD, infrastructure-as-code, containers (Docker, Kubernetes), and cloud (AWS/GCP/Azure).
• Understanding of LLM/ML failure modes and evaluation concepts.
• Strong debugging and root cause analysis skills across service, data, and infrastructure layers.
• Excellent collaboration and communication skills.

Nice to Have

• Experience with ML platform tools (MLflow, Feast, KServe/Seldon, Ray, Kubeflow, SageMaker/Vertex AI) or LLM infra (vLLM, model gateways, vector DBs such as Milvus/Pinecone).
• Experience with observability and monitoring stacks (Prometheus, Grafana, Datadog).
• Experience testing multi-tenant, high-throughput SaaS platforms.
• Contributions to internal test frameworks or developer tooling.
• Exposure to chaos or reliability engineering