Role Summary 

Splunk is seeking a senior individual contributor to serve as the technical lead for the AI agents that make our technical sellers more productive, setting direction through thought leadership and influence across field teams. The role spans the full agent lifecycle across multiple platforms: setting best practices for cost, performance, design, and integrations; deciding which agents are built where; completing the last-mile build; maintaining a subset in production; and directing and reviewing a contractor who builds additional agents. You will combine agentic engineering (designing and building agents) with agentic operations (running them reliably, safely, and cost-effectively) as the single technical authority guiding quality from design through production. Your immediate mission is to deliver an initial 3 to 5 agents that streamline technical sellers’ daily workflows, freeing them to spend more time customer-facing and advancing Splunk’s revenue and customer-success goals. 

Responsibilities 

Agentic Platform Strategy (Architect) 

Reference Design & Standards: Advise on the end-to-end design for agentic systems across platforms, contributing inputs to the reference designs, patterns, and standards agents are built against, and informing system-level decisions on how agents reason, integrate, and scale. 

Cross-Platform Best Practices: Define, document, and evangelize best practices for cost, performance, design, and integrations across the organization’s agent platforms, so teams build consistently and avoid duplicated effort. 

Platform Selection & Build Allocation: Guide which agents are built on which platform, balancing capability, cost, integration fit, security, and maintainability, and set the criteria the organization uses to decide. 

Contractor Direction & Review: Set technical direction for a contractor building agents, specifying what to build and where, and reviewing their work for quality, security, and adherence to standards. (Technical direction and review, not people management.) 

Technical Thought Leadership & Influence: Serve as the recognized technical authority for agentic systems across field teams, contributing to reference designs and standards, mentoring engineers, and driving alignment with IT, Engineering, Cloud Operations, Security, Product, and field stakeholders through influence. 

Strategic Impact: Advance Splunk’s revenue and customer-success goals by increasing technical sellers’ customer-facing time and making the agents behind demonstrations and Field Solutions Apps more capable, reliable, safe, and cost-effective. 

Agent Delivery & Engineering (Build) 

Deliver Seller-Productivity Agents: Own delivery of an initial 3 to 5 production agents that streamline technical sellers’ daily workflows, prioritizing the use cases that return the most selling time and measurably reduce manual effort. 

Last-Mile Build & Integration: Complete the last-mile build of agents, wiring up tools, Model Context Protocol (MCP) servers and clients, data sources, and integrations, and hardening them from prototype to dependable production use. 

Agent Design & Orchestration: Translate the reference design into working agentic systems with evaluation and guardrails designed in from the start, and define retrieval (RAG) and tool-integration strategies that keep agents grounded and accurate. 

Agentic Operations (Operate) 

Observability & Monitoring: Establish comprehensive logging, monitoring, and alerting for agents in production. Because agents make non-deterministic decisions, ensure deep visibility into agent health, behavior, and performance, and lead root-cause analysis when agents fail. 

Performance & Cost Evaluation: Track resource consumption across agentic workloads, including API usage and token limits across LLMs. Define and monitor SLOs/SLIs, manage cost-per-task economics, and drive model-routing and efficiency decisions that keep performance high and spend predictable. 

Governance & Guardrails: Define the boundaries that keep agents safe, within budget, and compliant with security and regulatory standards, establishing policy, permissions, audit trails, and human-in-the-loop checkpoints so every agent action is bounded, traceable, and auditable. 

Continuous Optimization & Maintenance: Maintain a subset of production agents, resolving regressions and keeping them current as platforms and models evolve, while using telemetry and evals to refine prompts, tools, topologies, and guardrails. Validate changes in simulation before release and automate away manual toil. 

Qualifications 

Technical Skills: 

3+ years of software, ML, or platform engineering experience, including a demonstrated track record of leading technical direction through influence (no direct-report or people-management requirement) 

Proven ability to act as a technical authority and force multiplier (contributing to reference designs and standards, mentoring engineers, and driving adoption) without formal management authority 

Hands-on experience designing and building production agentic systems, including multi-agent orchestration and Model Context Protocol (MCP) integrations 

Experience selecting among and building across multiple agent platforms, with the judgment to match use cases to the right platform 

Strong tool and API integration skills, including designing retrieval (RAG) strategies that keep agents grounded and accurate 

Experience building evaluation harnesses and offline/online evals, and validating agentic systems in simulation before release 

Experience implementing guardrails, policy and permission models, and audit trails for autonomous systems 

Familiarity with token and cost management across multiple LLM providers, including model routing and fallback strategies 

Experience directing and reviewing the work of contractors, vendors, or peer engineers through technical review (without formal management authority) 

Hands-on experience operating agents in production, including observability and monitoring with Splunk and Splunk Observability (Splunk o11y) 

Experience with Splunk products (Splunk Cloud/Enterprise, Splunk IT Service Intelligence, Splunk Observability, or equivalent) for telemetry, monitoring, and observability 

Proficiency in cloud-native architectures, preferably AWS 

Experience in programming/scripting languages such as Python and Bash 

Understanding of distributed systems, microservices architecture, and API design 

Strategic & Business Skills: 

Strong strategic thinking and ability to translate business objectives into technical roadmaps 

Excellent communication skills with ability to present technical concepts to non-technical stakeholders 

Track record of delivering results in fast-paced, dynamic environments  

Data-driven decision-making approach with focus on measurable outcomes 

Core Values and Mindsets:  

Commitment to Excellence: A dedication to delivering high-quality solutions, strengthening technical skills, and taking pride in continuous improvement to consistently raise the bar. 

Adaptability / Flexibility: A growth mindset, openness to feedback, and the ability to respond effectively to new challenges and rapid change. 

Collaboration: A strong team player attitude, excellent communication skills, and a commitment to empowering colleagues and fostering an open, supportive environment. 

Ownership: A proactive approach to improvement, accountability for outcomes, and a drive to deliver on commitments without waiting for direction. 

Innovation: A passion for creative problem-solving, technical rigor, and pushing boundaries while maintaining high standards. 

Fun: An ability to embrace fun, celebrate success, and contribute to a positive and energetic culture where everyone can thrive.