Lakehouse Performance Engineer

At IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale. Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation. With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape. Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world. IBM is building the next generation of watsonx.data: a GPU-accelerated, open data lakehouse engineered to deliver category-leading price‑performance for analytics and AI workloads. We are hiring a Performance Engineer to be a hands-on individual contributor focused on measuring, defending, and improving the performance and cost‑per‑performance of the platform across every release.

You will run the benchmarks, build the harnesses, and operate the backing infrastructure that the entire watsonx.data organization relies on to characterize performance. That includes the dedicated benchmark labs, GPU and CPU test fleets, dataset stores, result warehouses, and the automation that ties them together. Engineering, product, field, and competitive intelligence will all consume what you produce: regression signals in CI, executive scorecards, customer-facing dashboards, and the data behind public claims that we are the market-leading open lakehouse. Benchmarking & Workload Engineering Industry-standard benchmarks: Run, maintain, and continuously improve reproducible benchmarks across watsonx.data configurations and against competitive offerings. Customer-representative workloads: Build and curate workload suites that reflect real customer query mixes, data volumes, concurrency profiles, and freshness requirements. Reproducibility & rigor: Ensure every published result is reproducible end-to-end: controlled environments, pinned versions, locked datasets, documented methodology, variance analysis, and statistically defensible reporting. Cost-per-performance metrics: Operationalize the canonical price‑performance KPIs ($/query, $/TB scanned, $/training‑token, queries/sec/$, TCO at workload mix); instrument workloads, collect data, and produce repeatable scorecards. Performance Observability & Analysis Telemetry pipeline: Build and maintain the metrics, traces, profiles, GPU/CPU utilization, query plan, and IO telemetry that flow from benchmark runs into the performance data store. Dashboards & scorecards: Develop dashboards that surface trends, regressions, and competitive position to engineering, leadership, and external audiences. Regression gates: Operate performance regression gates in CI/CD; triage failures, file and drive issues with engine, storage, and GPU teams, and verify fixes. Root-cause analysis: Drill into slow queries and GPU/CPU bottlenecks using profilers (Nsight, perf, async-profiler, pprof, flamegraphs) and query plan inspection to pinpoint regressions and improvement opportunities. Backing Infrastructure for Performance Performance environment ownership: Own the lifecycle of the dedicated performance environment(s) supporting watsonx.data: GPU and CPU clusters, networking, storage, and the orchestration that schedules workloads onto them. Test fleet automation: Build and maintain infrastructure-as-code (Terraform/Ansible/Helm) for provisioning, configuring, and resetting test environments deterministically across on-prem hardware and cloud (IBM Cloud and partner clouds). Benchmark harness platform: Develop and operate the benchmark harness itself: job scheduler, run orchestration, dataset provisioning, result capture, artifact storage, and the API/CLI other teams use to launch runs. Dataset & result warehouse: Own the curated datasets used for benchmarking and the warehouse of historical results that powers trend analysis, regression detection, and competitive comparisons. Capacity & utilization: Manage capacity and utilization of the performance lab so concurrent campaigns from different teams (query engine, storage, GPU acceleration, AI) run cleanly and without interference. 2+ years of professional software engineering experience with experience in performance engineering, databases, distributed systems, or comparable infrastructure product. Strong programming skills in at least one of Python, Go, Java, plus comfort with shell scripting and modern automation tooling. Experience operating real infrastructure: Linux servers, Kubernetes, container runtimes, networking basics. Infrastructure-as-code fluency in at least one of Terraform, Ansible, or Helm; comfort writing and maintaining the automation, not just consuming it. Comfort with observability tooling: metrics (Prometheus), tracing/telemetry (OpenTelemetry), and dashboards (Grafana or equivalent). Hands-on experience with GPU-accelerated data processing (RAPIDS/cuDF, Velox/Theseus‑class engines, CUDA) and the GPU memory hierarchy (HBM, NVLink, PCIe trade‑offs). Experience operating a multi-tenant performance lab or shared test fleet where multiple teams ran experiments concurrently. Experience building bespoke benchmark harnesses or workload generators, including dataset generation at TB+ scale. Familiarity with vector search, retrieval-augmented generation (RAG), and AI inference/training performance characterization. Contributions to relevant open-source projects (Iceberg, Trino, Spark, Arrow, Velox, RAPIDS, OpenTelemetry, perf-tooling, etc.). United States Software Engineering Hybrid Professional Austin, US