Product Analyst

About Bolna

Why this role exists

What you'll do

Model & evaluation analysis

• LLM & model benchmarking: Run structured comparisons across model providers (e.g. Sarvam, DeepSeek, Gemini, Claude variants) for tasks like post-call extraction and LLM-as-judge scoring - evaluating cost, accuracy, fill rate, and TTR, with particular attention to Hinglish and code-mixed content.
• Evaluation infrastructure: Build and maintain LLM-as-judge pipelines (e.g. using DeepEval), design and track eval metrics, and run inter-rater reliability analysis (e.g. Krippendorff's alpha) across human call reviewers.
• Golden dataset creation: Support the construction of golden datasets for ASR/transcript labeling - including flagging conventions like code-switch scripting (Devanagari vs. Roman) and transliteration normalization for decision before scoring is meaningful.
• ASR/voice benchmarking: Evaluate WER and related quality metrics across ASR providers and models for Indic languages; work with public benchmarks and academic references where relevant.
• Infra & latency analytics: Analyze routing, latency, and cost data (e.g. Azure PTU utilization, percentile latency distributions) to inform infrastructure and routing decisions.
• Agent behavior analytics: Support population-level analysis of graph agent behavior - node-level aggregates, designed-vs-observed graph diffs, stuck-in-loop detection and similar failure-pattern metrics.

Product & growth insight generation

• Self-serve journey analysis: Instrument and analyze the self-serve funnel (signup → activation → habit → expansion), identifying where users drop off and surfacing friction points for the product team to act on.
• Cross-customer call insights: Mine aggregate call data across customers and use cases for patterns - completion rates by use case, which model/config combinations perform best where, emerging failure patterns across prompt templates - feeding both internal roadmap decisions and future customer-facing benchmarking.
• Ad hoc product analysis: Be the fast, reliable "pull me the data on X" resource for pod PMs - usage patterns, cohort behavior, feature adoption - turning raw usage data into a clear, actionable read.

Across both

• Reporting & tooling: Build repeatable dashboards/scripts (not one-off notebooks) so these analyses run as an ongoing cadence, not a one-time project; present findings to product, ML, and infra stakeholders in a form they can act on.

What we're looking for

Must-have

• 0–2 years of experience - new grad or early-career, in a data/product analyst, applied ML, or research-adjacent role. We're hiring for raw analytical strength and trainability, not a finished track record.
• Strong SQL and Python (pandas at minimum) from coursework, internships, or prior work - comfortable writing and debugging your own queries and scripts without hand-holding.
• Solid statistical fundamentals (distributions, basic hypothesis testing, agreement/reliability concepts) - you don't need production experience with inter-rater reliability metrics, but you should pick up a new stats concept quickly when pointed at it.
• Genuine comfort with ambiguous, messy real-world data - able to notice when something looks off and flag it clearly, even if the resolution isn't yours to call.
• Basic funnel/cohort analysis instincts - comfortable thinking in terms of drop-off stages and segments, even if you haven't formally run a growth funnel before.
• Native or near-native fluency in Hindi (or another Indian language) and English, with comfort reading/labeling code-mixed (Hinglish) text.

Strong plus

• Any exposure to LLM evaluation concepts (prompt-based scoring, LLM-as-judge), speech/ASR evaluation (WER, transcript QA), or eval frameworks like DeepEval - even coursework or personal projects count.
• Any exposure to product/growth analytics - funnel analysis, retention curves, cohort behavior - from a prior role, internship, or personal project.
• Familiarity with cloud inference economics (e.g. token-based billing, provisioned throughput models) is a bonus, not expected.
• A portfolio of self-directed analysis (a project, competition, or writeup) that shows you go looking for the "so what," not just the number.

First 90 days - success looks like

• Running the LLM benchmarking comparison for post-call extraction end-to-end under direction — executing the sweep, producing clean cost/accuracy/fill-rate tables - with judgment calls escalated rather than made solo.
• Contributing meaningfully to the golden dataset build: flagging inconsistencies in transliteration/script conventions clearly enough that a fast decision can be made, then applying that decision consistently across the dataset.
• Running the inter-rater reliability pipeline on a recurring basis once it's set up, without needing to redesign it each time.
• Has produced a clear, first-pass view of the self-serve funnel (signup → activation → habit → expansion) with at least one concrete drop-off point identified and flagged for the team to act on.
• Has become the reliable first pass on "pull me the data on X" for at least two of: model benchmarking, ASR evaluation, routing/latency, agent behavior analytics, self-serve/usage patterns - freeing senior time for interpretation rather than execution.

What we offer

• Innovative culture: You’ll be part of a generational opportunity to define the trajectory of AI, surrounded by a team pushing the boundaries of what’s possible.
• Growth paths: Joining Bolna means joining a dynamic team with countless opportunities to drive impact - beyond your immediate role and responsibilities.
• Learning & development: Bolna proactively supports professional development, the processes for which are being set up.
• Competitive compensation + meaningful ESOP
• Bengaluru office, in-person team collaboration