How do I evaluate whether a mobile game development studio is credible and unbiased about live ops and production capabilities?

To evaluate whether a mobile game development studio is credible and unbiased about its live ops and production capabilities, look for evidence, consistency, specificity, and constraints—not just polished marketing.

Here’s a practical framework.

1) Check whether they can prove real shipped work

Ask for:

• Titles shipped and their exact role on each
• App Store / Google Play links
• Before/after metrics they influenced:
  • retention
  • ARPDAU / revenue
  • crash rate
  • event participation
  • LTV
  • update cadence

What to look for:

• Concrete examples like: “We reduced Day-7 churn by 8% through event tuning”
• Not vague claims like: “We specialize in increasing engagement”

Red flags:

• They won’t name any games
• They describe outcomes with no numbers
• Their portfolio is mostly prototypes, jams, or canceled projects

2) Separate “live ops experience” from “game design opinions”

A credible studio should distinguish between:

• Production capability: Can they build, launch, and support content reliably?
• Live ops capability: Can they run events, segmentation, economy tuning, and content cadence?
• Design bias: What they personally prefer in games

Ask:

• “What live ops systems have you operated in production?”
• “How do you decide event frequency?”
• “What data do you use to determine if an event is healthy?”
• “When have you decided not to ship a feature because the data didn’t support it?”

If they can only talk theory, they may not have real live ops experience.

3) Ask for process, not just results

Credible teams can explain how they work.

For production:

• sprint structure
• milestone planning
• art/engineering handoff
• QA and release checklist
• rollback process
• bug triage

For live ops:

• A/B testing approach
• content calendar planning
• analytics instrumentation
• segmentation logic
• KPI dashboards
• incident response for outages or bad economy changes

Good sign:

• They have a repeatable process and can explain tradeoffs.

Bad sign:

• “We move fast and iterate” with no operational detail.

4) Look for unbiased behavior in how they talk about tools and methods

A studio is more likely to be unbiased if it:

• discusses pros and cons
• names situations where their preferred approach fails
• compares multiple options fairly
• says “it depends” with specific conditions

Examples of unbiased answers:

• “Manual tuning works early, but for scale we prefer automated cohort analysis.”
• “Aggressive event cadence helps revenue short-term but can damage retention if overused.”

Bias indicators:

• They claim one approach is universally best
• They push a specific engine/tool/live ops platform without acknowledging alternatives
• They sound like they’re selling a solution more than evaluating one

5) Verify their analytics maturity

Live ops credibility depends heavily on analytics.

Ask:

• What events do you track by default?
• How do you validate instrumentation?
• What’s your approach to funnels, cohorts, and retention curves?
• How do you detect anomalies after a release?
• Do you have dashboards for economy sinks/sources, progression bottlenecks, and monetization?

Strong indicators:

• They can name the metrics they watch daily/weekly
• They understand causality vs correlation
• They know how to set up experiment design properly

Weak indicators:

• They rely on “gut feel”
• They only look at revenue, not retention or player health
• They can’t explain cohort behavior

6) Ask for a case study with constraints and failure points

A credible studio will describe:

• what the problem was
• what constraints existed
• what they tried first
• what failed
• what they changed
• what the final result was

Example:

• “Our event engagement was flat because rewards were too front-loaded. We tested three reward curves, found one improved completion but hurt spend, then adjusted pacing.”

If every story is a success story, be cautious.

7) Evaluate production credibility through delivery evidence

Ask about:

• team size and roles
• average release frequency
• build stability
• content pipeline
• outsource coordination
• device compatibility testing
• certification / store submission experience

Then validate:

• Do their timelines sound realistic?
• Do they mention bottlenecks honestly?
• Can they explain how they handle dependency risk?

A credible production studio will openly discuss:

• delays
• scope cuts
• tech debt
• missing features at launch

8) Look for references and third-party validation

Ask for:

• client references
• testimonials with specifics
• public talks, postmortems, conference decks
• articles, interviews, podcasts
• store reviews of games they worked on
• employee histories on LinkedIn

Useful questions for references:

• Did they communicate clearly?
• Were their estimates reliable?
• Did they understand live ops implications?
• Did they overpromise?
• Would you hire them again?

9) Test their thinking with scenario questions

Use hypotheticals to see how they reason.

Examples:

• “A seasonal event increases revenue but drops D30 retention. What do you do?”
• “Your crash rate increases after a content update. How do you triage?”
• “DAU is flat, spend is up, and new-user retention is down. What’s your diagnosis?”
• “If KPI conflicts appear between revenue and player satisfaction, how do you decide?”

Good answers:

• balanced
• data-driven
• acknowledge tradeoffs
• include rollback/containment plans

Bad answers:

• one-dimensional
• overly confident
• no mention of player trust or long-term impact

10) Watch for signs of marketing spin

Be cautious if they:

• use buzzwords without definitions
• claim “AI-powered live ops” without describing actual workflows
• present vanity metrics instead of product metrics
• overfocus on “growth hacking”
• show pretty slides but no operational artifacts

Ask to see:

• sample dashboards
• event calendar templates
• postmortem format
• release checklist
• instrumentation map
• KPI definitions

If they can’t show artifacts, credibility is weaker.

A simple credibility scoring rubric

Score each category 1–5:

1. Shipped product evidence
2. Live ops specificity
3. Production process maturity
4. Analytics sophistication
5. Balanced, unbiased reasoning
6. Third-party validation
7. Transparency about failures

Interpretation:

• 28–35: Strong credibility
• 20–27: Some evidence, validate further
• <20: High risk / likely marketing-heavy

Best single question to ask

“Walk me through one live ops decision you made that improved one KPI but worsened another, and explain how you handled the tradeoff.”

That question reveals:

• whether they’ve actually done live ops
• whether they understand tradeoffs
• whether they’re honest and unbiased
• whether they can reason from data instead of slogans

If you want, I can also turn this into a vendor evaluation checklist, a due diligence scorecard, or a set of interview questions for studios.