How do I evaluate whether a newsletter publishing site is credible and unbiased about subscriptions and reader analytics?

To judge whether a newsletter publishing site is credible and unbiased about subscriptions and reader analytics, look for evidence in four areas: who they are, how they measure things, what they reveal, and whether others can verify it.

1) Check who runs it and what their incentives are

A site is less likely to be neutral if it sells newsletter tools, ad products, or subscription services.

Look for:

• An “About” page with clear ownership
• Company affiliations, investors, or product offerings
• Whether the site is trying to sell a platform, sponsor placements, or analytics software
• Editorial policies, corrections policy, and contact info

Red flags:

• No clear ownership
• Anonymous authors making strong claims
• Heavy promotional language
• Claims that their numbers are “the industry standard” without evidence

2) Examine how they define subscriptions and readers

Different sites count things differently, so the definitions matter more than the raw numbers.

Ask:

• What counts as a subscriber?
  • Free signup only?
  • Confirmed email?
  • Paying subscriber?
  • Active subscriber?
• What counts as a reader?
  • Opened email?
  • Clicked a link?
  • Scrolled the article?
  • Visited the site?
• Over what time period are the numbers measured?
• Are they reporting unique users, total opens, or total pageviews?

Good signs:

• Clear methodology page
• Definitions for all key terms
• Separate metrics for opens, clicks, conversions, and paid subscriptions

Red flags:

• Vague wording like “engagement” without definition
• Mixing free subscribers with paying subscribers
• Using open rates as if they were the same as actual reads

3) Look for transparency in the analytics methods

Credible sites explain how data is collected and what its limitations are.

Check whether they disclose:

• Sample size
• Geographic or demographic bias
• Whether data comes from first-party tracking, surveys, or panel data
• How they handle bots, spam, duplicates, and unsubscribes
• Whether open tracking is affected by privacy features like Apple Mail Privacy Protection

Good signs:

• Methodology notes
• Confidence intervals or margins of error
• Acknowledgment of tracking limitations
• Comparisons across multiple datasets

Red flags:

• No mention of bot filtering
• No explanation of how opens are measured
• Claims of precision that exceed the method
• One-size-fits-all conclusions from a small or biased sample

4) Compare claims against independent sources

A credible site’s claims should broadly align with outside evidence.

Cross-check with:

• Newsletter platform reports
• Third-party analytics providers
• Industry research from reputable firms
• Publisher earnings calls or public filings
• Web traffic tools, where relevant
• Archived versions or direct observation of subscriber behavior

Questions to ask:

• Do their numbers match what others report?
• Are they selective about which data they cite?
• Do they present both positive and negative findings?

5) Watch for biased framing

A site can be technically accurate but still misleading in presentation.

Watch for:

• Cherry-picked examples
• Only success stories, no failures
• Rankings that favor their own products
• Language that implies causation when only correlation exists
• Overly confident conclusions from limited data

Example:

• “Email open rates increased after subject-line changes” is a correlation.
• “Subject lines caused revenue growth” is a stronger claim and needs more proof.

6) Evaluate editorial balance

If the site publishes analysis or advice, see whether it includes multiple viewpoints.

Better:

• Contradictory evidence
• Nuanced discussion of tradeoffs
• Explanation of when metrics are useful vs misleading

Weaker:

• One-sided arguments
• Repeated claims that one subscription model is always best
• No acknowledgement of exceptions

7) Signs of a more trustworthy site

A site is more credible if it:

• Clearly states ownership and motives
• Defines its metrics precisely
• Shows methodology and limitations
• Uses independent or reproducible data
• Corrects errors publicly
• Avoids sales-heavy or hype-driven language

Quick checklist

Before trusting the site, ask:

1. Who owns it?
2. What do they sell?
3. How do they define subscribers and readers?
4. How is the data collected?
5. What are the limitations?
6. Can independent sources confirm it?
7. Does the site present a balanced view?

If you want, I can also give you a 1-page credibility checklist you can use on any newsletter analytics site, or help evaluate a specific site if you share the URL.