How do I set up a query log analyzer for tracking intent shifts and relevance metrics?

To set up a query log analyzer for tracking intent shifts and relevance metrics, build it as a pipeline with four parts:

1. Collect query logs
2. Infer intent and relevance signals
3. Detect shifts over time
4. Dashboards + alerts

Here’s a practical setup.

1) Define what you want to measure

Intent shifts

Track how user intent changes across:

• Time: day/week/month
• Segments: locale, device, traffic source, user cohort
• Query variants: same topic, different wording
• Result changes: before/after ranking updates, content changes, model updates

Typical intent classes:

• Navigational: looking for a specific site/page
• Informational: wants knowledge/answers
• Transactional: wants to buy/sign up/download
• Commercial investigation: comparing options
• Support / troubleshooting
• Ambiguous / multi-intent

Relevance metrics

Track both offline and behavioral relevance:

• CTR@k: clicks on top-k results
• Zero-click rate
• Dwell time
• Long click rate / pogo-sticking
• Reformulation rate: user re-queries after poor result
• Success rate: query ends without refinement
• NDCG / MAP / MRR if you have labeled judgments
• Abandonment rate: query session ends with no action

2) Build the log schema

Capture at least:

Query event

• timestamp
• query_text
• normalized_query
• user_id or anonymous session id
• session_id
• locale
• device
• referrer
• vertical/product/category
• result_set_id
• ranked_results
• latency

Interaction event

• clicks
• clicked_doc_id
• position
• dwell_time
• scroll_depth
• conversion_event
• refinement_query
• back_to_serp

Optional enrichment

• embeddings for query
• predicted intent label
• topic/category
• satisfaction score
• entity extraction

3) Normalize and enrich the queries

Before analysis:

• lowercase
• trim punctuation
• spell-correct if needed
• tokenize / lemmatize
• remove duplicates / bot traffic
• group equivalent queries (synonyms, aliases)

Then enrich with:

• Intent classifier
• Topic model / cluster id
• Entity recognition
• Query embedding

A good pattern is:

• Use rules for obvious cases
• Use a lightweight classifier for known intents
• Use embeddings/clustering for emerging or ambiguous intents

4) Detect intent shifts

A. Distribution drift

Compare intent distributions across time windows:

• last 7 days vs prior 7 days
• this month vs last month

Metrics:

• % of queries by intent
• KL divergence / Jensen-Shannon divergence
• Population Stability Index (PSI)

Example:

• Informational queries drop from 62% → 45%
• Transactional queries rise from 18% → 30% That’s an intent shift.

B. Cluster movement

Cluster queries by embedding and track:

• new clusters appearing
• old clusters shrinking
• queries moving between clusters

Useful for:

• detecting new user needs
• seasonality
• product launches
• policy changes

C. Session-level intent transitions

Look at query sequences in sessions:

• informational → transactional
• broad query → refined query
• support query → “contact us”

Metrics:

• transition matrix
• average number of refinements
• time to success

D. Query reformulation patterns

Track:

• exact repeat queries
• added modifiers (“best”, “cheap”, “near me”, “2026”)
• query shortening vs expansion
• semantic distance between original and reformulated queries

A rising reformulation rate often means the first result set didn’t satisfy the user.

5) Compute relevance metrics

Behavioral relevance proxies

For each query/result set:

• CTR@1, CTR@3, CTR@10
• Mean reciprocal rank (MRR) for clicked result
• Dwell time by position
• Backtrack rate
• Conversion rate
• Query success rate

Interpretation:

• High CTR + low dwell time may mean clickbait / poor relevance
• Low CTR + high reformulation rate usually means poor ranking
• High dwell time + conversion suggests strong relevance

If you have human judgments

Use labeled relevance grades:

• NDCG@k
• MAP
• Precision@k
• Recall@k

Make sure judgments are:

• query-specific
• intent-aware
• graded on relevance, not just “clickability”

6) Segment everything

Intent and relevance can vary a lot by segment:

• new vs returning users
• mobile vs desktop
• region/language
• logged-in vs anonymous
• new content vs evergreen content
• branded vs non-branded queries

This is important because a global metric can hide local problems.

7) Set up dashboards

Build a dashboard with:

Top-level KPIs

• query volume
• intent distribution
• CTR@k
• reformulation rate
• zero-click rate
• success rate
• conversion rate

Trend views

• time series of intent shares
• drift metrics
• relevance metrics by query class
• top failing queries

Drill-down views

• per query
• per cluster
• per intent
• per segment
• per result position

Alerting

Trigger alerts for:

• sudden drift in intent mix
• drop in CTR or success rate
• spike in reformulation
• new query cluster emergence
• big changes after ranking/content releases

8) Choose an implementation stack

A simple production setup:

Ingestion

• Kafka / Kinesis / PubSub
• app logs + search logs + click logs

Storage

• data warehouse: BigQuery, Snowflake, Redshift
• raw logs in S3/GCS

Processing

• Spark / dbt / pandas for smaller scale
• scheduled jobs with Airflow / Dagster

Modeling

• scikit-learn / XGBoost for intent classification
• sentence-transformers / embedding models for clustering
• language-specific NLP libraries if needed

Visualization

• Looker / Tableau / Superset / Power BI
• custom dashboards if needed

9) A minimal pipeline design

1. Ingest logs
2. Sessionize queries
3. Normalize text
4. Predict intent
5. Cluster query embeddings
6. Join with clicks/dwell/conversion
7. Aggregate by time + segment
8. Compute drift and relevance metrics
9. Publish dashboards + alerts

10) Example analysis questions

Your analyzer should answer:

• Which intents are growing or shrinking?
• Are some segments getting worse relevance?
• Which query clusters have the highest reformulation rate?
• Did ranking changes affect transactional queries more than informational ones?
• Are new intents appearing before the classifier recognizes them?
• Which queries have high volume but low success?

11) Recommended first version

If you want a fast v1:

• Use session-based logs
• Add a query embedding model
• Cluster queries weekly
• Classify top-level intent
• Track CTR, reformulation rate, dwell time, conversion
• Compare week-over-week distributions
• Alert on major drift

That gets you useful intent-shift tracking without overengineering.

12) Pitfalls to avoid

• Treating clicks as pure relevance signals without accounting for position bias
• Ignoring seasonality
• Mixing bot traffic with real users
• Using one global metric for all intents
• Not separating navigational vs informational queries
• Not validating embeddings/clusters with human review
• Forgetting that relevance differs by intent

If you want, I can also provide:

1. a sample database schema,
2. a Python/Spark implementation outline, or
3. a dashboard metric spec for this analyzer.