How do I set up NER annotation software for a customer support analytics team doing intent classification?

Here’s a practical way to set up NER annotation software for a customer support analytics team that’s building data for intent classification.

1) Clarify what you’re annotating

For intent classification, you usually need more than just NER. Decide whether your labels are:

• Intent labels: e.g. refund_request, cancel_subscription, reset_password
• Entity labels: e.g. product_name, order_id, date, account_type
• Trigger phrases / spans: text that indicates the intent
• Multi-label cases: one message can contain more than one intent

If your goal is pure intent classification, NER is often used to capture key slots/entities that help the model. If the goal is to train an intent model, consider a tool that supports text classification + span annotation.

2) Pick software with the right features

Look for software that supports:

• Span-based annotation for entities
• Text classification labels for intents
• Custom label sets
• Reviewer/QA workflow
• CSV/JSON export
• API or easy import/export
• Inter-annotator agreement metrics if multiple annotators are involved

Common options:

• Label Studio: flexible, open source, good for span + classification
• doccano: simple, open source, good for text classification and sequence labeling
• Prodigy: great for active learning, commercial
• INCEpTION: powerful for NLP annotation, more complex
• Commercial platforms: if you need enterprise workflow, auth, audit logs, etc.

For a customer support analytics team, Label Studio or doccano are often the easiest starting points.

3) Design your label taxonomy

This is the most important part.

For intents

Keep labels:

• Mutually exclusive if possible
• Business-aligned
• Specific enough to be useful
• Not too granular at first

Example intent set:

• refund_request
• cancel_subscription
• change_billing_info
• shipping_status
• technical_issue
• account_access
• product_inquiry

For entities

Use only the entities that help downstream analysis:

• order_id
• ticket_id
• product_name
• billing_amount
• date
• email
• phone_number
• shipping_carrier

Avoid over-labeling low-value entities.

4) Define annotation guidelines

Create a short, unambiguous guide covering:

• What each intent means
• Edge cases
• When to assign multiple labels
• How to handle sarcasm, ambiguous requests, or mixed intents
• Entity boundary rules
• Examples and counterexamples

For customer support text, define rules for things like:

• “I want to cancel but keep access until the end of the month”
• “Where is my order and can I get a refund?”
• “My app crashes after the latest update”

This is where most annotation quality comes from.

5) Prepare the data

Before loading into the tool:

• Remove or mask PII if needed
• Split long conversations into manageable units
• Decide whether to annotate:
  • individual customer messages
  • full conversation turns
  • entire threads
• Deduplicate repeated messages
• Balance your sample set across known intents if possible

If you only annotate customer messages, your intent labels are often cleaner. If you annotate full threads, make sure the context is still clear in the UI.

6) Configure the annotation interface

Set up the project so annotators see only the needed fields.

Typical setup:

• Show the message text
• Provide intent labels as a classification section
• Provide entity span labels if needed
• Add optional free-text notes for ambiguous cases
• Enable keyboard shortcuts for speed

If using Label Studio, you can create a layout with:

• a text area
• a classification block for intent
• a span labeling block for entities

7) Establish a workflow

A simple production workflow:

1. Annotator labels sample
2. Second annotator reviews a subset
3. Adjudicator resolves disagreements
4. Export cleaned dataset
5. Train model
6. Review model errors
7. Iterate labels and guidelines

For early stages, review at least 10–20% of annotations to catch taxonomy problems.

8) Measure quality

Track:

• Inter-annotator agreement
• Label distribution
• Most common disagreements
• Time per annotation
• Ongoing error patterns

If agreement is low, the issue is usually:

• unclear label definitions
• too many labels
• overlapping intents
• messy source text

9) Integrate with downstream analytics

Make sure your export format fits the ML pipeline.

Typical outputs:

• JSON with text + intent + entity spans
• CSV for classification
• BIO/IOB format for sequence labeling

Also store metadata:

• ticket channel
• language
• product line
• customer segment
• timestamp

This helps you analyze intent drift and segment-specific patterns later.

10) Start small, then expand

A good rollout plan:

• Begin with 5–10 core intents
• Annotate a few hundred to a few thousand examples
• Use model-assisted prelabeling if available
• Refine labels after reviewing confusion cases
• Expand taxonomy only when needed

Recommended setup for a support team

If you want a simple, effective setup:

• Tool: Label Studio or doccano
• Labels: intents + a small set of entities
• Workflow: 2-pass review on a subset
• Guidelines: short rules doc with examples
• Data: customer messages only at first
• Export: JSON for modeling
• Iteration: monthly taxonomy review

If you want, I can also give you:

1. a sample label taxonomy for customer support intents,
2. a Label Studio configuration example, or
3. a step-by-step rollout plan for a 5-person annotation team.