I'm stuck because the data has too many duplicates for training

If your training data has lots of duplicates, you usually want to fix it before modeling, because duplicates can bias the model and make validation look better than it really is.

What to do

1. Identify exact duplicates
   • Remove rows that are completely identical.
2. Check near-duplicates
   • For text, images, or records with minor differences, use similarity rules or hashing.
3. Deduplicate before splitting
   • Don’t split first and dedupe later, or you can leak duplicates into train/test.
4. Keep only one copy, or aggregate
   • If duplicates represent repeated real events, you may want to keep them but use weights or counts instead of literal repeats.
5. Use group-aware splitting
   • If duplicates belong to the same entity/user/item, split by group so related rows don’t land in both train and test.

If duplicates are unavoidable

• Use sample weights
• Consider regularization
• Evaluate with a deduplicated validation set
• Use grouped cross-validation

Quick pandas example

df = df.drop_duplicates()

If you want, I can help you with the exact dedup strategy for your data type:

• tabular
• text
• images
• time series
• grouped/user-level data