Prompt
I'm building a tool to identify the sources behind model responses about companies
Latest observation
That’s a useful tool.
If your goal is to identify the sources behind model responses about companies, the core problem is usually:
- What claim is being made?
- Which source supports it?
- How reliable/current is that source?
- Can the claim be traced to a specific passage or document?
A practical approach is to build the tool around claim extraction + source retrieval + attribution scoring.
Suggested pipeline
1. Parse the model response into atomic claims
Break a response into small, checkable statements, for example:
- “Company X acquired Company Y in 2023.”
- “Company X has 12,000 employees.”
- “The CEO is Jane Doe.”
This makes source matching much easier.
2. Retrieve candidate sources
Search across likely source types:
- company website / press releases
- SEC filings / annual reports
- earnings call transcripts
- reputable news articles
- databases like Crunchbase, LinkedIn, Wikidata, etc. if relevant
- government registries, patents, trademarks, court filings
3. Score evidence for each claim
For each candidate source, evaluate:
- Directness: does it explicitly state the claim?
- Recency: is it up to date?
- Authority: is it a primary source?
- Consistency: do multiple sources agree?
- Specificity: exact match vs. vague support
4. Produce attribution
Return something like:
- claim
- supporting source(s)
- quoted evidence snippet
- source type
- confidence score
- any conflicts or ambiguities
A good data model
You may want to store:
{
"claim": "Company X acquired Company Y in 2023",
"entities": ["Company X", "Company Y"],
"claim_type": "acquisition",
"sources": [
{
"url": "https://...",
"title": "...",
"published_at": "2023-05-12",
"source_type": "press_release",
"evidence": "Company X announced it acquired Company Y..."
}
],
"confidence": 0.94
}
Important design choices
Primary vs secondary sources
For company facts, prioritize:
- Primary: filings, press releases, official reports
- Secondary: reputable news, databases
- Tertiary: aggregators or summary sites
Temporal validity
Company facts change often:
- leadership
- headcount
- funding
- ownership
- office locations
So your tool should attach a timestamp and possibly answer:
- “As of when is this true?”
Citation granularity
Best results come from linking claims to exact spans in documents, not just URLs.
If you want high accuracy
Consider a two-step system:
- Retriever: gathers candidate documents
- Verifier: checks whether the documents actually support the claim
This helps avoid false attribution when a source merely mentions the company but doesn’t support the statement.
Example output format
{
"response_summary": "Company X expanded into Europe in 2022.",
"claims": [
{
"text": "Company X expanded into Europe in 2022",
"supported_by": [
{
"source": "press release",
"url": "https://companyx.com/news/europe-expansion",
"evidence": "In 2022, Company X launched operations in Germany and France."
}
],
"confidence": "high"
}
]
}
Pitfalls to avoid
- attributing a claim to a source that only mentions related entities
- mixing up similar company names
- using stale sources for current facts
- relying too heavily on one secondary source
- failing to distinguish “reported by” from “confirmed by”
If you want, I can help you design:
- a system architecture for this tool,
- a schema for evidence and citations, or
- a claim-to-source matching algorithm.