Detectors¶
Factory¶
make_detector(method, **kwargs)
¶
Create a detector of the requested type with the given parameters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
str
|
One of "transformer", "llm", or "rag_fact_checker". |
required |
kwargs
|
Passed to the concrete detector constructor. |
{}
|
Returns:
| Type | Description |
|---|---|
BaseDetector
|
A concrete detector instance. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If method is not supported. |
Base Detector¶
BaseDetector
¶
Bases: ABC
All hallucination detectors implement a common interface.
predict(context, answer, question=None, output_format='tokens')
abstractmethod
¶
Predict hallucination tokens or spans given passages and an answer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
list[str]
|
List of passages that were supplied to the LLM / user. |
required |
answer
|
str
|
Model-generated answer to inspect. |
required |
question
|
str | None
|
Original question ( |
None
|
output_format
|
str
|
|
'tokens'
|
Returns:
| Type | Description |
|---|---|
list
|
List of predictions in requested format. |
predict_prompt(prompt, answer, output_format='tokens')
abstractmethod
¶
Predict hallucinations from a pre-built prompt string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt
|
str
|
Full prompt (context + question already concatenated). |
required |
answer
|
str
|
Model-generated answer to inspect. |
required |
output_format
|
str
|
|
'tokens'
|
Returns:
| Type | Description |
|---|---|
list
|
List of predictions in requested format. |
predict_prompt_batch(prompts, answers, output_format='tokens')
abstractmethod
¶
Batch version of :meth:predict_prompt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompts
|
list[str]
|
List of full prompt strings. |
required |
answers
|
list[str]
|
List of answers to inspect. |
required |
output_format
|
str
|
|
'tokens'
|
Returns:
| Type | Description |
|---|---|
list
|
List of prediction lists, one per input pair. |
Transformer Detector¶
TransformerDetector(model_path, max_length=4096, device=None, lang='en', **tok_kwargs)
¶
Bases: BaseDetector
Detect hallucinations with a fine-tuned token classifier.
When the combined context + answer exceeds max_length tokens, the
context is automatically split into chunks. Each chunk is scored
independently and the per-token hallucination probability is aggregated
across chunks with max().
Initialize the transformer detector.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_path
|
str
|
Path to the pre-trained model. |
required |
max_length
|
int
|
Maximum length of the input sequence. |
4096
|
device
|
device | str | None
|
Device to use for inference. |
None
|
lang
|
Lang
|
Language of the model. |
'en'
|
tok_kwargs
|
object
|
Additional keyword arguments for the tokenizer. |
{}
|
predict(context, answer, question=None, output_format='tokens')
¶
Predict hallucination tokens or spans from the provided context, answer, and question.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
list[str]
|
List of passages that were supplied to the LLM / user. |
required |
answer
|
str
|
Model-generated answer to inspect. |
required |
question
|
str | None
|
Original question ( |
None
|
output_format
|
str
|
|
'tokens'
|
Returns:
| Type | Description |
|---|---|
list
|
List of predictions in requested format. |
predict_prompt(prompt, answer, output_format='tokens')
¶
Predict hallucination tokens or spans from the provided prompt and answer.
Note: unlike :meth:predict, this method does not chunk the prompt
automatically. If the prompt + answer exceed max_length, the prompt
will be truncated. Use :meth:predict with structured passages for
automatic chunking.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt
|
str
|
The prompt string. |
required |
answer
|
str
|
The answer string. |
required |
output_format
|
str
|
|
'tokens'
|
Returns:
| Type | Description |
|---|---|
list
|
List of predictions in requested format. |
predict_prompt_batch(prompts, answers, output_format='tokens')
¶
Predict hallucination tokens or spans from the provided prompts and answers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompts
|
list[str]
|
List of prompt strings. |
required |
answers
|
list[str]
|
List of answer strings. |
required |
output_format
|
str
|
|
'tokens'
|
Returns:
| Type | Description |
|---|---|
list
|
List of prediction lists, one per input pair. |