Scores - Bayesian Optimization as a Coverage Tool for Evaluating LLM" > Scores - Bayesian Optimization as a Coverage Tool for Evaluating LLM" >
Skip to content

Scores

bocoel.Score

Bases: Protocol

__call__ abstractmethod 

__call__(target: Any, references: list[Any]) -> float

Calculate the score of a target given references.

Parameters

target: Any The target to calculate the score for.

references: list[Any] The references to calculate the score from.

Returns

The score of the target given the references.

Source code in bocoel/models/scores/interfaces.py 
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@abc.abstractmethod
def __call__(self, target: Any, references: list[Any]) -> float:
    """
    Calculate the score of a target given references.

    Parameters
    ----------

    `target: Any`
    The target to calculate the score for.

    `references: list[Any]`
    The references to calculate the score from.

    Returns
    -------

    The score of the target given the references.
    """

    ...

bocoel.NltkBleuScore

Bases: Score

bocoel.SacreBleuScore 

SacreBleuScore()

Bases: Score

Source code in bocoel/models/scores/bleu.py 
22
23
24
25
26
27
28
29
30
31
32
def __init__(self) -> None:
    # Optional dependency.
    from sacrebleu import BLEU

    self._bleu = BLEU(
        smooth_method="exp",
        smooth_value=0.0,
        force=False,
        lowercase=False,
        tokenize="intl",
    )

bocoel.models.scores.rouge._RougeScoreType module-attribute 

_RougeScoreType = Literal['rouge-1', 'rouge-2', 'rouge-l']

bocoel.RougeScore 

RougeScore(metric: _RougeScoreType)

Bases: Score

Source code in bocoel/models/scores/rouge.py 
11
12
13
14
15
16
def __init__(self, metric: _RougeScoreType) -> None:
    # Optional dependency.
    from rouge import Rouge

    self._rouge = Rouge()
    self._metric = metric

bocoel.models.scores.rouge._RougeScore2Type module-attribute 

_RougeScore2Type = Literal['rouge1', 'rouge2', 'rougeL']

bocoel.RougeScore2 

RougeScore2(typ: _RougeScore2Type)

Bases: Score

Source code in bocoel/models/scores/rouge.py 
39
40
41
42
43
def __init__(self, typ: _RougeScore2Type) -> None:
    from rouge_score.rouge_scorer import RougeScorer

    self._scorer = RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
    self._typ = typ

bocoel.OneHotChoiceAccuracy

Bases: Score

bocoel.MultiChoiceAccuracy

Bases: Score