Skip to content

Learning & Refinement

RuleChef uses a buffer-first architecture: examples are collected, then committed to a dataset and used for rule synthesis during learn_rules().

Buffer-First Architecture

flowchart TD
    A[add_example / add_correction] --> B[Buffer]
    B --> C[learn_rules]
    C --> D[Buffer → Dataset]
    D --> E[Rule Synthesis]
    E --> F[Evaluation & Refinement]
    F --> G[Persisted Rules]
chef.add_example(input1, output1)   # Goes to buffer
chef.add_example(input2, output2)   # Goes to buffer
chef.add_correction(input3, wrong, correct)  # High-priority signal

chef.learn_rules()  # Buffer → Dataset → Synthesis → Refinement

learn_rules()

The main learning method orchestrates the full pipeline:

rules, eval_result = chef.learn_rules(
    max_refinement_iterations=3,  # Evaluation-refinement cycles
    incremental_only=False,       # If True, only patch (don't re-synthesize)
    holdout_fraction=0.2,         # Decide patch acceptance on a held-out dev split
    split_seed=42,                # Seed for the stratified split
)

What happens during learn_rules():

  1. Buffer examples are committed to the dataset
  2. Rules are synthesized using the LLM
  3. Rules are evaluated against the dataset
  4. Failed examples drive refinement iterations
  5. Each surviving rule is stamped with a validated precision (see Evaluation & Feedback)
  6. Rules are persisted to disk

Returns: A tuple of (List[Rule], Optional[EvalResult]). When holdout_fraction > 0, eval_result is measured on the held-out dev split.

Holdout Acceptance

By default the refinement loop accepts or rejects patches based on quality measured on the same data the patches were learned from, which can let rules memorize their training failures instead of generalizing. Set holdout_fraction to split off a development set:

chef.learn_rules(holdout_fraction=0.2, split_seed=42)

With a holdout active:

  • The training data is split into train and dev portions, stratified by class signature.
  • Corrections always stay in train — they are explicit user fixes and the highest-value patching signal, so holding them out would hide them from the learner.
  • Patch synthesis only ever sees the train portion, but acceptance is decided on dev: a patch is kept only if held-out F1 does not degrade (or precision improves).
  • Best-rule selection across iterations is also decided on dev.

This is the recommended setting for datasets with hundreds of examples or more. If the resulting dev split would be too small (fewer than 5 examples), no split is performed and acceptance falls back to the training data.

Failure-Mode Clustering

On large datasets an intermediate ruleset can produce thousands of failures — far more than fit in a patch prompt. Rather than truncating arbitrarily, RuleChef clusters failures by their signature — the expected class and the kind of error (missed span, spurious span, wrong type) — and gives the LLM:

  • the full distribution of failure modes (counts per signature), so it sees the shape of the problem, and
  • a round-robin sample of concrete instances drawn evenly across clusters.

This keeps patch prompts bounded while ensuring rare failure modes are not crowded out by common ones.

Synthesis Robustness

LLM-written rules can misbehave; RuleChef guards against the common failure modes automatically:

  • Catch-all ban. Patterns that match arbitrary text (e.g. .*, ^.{1,}$, pure negative-lookahead) are rejected at validation by probing them against generic strings, so a rule cannot inflate its score by matching everything.
  • Execution timeouts. Each rule runs under a wall-clock budget (RULE_TIMEOUT_S, 3s); a catastrophically-backtracking regex or a runaway code rule is aborted and skipped instead of freezing the loop.
  • Truncated-output recovery. If the LLM's JSON response is cut off, complete rule objects are salvaged from the partial output rather than discarding the whole response.

Synthesis Strategies

For multi-class tasks (NER, classification), RuleChef can synthesize rules per-class for better coverage:

# Auto-detect: per-class if >1 class, bulk otherwise (default)
chef = RuleChef(task, client, synthesis_strategy="auto")

# Force per-class synthesis
chef = RuleChef(task, client, synthesis_strategy="per_class")

# Force single-prompt bulk synthesis
chef = RuleChef(task, client, synthesis_strategy="bulk")

Per-class synthesis generates rules for each label separately — one LLM call per class. Each call receives:

  • Positive examples for that class (capped to max_samples, sampled using sampling_strategy)
  • Counter-examples from other classes (capped to max_counter_examples) to prevent false positives
  • Up to max_rules_per_class rules are generated per call

For a 5-class classification task with default settings, this means 5 LLM calls, each producing up to 5 rules, with up to 50 positive examples and 10 counter-examples per prompt.

Prompt Size Controls

These parameters control how much data goes into each LLM prompt:

chef = RuleChef(task, client,
    max_samples=50,             # Max examples per prompt (per-class positives + patch failures)
    max_rules_per_class=5,      # Max rules generated per class in per-class synthesis
    max_counter_examples=10,    # Max negative examples per class prompt
    max_rules=10,               # Max rules per bulk synthesis call
)
Parameter Default Applies to
max_samples 50 Per-class positive examples and patch failure sampling
max_rules_per_class 5 Per-class synthesis (rules generated per class)
max_counter_examples 10 Per-class synthesis (negative examples per prompt)
max_rules 10 Bulk synthesis (total rules per call)

When a class has more examples than max_samples, examples are sampled using the configured sampling_strategy.

Sampling Strategies

Control how training data is selected when there are more examples than max_samples:

chef = RuleChef(task, client, sampling_strategy="balanced")
Strategy Behavior
balanced Takes examples in order (default)
recent Favor recently added examples
diversity Evenly space picks across the dataset
uncertain Low-confidence examples first
varied Mix of recent, diverse, and uncertain
corrections_first Corrections first, then recent examples

All strategies always include corrections first — they're the highest-value signal.

Incremental Patching

After the initial learn, you can patch rules for specific failures without re-synthesizing everything:

# Initial synthesis
chef.learn_rules()

# Add corrections for failures
chef.add_correction(
    {"text": "some input"},
    model_output={"label": "wrong"},
    expected_output={"label": "correct"},
)

# Patch existing rules (don't re-synthesize)
chef.learn_rules(incremental_only=True)

Incremental patching:

  • Generates targeted rules for known failures
  • Merges new rules into the existing ruleset
  • Deletes underperforming rules when better replacements are provided
  • Prunes weak rules that don't contribute
  • Preserves stable rules that are working

During patching, the LLM can list rules in a "deleted_rules" field to remove them from the ruleset. This is used when a rule is too broad (high false positives) and the LLM provides narrower replacement rules in the same response.

A patch is accepted if micro F1 stays within 0.5%, or if precision improves (higher precision at the cost of some recall is considered a net quality win). Otherwise the patch is rejected and the previous rules are kept. When holdout_fraction > 0, this decision is made on the held-out dev split (see Holdout Acceptance).

Persistence

Rules and datasets are automatically saved to disk:

chef = RuleChef(task, client,
    dataset_name="my_project",       # Filename for the dataset
    storage_path="./rulechef_data",  # Directory for JSON files
)

Files saved:

  • {storage_path}/{dataset_name}.json — dataset with examples, corrections, rules

Loading happens automatically when a matching file exists at the storage path.