mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-02 10:34:46 -05:00
feat(pii): extend a detection over the previous word (extend_to_previous_word)
The mirror of extend_to_next_word. A name model often tags the surname with high confidence and scores the first name in front of it below min_score, so "Gast Greta Waldmeister" became "Gast Greta [REDACTED:ner:LASTNAME]". extend_to_previous_word lists entity groups whose accepted detections also cover the word directly before them on the same line. The preceding word must consist of letters (plus '-' and the apostrophe): a word with digits - an amount, a date, a room number - is never taken, nor is a protected term, and the search never crosses a line break. Docs: operations/middleware.md and advanced/model-configuration.md. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
This commit is contained in:
1 parent
dce0eacfcf
commit
b1845b8bed
10 files changed
+215
No files matched your search
@@ -1111,6 +1111,8 @@ PII redaction is NER-based and runs on the **request** (input) side. It has two
|
||||
EMAIL: mask
|
||||
extend_to_next_word: # optional: also mask the word right after these
|
||||
- ZIPCODE # groups on the same line (e.g. the town after a postal code)
|
||||
extend_to_previous_word: # optional: also mask the letters-only word right before these
|
||||
- LASTNAME # groups on the same line (e.g. the first name before a surname)
|
||||
protected_terms: # optional: values that must pass unchanged even if tagged
|
||||
- Hotel Seeblick # (plus protected_terms_files: one term per line, in the models path)
|
||||
```
|
||||
|
||||
@@ -121,6 +121,20 @@ A word is a run of letters, digits, combining marks, `-`, `'` and inner dots
|
||||
line break, and it does nothing when no word follows. Only one word is added,
|
||||
so a multi-word town ("Frankfurt am Main") keeps its tail.
|
||||
|
||||
`extend_to_previous_word` is the mirror for the word *before* a detection. A
|
||||
name model often tags the surname with high confidence and scores the first
|
||||
name in front of it below `min_score` (`Gast Greta [REDACTED:ner:LASTNAME]`):
|
||||
|
||||
```yaml
|
||||
pii_detection:
|
||||
extend_to_previous_word:
|
||||
- LASTNAME # "Gast Greta Waldmeister" -> "Gast [REDACTED:ner:LASTNAME]"
|
||||
```
|
||||
|
||||
Here the preceding word must consist of letters (plus `-` and `'`): a word
|
||||
with digits - an amount, a date, a room number - is never taken, nor a
|
||||
protected term. The search skips spaces and tabs but never a line break.
|
||||
|
||||
#### Protected terms
|
||||
|
||||
The opposite failure is a value the model tags although it must reach the
|
||||
|
||||
Reference in new issue
Block a user