mirror/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-06-05 15:05:56 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
421faa0263f28499e2ff9076ffca240a8fc262ca
ollama/tokenizer
History
Daniel Hiltgen ec9b4e9e47 tokenizer: fix multi-regex BPE offset handling (#15844) 
Use the current fragment offset when emitting unmatched spans during multi-regex BPE splitting. This avoids duplicating earlier prompt text and inflating token counts for multi-stage BPE tokenizers.
2026-04-27 14:14:27 -07:00
..
testdata
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
bytepairencoding_test.go
tokenizer: fix multi-regex BPE offset handling (#15844)
2026-04-27 14:14:27 -07:00
bytepairencoding.go
tokenizer: fix multi-regex BPE offset handling (#15844)
2026-04-27 14:14:27 -07:00
sentencepiece_test.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
sentencepiece.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
tokenizer.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
vocabulary_test.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
vocabulary.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
wordpiece_test.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
wordpiece.go
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00