·github.com
GigaToken debuts as a drop-in tokenizer claiming up to 1,000x faster throughput than Hugging Face tokenizers
GigaToken is an open-source language model tokenizer that claims GB/s throughput and up to ~1,000x faster performance than Hugging Face tokenizers. The project offers compatibility...
read →