tiktoken

toolopen source · MIT

Fast BPE tokenizer for OpenAI models: count tokens before they hit the context window.

Why is it popular?

Signals

  • 19,670

    Up 0.9% in 28 days

    api.github.com ↗ · updated Nov 16

  • 54,034,573/wk

    Up 12.7% in 28 days

    pypistats.org ↗ · updated Nov 16

  • 0

    History still building

    hn.algolia.com ↗ · updated Nov 16

  • 0

    History still building

    hn.algolia.com ↗ · updated Nov 16

How it stacks up

tiktoken next to similar tools, using only numbers both have. Bigger isn't always better: read each number's meaning first.

Relationship map

No mapped relationships yet