tiktoken
Fast BPE tokenizer for OpenAI models: count tokens before they hit the context window.
Fast BPE tokenizer for OpenAI models: count tokens before they hit the context window.

CLI that converts a codebase into one prompt with a source tree, templates and token counting.
Website captured Oct 19
Latest collected value
Latest collected value
Latest collected value
Latest collected value
Latest collected value
Latest collected value
Latest collected value
| Signal | tiktoken 7d | tiktoken 28d | code2prompt 7d | code2prompt 28d |
|---|---|---|---|---|
| GitHub stars | ▲0.2% | ▲0.9% | ▲0.1% | ▲0.5% |
| GitHub forks | ▲0.7% | ▲1.7% | ▲0.2% | ▲0.2% |
| PyPI downloads (last week) | ▲5.5% | ▲12.7% | ▼43.2% | ▼45.8% |
| Hacker News stories (7 days) | — | — | — | — |
| Hacker News points (7 days) | — | — | — | — |
| GitHub open issues & PRs | ▲0.7% | ▲4.2% | ▲5.0% | ▲16.7% |
| GitHub contributors | ▲0.0% | ▲4.0% | ▲0.0% | ▲4.9% |