pytokens

A Fast, spec compliant Python 3.12+ tokenizer that runs on older Pythons.

Installation

pip install pytokens

Usage

python -m pytokens path/to/file.py

Local Development / Testing

• Create and activate a virtual environment
• Run pip install -r requirements-dev.txt to do an editable install
• Run pytest to run tests

Type Checking

Run mypy .

Create and upload a package to PyPI

Make sure to bump the version in setup.cfg.

Then run the following commands:

rm -rf dist
python -m build

Then upload it to PyPI using twine:

twine upload dist/*