Installation

Requirements

• Python 3.7 or later
• Dependencies:
• • pandas
• • transformers
• • tqdm
• • rich

Install via pip

pip install unitok

Install from source

git clone https://github.com/Jyonn/UnifiedTokenizer.git
cd UnifiedTokenizer
pip install -e .