Source symbols: scripts/tok_eval.py

Deterministic, value-free symbol metadata extracted from public source. Implementations and literal values are not copied into this page.

• Revision: master@92d63d4e8bb4df75c3b71618f31ddde2378b2bcd · 2026-07-04T18:18:25Z
• Declarations: 1
• Referenced calls: 13

Declared symbols

• print_comparison — def at line 203; references print

File-level call references

• strip — line 15
• next — line 147
• parquets_iter_batched — line 147
• join — line 148
• append — line 161
• from_pretrained — line 170
• get_tokenizer — line 174
• get_vocab_size — line 176
• encode — line 180
• decode — line 181
• len — line 185
• print — line 198
• print_comparison — line 242

Source: scripts/tok_eval.py