Source symbols: scripts/tok_train.py

Deterministic, value-free symbol metadata extracted from public source. Implementations and literal values are not copied into this page.

• Revision: master@92d63d4e8bb4df75c3b71618f31ddde2378b2bcd · 2026-07-04T18:18:25Z
• Declarations: 1
• Referenced calls: 23

Declared symbols

• text_iterator — def at line 28; references parquets_iter_batched, len, time, train_from_iterator, print, get_base_dir, join, save, encode, decode, get_vocab_size, set, encode_special, get_special_tokens, range, append, decode_single_token_bytes, tensor, open

File-level call references

• ArgumentParser — line 16
• add_argument — line 17
• parse_args — line 20
• print — line 21
• parquets_iter_batched — line 35
• len — line 38
• text_iterator — line 44
• time — line 48
• train_from_iterator — line 49
• get_base_dir — line 56
• join — line 57
• save — line 58
• encode — line 67
• decode — line 68
• get_vocab_size — line 76
• set — line 77
• encode_special — line 77
• get_special_tokens — line 77
• range — line 79
• append — line 81
• decode_single_token_bytes — line 85
• tensor — line 87
• open — line 89

Source: scripts/tok_train.py