Source symbols: src/transformers/modeling_flash_attention_utils.py

Deterministic, value-free symbol metadata extracted from public source. Implementations and literal values are not copied into this page.

• Revision: main@9a0fe3f5dd36ffe1888133f09eb03f1eb14b8a6e · 2026-07-20T12:47:48Z
• Declarations: 18
• Referenced calls: 64

Declared symbols

• flash_attn_supports_top_left_mask — def at line 43; references is_flash_attn_2_available, is_flash_attn_3_available, is_flash_attn_4_available, is_npu_fa2_top_left_aligned_causal_mask, is_torch_npu_available, is_torch_xpu_available, is_rocm_platform
• is_flash_attn_available — def at line 53; references is_flash_attn_4_available, is_flash_attn_3_available, is_flash_attn_2_available, is_torch_npu_available, is_torch_xpu_available, is_rocm_platform
• _lazy_imports — def at line 143; references is_flash_attn_2_available, is_flash_attn_3_available, is_flash_attn_4_available, split_attention_implementation, is_torch_npu_available, get, load_and_register_attn_kernel, getattr, hasattr, ValueError, warning, signature
• _lazy_define_process_function — def at line 224; references signature
• lazy_import_flash_attention — def at line 248; references ValueError, _lazy_imports, _lazy_define_process_function, reshape, sum, nonzero, flatten, max, item, pad, cumsum, _index_first_axis, zeros, view, _get_unpad_data, arange, squeeze, unpad_input_func
• lazy_import_paged_flash_attention — def at line 275; references lazy_import_flash_attention, reshape, sum, nonzero, flatten, max, item, pad, cumsum, _index_first_axis, zeros, view, _get_unpad_data, arange, squeeze, unpad_input_func
• _index_first_axis — def at line 287; references reshape, sum, nonzero, flatten, max, item, pad, cumsum, zeros, view, _get_unpad_data, arange, squeeze, unpad_input_func
• _unpad_input — def at line 299; references sum, nonzero, flatten, max, item, pad, cumsum, _index_first_axis, zeros, view, _get_unpad_data, arange, squeeze, unpad_input_func
• _pad_input — def at line 332; references zeros, view, sum, nonzero, flatten, max, item, pad, cumsum, _get_unpad_data, _index_first_axis, arange, squeeze, unpad_input_func
• _get_unpad_data — def at line 351; references sum, nonzero, flatten, max, item, pad, cumsum, _index_first_axis, arange, squeeze, unpad_input_func
• _upad_input — def at line 379; references _get_unpad_data, _index_first_axis, arange, squeeze, unpad_input_func
• prepare_fa_kwargs_from_position_ids — def at line 458
• _prepare_from_posids — def at line 496; references contiguous, view, size, prepare_fa_kwargs_from_position_ids, arange, min, and, abs, sum, bool, warning_once, not
• _is_packed_sequence — def at line 534; references arange, min, and, abs, sum, bool, warning_once, not
• fa_peft_integration_check — def at line 550; references warning_once, not
• FlashAttentionKwargs — class at line 568; references not
• _process_flash_attention_kwargs — def at line 589; references not
• _flash_attention_forward — def at line 692; references lazy_import_flash_attention, fa_peft_integration_check, partial, size, _is_packed_sequence, all, _upad_input, str, clone, flash_varlen_fn, flash_kwargs, isinstance, pad_fn, _prepare_from_posids, reshape, view, flash_fn

File-level call references

• import — line 24
• get_logger — line 39
• is_flash_attn_2_available — line 44
• is_flash_attn_3_available — line 44
• is_flash_attn_4_available — line 44
• is_npu_fa2_top_left_aligned_causal_mask — line 49
• is_torch_npu_available — line 58
• is_torch_xpu_available — line 59
• is_rocm_platform — line 68
• find_spec — line 83
• replace — line 84
• get — line 84
• split_attention_implementation — line 162
• load_and_register_attn_kernel — line 190
• getattr — line 194
• hasattr — line 201
• ValueError — line 204
• warning — line 209
• signature — line 234
• partial — line 245
• _lazy_imports — line 265
• _lazy_define_process_function — line 270
• lazy_import_flash_attention — line 281
• reshape — line 295
• sum — line 316
• nonzero — line 318
• flatten — line 318
• max — line 320
• item — line 320
• pad — line 321
• cumsum — line 321
• _index_first_axis — line 324
• zeros — line 346
• view — line 348
• _get_unpad_data — line 420
• arange — line 438
• squeeze — line 442
• unpad_input_func — line 446
• cat — line 479
• tensor — line 482
• size — line 482
• diff — line 490
• contiguous — line 525
• prepare_fa_kwargs_from_position_ids — line 529
• min — line 545
• and — line 547
• abs — line 547
• bool — line 547
• warning_once — line 563
• not — line 641
• getenv — line 657
• isinstance — line 678
• is_tracing — line 678
• fa_peft_integration_check — line 738
• _is_packed_sequence — line 764
• all — line 765
• _upad_input — line 771
• str — line 777
• clone — line 778
• flash_varlen_fn — line 780
• flash_kwargs — line 786
• pad_fn — line 791
• _prepare_from_posids — line 796
• flash_fn — line 824

Source: src/transformers/modeling_flash_attention_utils.py