Ig6ZddlZddlZddlmZmZddlZddlmcmZ ddl m Z m Z e r6ddl mZmZmZddlmZmZdeej*ej,vZdej0d eej0ej0effd Zd ej0d ej0d ej0dej0def dZdZ ddej0dej0dej0dej0dedededeej0deedeededeedefdZy)N)OptionalTuple)is_flash_attn_2_availableis_flash_attn_greater_or_equal)index_first_axis pad_input unpad_input)flash_attn_funcflash_attn_varlen_func window_sizeattention_maskreturncd|jdtj}tj|j dj }|j j }tjtj|dtjd}|||fS)aq Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. )dimdtypeF)as_tupler)rr) sumtorchint32nonzeroflattenmaxitemFpadcumsum)rseqlens_in_batchindicesmax_seqlen_in_batch cu_seqlenss h/var/www/html/answerous/venv/lib/python3.12/site-packages/transformers/modeling_flash_attention_utils.py_get_unpad_datar$!s &))b )DmmN224uEMMOG*..0557u||$4!5;;OQWXJ  query_layer key_layer value_layer query_lengthct|\}}}|j\}} } } t|j|| z| | |}t|j|| z| | |}|| k(r't|j|| zd| |}|} |} |}nn|dk(rLd} t j |dztj |j} | dd}|jd}n|dd| df}t||\}}} } ||||| |f| |ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rr)rdeviceN) r$shaperreshaperarangerr+squeezer )r&r'r(rr) indices_k cu_seqlens_kmax_seqlen_in_batch_k batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_qs r# _upad_inputr:<sUN6E^5T2I|2(9:FQR]_mFnC Y .C  |$  56  r%c|jd|jd|jd}|jd|jd|jd}|jd|jd|jd}|j}tj|jd|j tj }tj||dk(tj|j|j tj f}|jdz}||||||f||ffS)aK This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cummulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cummulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rr)r+rr) viewsizerrr.r+rcattensorr)querykeyvalue position_idsr9 cu_seq_lens max_lengths r#prepare_fa2_from_position_idsrGs@ JJr5::b>5::b> :E ((2sxx|SXXb\ 2C JJr5::b>5::b> :E'')L \..q1,:M:MUZU`U`aI)) la' ( LL**,\5H5HPUP[P[ \ K!!#a'J 3y; *DzS]F^ __r% query_states key_states value_states is_causaldropoutrD softmax_scalesliding_windowuse_top_left_masksoftcap deterministicc | s|} n |xr|dk7} txr| duxr|jd| kD}|rd| | fini}tdr*| #tjj dddk(} | |d<| | |d <|U|jd }t |||||\}}}}}}|\}}|\}}t|||f||||||| d |}t||||}|S||dk7rtj|d d k\jsz|jd }t||||\}}}}}}|\}}|\}}t|||f||||||| d |}|j|d |jd|jd }|St||||f|| d|}|S)a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API attention_mask (`torch.Tensor`): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`float`): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_top_left_mask (`bool`, defaults to `False`): flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. softcap (`float`, *optional*): Softcap for the attention logits, used e.g. in gemma2. deterministic (`bool`, *optional*): Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled. rNr z2.4.1FLASH_ATTENTION_DETERMINISTIC01rQrPr)r7r1 max_seqlen_q max_seqlen_k dropout_prMcausalr)rr<)rMrY)_flash_supports_window_sizer,rosenvirongetr:r r rdiffallr>rGr=r )rHrIrJrr)rKrLrDrMrNrOrPrQrYuse_sliding_windows flash_kwargsr3r9rE max_seq_lensr7r1r8r2attn_output_unpad attn_outputs r#_flash_attention_forwardresbP 0|q0 $kd(BkzGWGWXYGZ]kGkI\MNN#CDacL%g.  JJNN+JCPTWWM(5 _%") Y!!''* Wb *lNLX T j, ; &1" l7C442     &%..'     19j,W H =  !la&7L^`AaefAf@k@k@m!&&q) Wt *lLX T j, ; &1" l7C44,     &%..'    "&&z2{7G7G7K[M]M]^`Mab   & *lG KXag kw  r%)gNNNFNN) inspectr[typingrrrtorch.nn.functionalnn functionalrutilsrrflash_attn.bert_paddingrr r flash_attnr r list signature parametersrZTensorintr$r:rGboolfloatrer%r#rvs  " LPPB"/48I8I8I/8Z8e8e3f"fELLU5<<WZ;Z5[6FF||FFLL F  FR/`r+/%)$(##v,,v v,,vLL v  v  vv5<<(vE?vSMvve_vvr%