"h7NddlmZddlZddlmZmZmZmZmZddl Z ddl m Z ddl m Zddl mZed ZGd d eZdnd Zdnd ZdodZ dpdZdqdrdZdsdZ dt dudZdvdZ dwdZ dwdZ dwdZdxdZdxdZ dydZ dxdZ!dzdZ"dzdZ#d{dZ$ d|dZ%d}dZ&d}d Z'd}d!Z(d}d"Z)d}d#Z*d~d$Z+d}d%Z,d}d&Z-d}d'Z.dd(Z/dd)Z0dd*Z1dd+Z2d}d,Z3d}d-Z4d}d.Z5d}d/Z6d}d0Z7d}d1Z8dd2Z9dd3Z:dd4Z;dd5Zdd8Z?dd9Z@dd:ZAdd;ZBdd<ZCdd=ZDdd>ZE d dd?ZFd@ZGdAZHdBZIdCZJdDZKdEZLdFZMdGZNdHZO ddIZP ddJZQddKZR ddLZSddMZTdNZUdOZV ddPZWddQZX ddRZYddSZZddTZ[ddUZ\ddVZ]ddWZ^ddXZ_ ddYZ`dZZa dd[Zbdd\Zc dd]Zddd^Zed_Zfdd`Zg ddaZhddbZiddcZjdddZkddeZlddfZmddgZnddhZoddiZpdjZqdqdkZrddlZsddmZty)) annotationsN)ListOptionalSequenceTupleTypeVar)ir)core)mathTceZdZfdZxZS)IncompatibleTypeErrorImplc||_||_d|jjzdz|jjz|_tt ||jy)Nzinvalid operands of type  and )type_atype_b__repr__messagesuperr__init__)selfrr __class__s U/var/www/html/sandstorm/venv/lib/python3.12/site-packages/triton/language/semantic.pyrz"IncompatibleTypeErrorImpl.__init__sX  2T[[5I5I5KKgUX\XcXcXlXlXnn  '7 E)__name__ __module__ __qualname__r __classcell__)rs@rrrsFFrrc|dvrtd|tj|j|tjS)Nrr r z+program_id axis must be 0, 1, or 2 but got ) ValueErrortltensorcreate_get_program_idint32axisbuilders r program_idr+s= 9FtfMNN 99W2248"(( CCrc|dvrtd|tj|j|tjS)Nr"z-num_programs axis must be 0, 1, or 2 but got )r#r$r%create_get_num_programsr'r(s r num_programsr."s= 9HOPP 99W44T:BHH EErc`|j}|j}|j}|j}||k(r ||kDr|S|S|tjjj k(r ||k\r|S|S|tjjj k(r ||k\r|S|St d|d|)Nzunexpected signedness r) int_bitwidthint_signednessr$dtype SIGNEDNESSUNSIGNED TypeError)a_tyb_tya_rankb_ranka_snb_sns rinteger_promote_implr<-s   F   F   D   D t|t0D0 $$-- -'t1T1 $$-- -'t1T1 ,TF%v> ??rc||k7rx|r||fn||f\}}|jj|jjkr6|r2|tjtjfvrtj S|S|j s|j rtjS|js|jrtj S|js|jr"|rtj StjS|js|jrR|rtj S|jr |jrtjStj S|jr'|jr||k(r|StjS|jr|jstd|d||rL|j|jk7r3td|jzdz|jzdzt!||S)Nunexpected type rzCannot use /, #, or % with x because they have different signedness;this is unlikely to result in a useful answer. Cast them to the same signedness.)kindvaluer$float16bfloat16float32is_fp64float64is_fp32is_fp16is_bf16is_fp8is_intr5r1rr<)r6 a_is_scalarr7 b_is_scalar div_or_mod scalar_ty tensor_tys rcomputation_type_implrQ=s k!/:d|t  9 >>  ! !Y^^%5%;%; ;yRZZ,EEzz!  ||~zz ||~zz ||~ :: ::  ||~ ::  <<>dlln;; zz {{}t|t33 ;;= *4&dV<==d))T-@-@@5 G'QTXTaTaTcckkl l d ++rct|tr3tj|j |tj St|t rd|cxkrdkrnntj}nld|cxkrdkrnntj}nMd|cxkrdkrnntj}n.d|cxkrdkrnntj}ntd|dtd ||| St|trnd }d d dzz}td|}|tdk(s|dk(s||k7s||cxkr|krnntj}ntj }td ||| St|tj"rt%|j&|St|tjr|S|rt)d|dt+|d|S)NllzNonrepresentable integer .r2r*g8g?r absinfzcannot convert z of type z to tensor) isinstanceboolr$r%get_int1int1intr'uint32int64uint64r#fullfloat __builtins__rDrF constexpr to_tensorrAr5type)xr* check_typer2 min_float32 max_float32abs_xs rrjrjos!Tyy))!,bgg66 As  Q  HHE a % IIE q 5 HHE a % IIE81=> >B88 Au  !QV+ U#A& E%L C< 6 % .; .JJEJJEB88 Ar|| $'** Aryy !/!Id1gYjIJJ Hrc|jrL|s t|||jr||k7r t|||jr t||yyN)is_ptrr is_floating)rr allow_ptr_as rcheck_ptr_type_implrvs[ }}+FF; ; ==?& 0+FF; ;    +FF; ; rct|tj}t|tj}|r|} t||}|r|} t||}|jj } |jj } t | | |t | | ||r| js| jst| || ||} |r dkr| js|r  dkr| jr td|rtd | |n t|| |}|rtd | |n t|| |}t|||\}}||fS)Nrz{Cannot perform a binary operation between an unsigned tensor and a negative scalar. Perform a explicit cast on one of them.rXrY)r^numbersNumberrjrkscalarrvrsrQis_int_unsignedr#rfcastbroadcast_impl_value)lhsrhsr* allow_lhs_ptr allow_rhs_ptrarithmetic_checkrN lhs_is_scalar rhs_is_scalar lhs_scalar rhs_scalar lhs_sca_ty rhs_sca_ty ret_sca_tys rbinary_op_type_checking_implrscsGNN3MsGNN3M W% W%JJ J > J > 1 1 3J xx##r)1R1RJJ  ## #      sBHHg &C sBHHg &C CeW -C,,.I '++I6AI,,.I '++I6AI  3 73]3 SZ5[]d eD  ''((I)J\J\I] ^C$W%rct|||dd\}}|jj}|jj}|jr|jr t d|jr@|js0||}}|jj}|jj}|jrDt j |j|j|j|jS|jrDt j |j|j|j|jS|jrX|rt|||tt j |j|j|j|jSt d|)NTzcannot add pointers togetherr>)rrkrzrsr5r$r% create_addptrhandlert create_faddrKradd create_add)inputotherrr*input_scalar_tyother_scalar_tys rrrs`/ugtTRLE5jj''Ojj''OO$:$:$<677(>(>(@eu**++**++yy..u||U\\JEJJWW  $ $ &yy,,U\\5<<H%**UU    !  ,UE7C Hyy++ELL%,,GTT &&78 99rc t|||dd\}}|jj}|jrNt j |j |jt||j|jS|jrDt j |j|j|j|jS|jrX|rt|||tt j |j|j|j|jStd|)NTFr>)rrkrzrsr$r%rrminusrt create_fsubrKrsub create_subr5rrrr*rOs rrrs/ugtUSLE5 !!Iyy..u||U5'=R=Y=YZ\a\f\fggyy,,U\\5<<H%**UU      ,UE7C Hyy++ELL%,,GTT &yk2 33rct|||\}}|jj}|jrDt j |j |j|j|jS|jrX|rt|||tt j |j|j|j|jStd|Nr>) rrkrzrtr$r% create_fmulrrKrmul create_mulr5rs rrrs/ugFLE5 !!Iyy,,U\\5<<H%**UU      ,UE7C Hyy++ELL%,,GTT &yk2 33rc 6t|||dddd\}}|jj}|jj}|jr|j rt |||}n|j r|jrt |||}n|j rG|j r7t |t j|}t |t j|}nc|jrE|jr5|j|jkDrt |||}nt |||}ntd|t j|j|j|j|jSNFTr>) rrkrzrtrKr|r$rDfp_mantissa_widthr5r% create_fdivr)rrr*rrs rtruedivrsH/ugueUY[_`LE5jj''Ojj''O""$)?)?)AUOW5    !o&A&A&CUOW5    !o&<&<&>UBJJ0UBJJ0  $ $ &?+F+F+H  , ,/P/P P9E9E*?*;<== 99W((u||Dejj QQrc Zt|||dddd\}}|jj}|jj}|jr|jrt ||}t |||}t |||}|j rDtj|j|j|j|jStj|j|j|j|jStd|r) rrkrzrKr<r| is_int_signedr$r% create_sdivr create_udivr5)rrr*rrret_tys rfloordivr0s/ugueUY[_`LE5jj''Ojj''OO$:$:$<%oGUFG,UFG,    !99W00u||LejjY Y99W00u||LejjY Y &&78 99rc d|jj}|jj}|jr|js tdt |||dddd\}}|j |j |j }tj||jS)Nz4both operands of fdiv must have floating scalar typeFT) rkrzrtr5rrrr$r%)rr ieee_roundingr*rrrs rfdivr?sjj''Ojj''O  & & (0K0K0MNOO/ugueUZ\`aLE5   ellELL 9C 99S%** %%rc $t|||dddd\}}|jj}|jj}|jr?t j t ||d||}t|t||d|d|}|S|jr|j|jk7r3td|jzdz|jzdz|jrDtj|j!|j"|j"|jStj|j%|j"|j"|jStd|)NFT_builderz Cannot mod z by r?r>)rrkrzrtr floorrrrrKr1r5rrr$r% create_sremr create_urem)rrr*rOrrrs rmodrJsW/ugueUY[_`LE5 !!Ijj''O 4ueW=P%UE494I      # #'E'E EMI,>,>,@@6IOLdLdLffjoop p  " " $99W00u||LejjY Y99W00u||LejjY Y &yk2 33rcft|||\}}|j}|jr|tjj k(rDtj |j|j|j|jS|tjjk(rDtj |j|j|j|jStd||jrDtj |j|j|j|jS|jrDtj |j!|j|j|jSt#d|NzUnexpected propagate_nan Unexpected dtype )rr2rtr$ PropagateNanALLr%create_minimumfrrkNONEcreate_minnumfr#r create_minsir{ create_minuir5rly propagate_nanr*r2s rminimumrf3 '1g 6DAq GGE  BOO// /99W44QXXqxxH!&&Q Q boo22 299W33AHHahhGP P8HI I    yy--ahhA166JJ    yy--ahhA166JJ+E7344rcft|||\}}|j}|jr|tjj k(rDtj |j|j|j|jS|tjjk(rDtj |j|j|j|jStd||jrDtj |j|j|j|jS|jrDtj |j!|j|j|jSt#d|r)rr2rtr$rrr%create_maximumfrrkrcreate_maxnumfr#r create_maxsir{ create_maxuir5rs rmaximumrxrrcXt|||\}}t|||\}}t|||\}}|j}|jrPtj|j |j |j |j ||jStd|d)Nrz(. Only floating point clamp is supported) rr2rtr$r% create_clampfrrkr5)rlminmaxrr*r2s rclamprs+Cg>HC )!S' :FAs )!S' :FAs GGE yy..qxxSZZQ^_abagaghh+E72Z[\\rc:t|||\}}|jj}|jj}|jr|js t ||t ||}||k7r t |||}||k7r t |||}||fSrr)rrkrzrKrr<r|)rrr* input_sca_ty other_sca_tyrs rbitwise_op_type_checking_implrs/ugFLE5::$$L::$$L    (;(;(=' lCC%lLAJ\!UJ0\!UJ0 %<rct|||\}}tj|j|j|j|j Srr)rr$r% create_andrrkrrr*s rrr>0wGLE5 99W'' ellCUZZ PPrct|||\}}tj|j|j|j|j Srr)rr$r% create_orrrkrs ror_rs>0wGLE5 99W&&u||U\\BEJJ OOrct|||\}}tj|j|j|j|j Srr)rr$r% create_xorrrkrs rxor_rrrc|jjs t|tjd|}|jjs t|tjd|}t |||SNra)rkis_int1bitcastr$r2rrs r logical_andrs_ ::   rxx/9 ::   rxx/9 ug &&rc|jjs t|tjd|}|jjs t|tjd|}t |||Sr)rkrrr$r2rrs r logical_orrs_ ::   rxx/9 ::   rxx/9 ueW %%rc|jjs t|tjd|}t ||Sr)rkrrr$r2invert)rr*s rnot_rs6 ::   rxx/9 % !!rct|||\}}tj|j|j|j|j Srr)rr$r% create_lshrrrkrs rlshrr>0wGLE5 99W((u||Dejj QQrct|||\}}tj|j|j|j|j Srr)rr$r% create_ashrrrkrs rashrrrrct|||\}}tj|j|j|j|j Srr)rr$r% create_shlrrkrs rshlrrrc|SrrrX)rs rplusrs Lrc|jj}|jrtd|j zdzt j |j|j||}t||d|S)Nz$wrong type argument to unary minus ()T) rkrzrsr#rr$r%get_null_valueto_irr)rr*r_0s rrrst::$$L?,BWBWBYY\__`` 7)),*<*?? GE ]]288U +F 99W..uc:F CCrct|tjr.|jjdk(sJdt |||}nj| t d|dk(r!|j|j|}n!t|d|j}||}tj||}t|||S)Nr zonly accepts size-1 tensorz2dtype must be specified when value is not a tensorrget_) r^r$r%numelrAr|r#rrgetattrnamesplat)rrAr2r* get_value_fns rrfrfks%#{{  A%C'CC%UE7+ =QR R A:**5;;w+?@E"7d5::,,?@L 'E %' w ''rc|jjrJdt|dk(r|Stj|j |}tj |j|j||S)NzCannot splat a block tensorr) rkrlenr$rr2r% create_splatr)rArr*rs rr>r>sdzz""$C&CC $ 5zQ ]]5;; .F 99W))%,,> GGrcd}|D]}||z} |jj|k7r tdtj|jj |}tj |j|j|||S)Nr z:reshape() cannot change total number of elements in tensor) rkr;r#r$rrzr%create_reshaper)r dst_shape can_reorderr*r;srs rreshaperHs| E    zz5 UVV ]]5::,,i 8F 99W++ELL)[QSY ZZrc|jDcgc]}tj|}}|j|d|jj st |||Stj|jj|}tj|j|j||Scc}w)Nr )rr*) rr$_constexpr_to_valueinsertrkrr>rrzr%create_expand_dimsr)rr)r*rlrErs r expand_dimsrMs49KK@q''*@I@ T1 ::   U)W== ]]5::,,i 8F 99W// dCV LLAsCcL|sJdt|jdk(sJtj|jj |jd|jdzg}tj |j|j|j|S)Nz;current implementation of `cat` always may reorder elementsr r) rArr$rrkrzr% create_catr)r~rrFr*ret_types rcatrQs| UUU; syy>Q  }}SXX__syy|ciil/J.KLH 99W'' CJJ? JJrct|||\}}|jgk(}|rt|d|}t|d|}t|jdtj rt j d}nd}|j|gz}t j |jj|}t j|j|j|j|}|rt|dgd|}|S)Nrr FrFr*) r}rrMr^r$rirrkrzr% create_joinrrH)abr* was_rank_1two new_shaperPrs rjoinr[s 1g .DAqBJ 1g & 1g &!''"+r||,ll1o3%I}}QVV]]I6H ))G''!((;X FCcA3E7C Jrct|jdkDsJtj|jddk(sJ|jdd}tj|j j |}|j|j\}}tj||tj||fS)NrrSr ) rArr$rJrrkrz create_splitrr%)rVr*rZrPoutLHSoutRHSs rsplitr`s L1    " "1772; /1 45 4 I}}QVV]]I6H))!((3NFF &(# &(# rct|jt|k7r tdtd|Dt t t|k7rtd|t j|jj|Dcgc]}|j|c}}t j|j|j||Scc}w)Nz5permute dims must have the same length as input shapec3FK|]}tj|ywrr)r$rJ).0ds r zpermute..s 6Ab$$Q' 6s!z?permute dims must be a permutation of 0, 1, ..., n-1, but were ) rArr#sortedlistr7r$rrkrzr% create_transr)rdimsr*rdrPs rpermuterjs 5;;3t9$PQQ 6 66$uSY?O:PPZ[_Z`abb}}UZZ..0NAQ0NOH 99W))%,,=x HH1OsC c |jjsPtj|j|}tj|j |j ||S|jj}t|t|k7rtd|d|||k(r|St|D]0\}}|||k7s|dk7std||d|d|d|d| tj|jj|}tj|j|j ||S)Nz!Cannot broadcast, rank mismatch: z, r z3Cannot broadcast, the expanded size of the tensor (z ) must match the existing size (z) at non-singleton dimension : ) rkrr$rr%rBrget_block_shapesrAr# enumeraterzcreate_broadcast)rrr*r src_shapeiitems rbroadcast_impl_shaperss; ::   uzz51yy--ellEBFKK ++-I 9~U#tdt!| zdzt!| zdzt!| z||k7rPtj|j|} tj |j#|j|| }||k7rPtj|j|} tj |j#|j|| }||fS)Nrr z?Cannot make_shape_compatible: incompatible dimensions at index rlr)rkrr$rrzrr%rBrrmrAr7rLrnappendr#strro) r~rr*lhs_tyrhs_ty lhs_shape rhs_shape_ ret_shaperqleftrightrs rr}r}sQ XXF XXF!2v}}fll;ii,,SZZ9P9P9RSU[\V 8OS__ 6??#4v}}fll;ii,,SZZ9P9P9RSU[\N 8OK  v0++- ++- y>C N *3y>3y>: 6ii : :3::q I " fmmaS9_ MO"335  6 ^c)n ,3y>3y>: 6ii : :3::q I " fmmaS9_ MO"335  6 9~Y///  + aGAtaLEqy  '1*%4-  & "-/21v"68<"=?B4y"IKR"SUXY^U_"`aa a  !]]6==)? ? 99W++ELL&,,w:OPRX YYrcf|j}t|tjr |j}t|tjr |j}|j r8tj |j|jj}||k(r|S|j}|j}t|}d}|jrf|jrV|j|jkr=|tjj}nH|tjjk7r+d}n(|&tdt!|zdzt!|z|j#s|j#r<|j$j'd Jd|j$d||||S|j)r|js"|jr|j)s|r@tj*|j-|j.|j1|||S|j3r|j5r |j7r6|j5s&t9t9|tj:|||S|jxr+|jxr|j|jkD}|r?tj*|j=|j.|j1||S|jxr+|jxr|j|jk} | r?tj*|j?|j.|j1||S|jAr |jAr|jB|jBk7s|jD|jDk7r|jGxr|jI } |jIrW|jJj1|} tj*|jM| |jJ} tO|| |Stj*|jQ|j.|j1|| |S|jSr|jAr|jIrW|jJj1|} tj*|jM| |jJ} tO|| |S|jGr?tj*|jU|j.|j1||Stj*|jW|j.|j1||S|jAr|jSr|jIs|jGs?tj*|jY|j.|j1||Stj*|j[|j.|j1||S|j]r|jAr|jB} | dk(r?tj*|j_|j.|j1||S| d k(rWtOt9|tj`|tj*|jcd tj`|S|jArO|j]r?tj*|je|j.|j1||S|j]rO|j]r?tj*|jg|j.|j1||SJd |d |) NFTz]fp_downcast_rounding should be set only for truncating fp conversions. Source scalar type is z and destination type is convert_custom_typesz0target doesn't provide conversion for this type.rrr rz cannot cast z to )4rkr^r$rirArrrzrmrrtrr rrr#rv is_fp8e4b15 codegen_fnsgetrJr%create_fp_to_fprrrHrGrIr|rDcreate_fp_trunc create_fp_extrKr0r1ris_boolr2rr/create_int_castis_standard_floatingcreate_fp_to_sicreate_fp_to_uicreate_ui_to_fpcreate_si_to_fprscreate_ptr_to_intrdrcreate_int_to_ptrr)rrr*fp_downcast_roundingrrruse_custom_rounding truncate_fpext_fp sign_extendtyr bitwidths rr|r|?s ZZF&",,'& 5399 v}}ejj.I.I.KL  JJ11EFJ$:$:%  ' '**G*G G  '@P@P@U@U)= !R%5%5%:%: :RVW>W[e[t[t>t ..0M9K9K9M5M    ""7+B711"5u{{CBUB0 099W44U\\6<->-@    ""7+B711"5u{{CBUB0 0  % % '99W44U\\6<>@    z'?'?'A99W44U\\6<99W66u||V\\RYEZ[]cd d q=T%7;RYYwGXGXYZG[]_]e]e=fhop pz002yy225<<gAVWY_``z002yy// fll7>STV\]]4LtF8445rc"tjj}|rr|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj }|St d|d|S)Nz.ca.cgz.cvCache modifier  not supported)r CACHE_MODIFIERrCACGCVr#cache_modifiercaches r_str_to_load_cache_modifierrs    " "E U "%%((E L u $%%((E L u $%%((E L~.>nMN N Lrcdtjj}|r|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj }|S|dk(rtjj }|Std|d|S)Nz.wbrz.csz.wtrr)r rrWBrCSWTr#rs r_str_to_store_cache_modifierrs    " "E U "%%((E Lu $%%((E L u $%%((E L u $%%((E L~.>nMN N Lrctjj}|rQ|dk(rtjj}|S|dk(rtjj}|St d|d|S)N evict_last evict_firstzEviction policy r)r EVICTION_POLICYNORMAL EVICT_LAST EVICT_FIRSTr#)eviction_policyevictions r_str_to_eviction_policyrsu!!((H l *))44H O  -))55H O//@OP P Orcd}|rQ|dk(rtjj}|S|dk(rtjj}|St d|d|S)NzeronanzPadding option r)r PADDING_OPTIONPAD_ZEROPAD_NANr#)padding_optionpaddings r_str_to_padding_optionrshG V #''00G N u $''//G N~.>nMN N Nrcdtjj}|r|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj }|St d|d|S)Nacquirereleaseacq_relrelaxedMemory semantic r)r MEM_SEMANTICACQUIRE_RELEASEACQUIRERELEASERELAXEDr#) sem_optionsems r _str_to_semrs // ) )C  "//))C J9 $//))C J 9 $//11C J 9 $//))C J/ |>JK K Jrc"tjj}|rr|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj}|St d|d|S)Ngpuctasysrr)r MEM_SYNC_SCOPEGPUCTASYSTEMr#) scope_optionscopes r _str_to_scopers    ! !E 5 %%))E L U "%%))E L U "%%,,E L/ ~^LM M Lrc~|rt|ds|g}|Dcgc]*}t|tjr |jn|,}}|D]+}t|t rd|cxkrt |kr(JJt |dkDsJt |t t|k(sJdt|Sycc}w)N__iter__rz'Duplicate dimension in `boundary_check`rX) hasattrr^r$rirArbrAsetrf)boundary_check block_shapeelemdims r_canonicalize_boundary_checkrs~z2,-N]klUY 4(F$**DPll! HCc3'A,Gs;7G,G GG,G GG H>"Q&&&>"c#n*=&>>i@ii>n%% ms/B:c || td|jjj} | tjk7sJd| j r(|t jjk(r td|jj} t|| j}tj|j|j|||||| S)NK`mask` and `other` arguments cannot be specified for loading block pointers3`tl.int1` should be rewrited in `tl.make_block_ptr`z@Padding option `nan` is not supported for integer block pointers)r#rk element_tyr$rarKr rrrrmr%create_tensor_pointer_loadr) ptrmaskrrrrr is_volatiler*elt_tyrs r_load_block_pointerrs 5,fgg XX + +F RWW SSS  }}7b&7&7&?&??[\\XX F2.&BYBYB[\N 99**3::~wPUW_almou wwrc T|jjjs'td|jj d| | td|s|r td|jj sN|r%|jj r td|r%|jj r td|jj rN|%t ||jj|}|%t ||jj|}|jj} | j} | tjk(} | r=tj} tj| | j} t|| |}| t|| |}|jj r1|jj} tj| | } n| } |3tj |j#|j$|||| }nLtj |j'|j$|j$|r |j$nd|||| }| rt|tj|}|S)NUnsupported ptr type z in `tl.load`z)`other` cannot be provided without `mask`z`padding_option` or `boundary_check` argument is not supported for loading a tensor ofpointers or loading a scalar. Because the compiler does not know the boundary; please use block pointers (defined by `make_block_ptr`) insteadEMask argument cannot be block type if pointer argument is not a blockzFOther argument cannot be block type if pointer argument is not a block)rkrzrsr#rrrsrmrr$raint8 pointer_type address_spacer|rr% create_loadrcreate_masked_load)rrrrrrrrr*ptr_tyrrrrrs r _load_legacyr*sA 88?? ! ! #01B1B1D0E]STT |)DEE.TU U 88    DII&&(de e UZZ((*ef f xx  'chh.G.G.I7SD  (0I0I0KWUEXX__F   FG)=)=>3( UFG, xx))+vu- |ii++CJJxUW]^ii  & &szz4;;PU [_afhp'2 45;=3) Jrc  t|} t|} t|} |jj r7|jj j rt||||| | | || St||||| | | || Srr) rrrrkrsrrrr) rrrrrrrrr*rrrs rloadrhs ( 7E&7H$^4G xxSXX0099;"3e^WeU]_jlsttCunguhXcelmmrc t||d}|j|j||j|t |t |}t j||SNF require_i64)_convert_to_ir_valuescreate_descriptor_loadrrrrr$r%)desc_ptroffsetsrrrkr*rls rdescriptor_loadr xsW#GW%HG&&xGAT'B>'R'>'O QA 99Q rct||d}tj|j|j|j|tj Sr)rr$r%create_descriptor_storervoid)r rAr r*s rdescriptor_storers@#GW%HG 99W44X__ellT[\^`^e^e ffrc |r"|djtjk(sJtj| j |j |j |D cgc]} | j c} |D cgc]} | j c} |D cgc]} | j c} |D cgc]} | j c} |||| tj Scc} wcc} wcc} wcc} w)Nr)r2r$rdr%create_tensormap_createrr) r global_addressbox_dim global_dim global_strideelement_stride elem_typeinterleave_layout swizzle_mode fill_moder*rls rtensormap_creaters a 0 6 6"(( BB B 99'' OO  ! !& '!QXX ') *!QXX *, -!QXX -- .!QXX .        ( * - .sC2C C$C$c|tj|j|jtjSrr)r$r%#create_tensormap_fenceproxy_acquirerr)r r*s rtensormap_fenceproxy_acquirers) 99W@@QSUSZSZ [[rc | td|jjj}|jj s t |||}|jj sJd||jjk(s&Jd|d|jjd|jjj|jjk(s@Jd|jjjd|jjd|jjj}|t jk7sJdt||}t|||}t j|j|j|j|||t jS) Nrz-Value argument must be block type or a scalarz Block shape(z) and value shape(z ) mismatchzBlock element type(z) and value element type(r)r#rkrrmrrsr$rarr|r%create_tensor_pointer_storerr) rvalrrrrr*rrs r_store_block_pointerr#s fgg((%%668K 88   "3 W= 88   O OO  #((33 ] k]"4SXX5N5N5P4QQ[\]  88   ) )SXX-@-@ @qDWX[X`X`XkXkXvXvWwxQRURZRZReReQffpCqq @ XX + +F RWW SSS 2.+NN sFG $C 99W88SZZQ_afhpqWW rc |jjjs'td|jj d|r td|jj sL|jj r td|r%|jj r td|jj rLt ||jj|}|%t ||jj|}|jj}|j}|tjk(r=tj}tj||j}t|||}t|||}|sJtj|j!|j"|j"||tj$S|jjj's tdtj|j)|j"|j"|j"||tj$S)Nrz in `tl.store`z`boundary_check` argument is not supported for storing a tensor of pointers or storing a scalar. Because the compiler does not know the boundary; please use block pointers (defined by `make_block_ptr`) insteadzFValue argument cannot be block type if pointer argument is not a blockrz"Mask must have boolean scalar type)rkrzrsr#rrrsrmrr$rarrrr|r% create_storerrrcreate_masked_store) rr"rrrrr*rrs r _store_legacyr's 88?? ! ! #01B1B1D0E^TUUAB B 88    88   ef f DII&&(de e xx"3(A(A(CWM  'chh.G.G.I7SD XX__F   F)=)=>3( sFG $C yy--cjj#**eXVXZX_X_`` 99   # # %=>> 99W00SZZV[]efhjhoho pprc t|}t|}|jjs$|jjjr t d|jj r5|jjjrt|||||||St|||||||S)N"Cannot store to a constant pointer) rrrkis_constrzr#rsrrr#r') rr"rrrrr*rrs rstorer+s ) 8E&7H xxchhoo668=>> xxSXX0099;#CdNE8U\]]S#t^UHgVVrc Bt|}t|}|jjj}|j dvr t dtj|j|j|j|j|||jS)N)r1rz9atomic_cas only supports elements with width {16, 32, 64}) rrrkrzrrr#r$r%create_atomic_casr)rcmpr"rrr*rs r atomic_casr0s| c C % E++J$$L8TUU 99W..szz3::szzSVX]^`c`h`h iirc|jjjs&td|jj z|jj s$|jj j r td|jjj }|tjur|dk7rtd|zdz|tjtjtjtjfvrtd|zdzt|z|jjrN|%t||jj!|}|%t||jj!|}t#||jjj |}|s|j%d}tj}|jjrf|j'||jj!}tj(tj|jj!}tj*||}|||fS)Nz)Pointer argument of store instruction is r)ratomic_z does not support fp16z does not support T)rkrzrsr#rr*rr$rBrarint16rCrvrrsrmr|r`rBrr%)rr"ropr*rmask_irmask_tys ratom_red_typechecking_implr7s 88?? ! ! #DsxxGXGXGZZ[[ xxchh11::<=>>++JRZZB%KR*BBCCbggrww"++>>R*>>ZPQQ xx  'chh.G.G.I7SD ?&sCHH,E,E,GQC sCHHOO.. 8C ""4('' 88   **7CHH4M4M4OPGmmBGGSXX-F-F-HIGyy'* T>rc t|||d|\}}}t|}t|}|jj}|j r|j rjtj|jtjj|j|j|j|||jStj|jtjj|j|j|j|||jS|tjtj hvrt#d|t%gd||}|tjk(rtj&ntj(}t+|||} t+|tj,|d|} |tjk(rtj.ntj0} t+|| |} t+|tj,| d|} t3|||}t5|||}tj|jtjj| j| jt7|||j||| j}tj|jtjj8| j| jt7|||j||| j}t;||||}t+|||S)Nrz#atomic_max not supported for dtype r]r )r7rrrkrzrKrr$r%create_atomic_rmwr ATOMIC_OPMAXrUMAXrDrFr5rfr'rdrrrcrerr"rUMINwhererr"rrrr*sca_tyri_typei_vali_ptrui_typeui_valui_ptrposnegpos_retneg_retrs r atomic_maxrK,/S$wONCd c C % E XX__F }}    !99))",,*:*:CJJ TXT_T_adfklnqnvnvx x99))",,*;*;SZZUYU`U`beglmorowowy y bjj"**--=fXFGG C )D2::-RXX288F C )E C3W =E!RZZ/biiRYYG S'7 +F S"//'15w ?F T7 +C Cw 'Cii!!",,"2"2ELL%,,"&tS'":"A"A3 OPUPZPZ\Gii!!",,"3"3V]]FMM"&tS'":"A"A3 OPVP[P[]G Wgw /C 3 ((rc t|||d|\}}}t|}t|}|jj}|j r|j rjtj|jtjj|j|j|j|||jStj|jtjj|j|j|j|||jS|tjtj hvrt#d|t%gd||}|tjk(rtj&ntj(}t+|||} t+|tj,|d|} |tjk(rtj.ntj0} t+|| |} t+|tj,| d|} t3|||}t5|||}tj|jtjj| j| jt7|||j||| j}tj|jtjj8| j| jt7|||j||| j}t;||||}t+|||S)Nrz#atomic_min not supported for dtype r]r )r7rrrkrzrKrr$r%r9r r:MINrr=rDrFr5rfr'rdrrrcrerr"rr<r>r?s r atomic_minrOSrLrc t|||d|\}}}t|}t|}|jj}|j rt jjnt jj}tj|j||j|j|j|||jS)Nr)r7rrrkrzrtr r:FADDADDr$r%r9r)rr"rrrr*r@r4s r atomic_addrSzs/S$wONCd c C % E XX__F$002   8H8HB 99W..r3::szz4;;X[]bcehemem nnrc (t|||d|\}}}t|}t|}tj|j t jj|j|j|j|||jS)Nand) r7rrr$r%r9r r:ANDrrkrr"rrrr*s r atomic_andrXx/S$wONCd c C % E 99W..r||/?/?SZZY]YdYdfikpqXX rc (t|||d|\}}}t|}t|}tj|j t jj|j|j|j|||jS)Nor) r7rrr$r%r9r r:ORrrkrWs r atomic_orr]sv/S$gNNCd c C % E 99W..r|| CJJX\XcXcehjopXX rc (t|||d|\}}}t|}t|}tj|j t jj|j|j|j|||jS)Nxor) r7rrr$r%r9r r:XORrrkrWs r atomic_xorrarYrc (t|||d|\}}}t|}t|}tj|j t jj|j|j|j|||jS)Nxchg) r7rrr$r%r9r r:XCHGrrkrWs r atomic_xchgresx/S$PNCd c C % E 99W..r||/@/@#**cjjZ^ZeZegjlqrXX rc|j|jjvs!Jd|jjd||j}|dk(rd}t t j |S)Nzinput_precision must be one of z. Got TF32X3TF32x3)lowerrallowed_dot_input_precisionsupperr<r INPUT_PRECISION)input_precisionr*s r_str_to_dot_input_precisionrnsx  "goo&R&R Rp )'//*V*V)WW]^m]nop R%++-O("" 2%% 77rc  |jjr|jjsJ|jjr|jjrn|jtj tj tjtjtjfvsJd|j|jtj tj tjtjtjfvsJd|j|j|jk(s!Jd|jd|j|jjs|jjr6t|tj|}t|tj|}||jj}t||}t|j }t|j }||cxk(rdk(s1n||cxk(rdk(s$nJd|j d|j d |j d j"|j d j"k(sVJd |j d |j d|j d j"d|j d j"d |j$j'dJd|j$d|j|j} |j d j"| dk\r>|j d j"| dk\r|j d j"| dk\sJd| dd| dd| d|jj(j+rP|jj(tj k(sJd|j-d} tj.} n|j1r t3d|jj(j5s$|jj(j1r"|j7d} tj} n4|j9r|j;dn|j7d} |} |jj d } |jj d } |jj d }|dk(r|jj dnd}t j<| |r|| | gn| | g}||j?| |r|| | gn| | g}n|j@}|j|k(sJ|N|jjr1|jjr|jjB}nNd}nK|jjr1|jjr||kDrt3d|d|d t jD|jG|j@|j@||||S)NzUnsupported lhs dtype zUnsupported rhs dtype z&Both operands must be same dtype. Got rr +Both inputs must be either 2D or 3D; (lhs: vs rhs: rrSzFirst input shape (z) and second input shape z= are not compatible for matmul (second index of first shape (z0) must be equal to first index of second shape ( min_dot_sizez2target doesn't provide lower shape bounds for dot.rr zInput shapes should have M >= z, N >= z and K >= zonly int8 supported!zhout_dtype=bfloat16 is unsupported. Please use out_dtype=float32/float16 and cast with `.to(tl.bfloat16)`zmax_num_imprecise_acc (z) must be <= K ()$rkrr2rJr$ruint8rBrCrDrr|rdefault_dot_input_precisionrnrArrArrrzrK get_int32r'rIr#rGget_fp32rHget_fp16rrBrmax_num_imprecise_acc_defaultr% create_dot)r~raccrmmax_num_imprecise_acc out_dtyper*lhs_rankrhs_rankrtr  ret_scalar_tyMNKBr acc_handles rdotrs 88   388#4#4#66 6 yycii..0 yyRWWbhh BKKZZ)) O,B399+*N O)yyRWWbhh BKKZZ)) O,B399+*N O)yyCII%k)OPSPYPY{Z_`c`i`i_j'kk% yy#))"7"7"93 G,3 G,!//EE1/7KO399~H399~H x $1 $H(A(AREpqtqzqzp{|EFIFOFOEPPQDRR A 99R=  #)) # E q( 3LSYYKXUVYV_V_`bVcViViUjjZ[^[d[deg[h[n[nZoopqq     " "> 2 >t@tt >67&&~6sxxJL 99R=  ,q/ 1ciim6I6I\Z[_6\ IIbM  <? 2r,\!_,=W\RS_DUU_`lmn`o_p qr 3 xxxx"'')A+AA)   q !     vx x  "chhoo&=&=&?   a  $-$5$5$7W  a W=M=Ma=P!  rA rA rA%]qA ]]=q1a)q!f EF {))"1q!Qi1a&I ZZ xx6!!!$ 99   #))"2"2"4$+OO$Q$Q !$% ! 99   #))"2"2"49NQR9R67L6MM]^_]``abc c 99W'' CJJ O]rs rcV|dk(rtjjS|dk(rtjjS|dk(rtjjS|dk(rtjj S|dk(rtjj Std|d)Ne4m3e5m2e2m3e3m2e2m1zInvalid float format: rW)r F8F6F4TYE4M3E5M2E2M3E3M2E2M1r#) float_formats r_str_to_fp_typersv{{v{{v{{v{{v{{ -l^1= >>rc |jjr|jjsJt|j} t|j} | | cxk(rdk(s1n| | cxk(rdk(s$nJd|jd|jdt |} t |} |dvs Jd||dvs Jd |t |t jxr|jdu} | sJd |jjd }|jjd d\}}|d k(rdnd }|||jjdzk(s"Jd|jd|jd|dk\s Jd|| dk(r|jjdnd}t j||r|||gn||g}|jd}||j||r|||gn||g}n|j}|j|k(sJ| rdn |j}t j|j|j|j| |j|| ||S)Nr rprqrrr)rrrzNYI: lhs_format )rrzNYI: rhs_format zNYI: rhs_scale not supportedrsrr rSzCReduction dimension should pack the same number of elements; (lhs: rz!scaled_dot NYI for K < 64. Got K=r)rkrrArrr^r$rirArrxrBrr%create_dot_scaled)r~ lhs_scale lhs_formatr rhs_scale rhs_formatr|r~r*rrlhs_format_enumrhs_format_enumrhs_scale_is_nonerrrPACKEDrrr rrhs_scale_handles r dot_scaledrs 88   388#4#4#66 6399~H399~H x $1 $H(A(AREpqtqzqzp{|EFIFOFOEPPQDRR A%j1O%j1O 1 1R5Ej\3RR 1 ) )J-=j\+JJ )"9bll;W SW@W <<<  rA 88>>"# DAq&QAF     tRSVS\S\R]]fgjgpgpfqqr st 7:8aT::7%]qA ]]91q!Qi1a& AF   ! B {))"1q!Qi1a&I ZZ xx6!!!0ti6F6F 99!!#**i.>.>QTQ[Q[]m"1: ?@F HHrc|jtjk7r"tjd|jt |tj|}t |||dd\}}|jjr!t|||\}}t|||\}}nt|||\}}|j}tj|j|j|j|j|S)Nzgtl.where with a non-boolean condition is deprecated and will error out in a future triton release. Got T) r2r$rawarningswarnr|rrkrr}r% create_selectr) conditionrlrr*r{rs rr>r>0s"''! uvwFwFvG H Y1I '1gtT BDAq~~ +Iq'B 1#Aq'21+Iq'B 1 VVF 99W**9+;+;QXXqxxPRX YYrcd|rtj||}n|}tj||Srr)r$rr%)rlrOr|res_tys r wrap_tensorrFs-y)4 99Q rc  |tfdDd}djj t }||ks Jd|dt Dcgc] \}}||k7s |c}} t fdDsJdj Dcgc]}|jc}||jt fdttDScc}}wcc}w)Nc3fK|](}t||jjgd*yw)TrTN)rHr;rA)rctr*s rrezreduction..Qs*fZ[wq177==/tWUUfs.1rz&reduction axis must be < inputs rank (rc3PK|]}|jjk(ywrr)rkr)rcrrs rrezreduction..Xs5qvv||u$5s#&z-all reduction inputs must have the same shapec3K|]7}tj||jj9ywrrr get_resultrkrz)rcrqinputs reduce_opr|s rrezreduction..^s4t\]Y11!4fQinn6K6KYWt=A) tuplerkrrArnall create_reducerverifyr7) rr)region_builder_fnr*rankrqrGrrr|rs ` ` @@@r reductionrOs |f_eff 1INN E u:D $;H@aHH;(/=tq!19=I 5f5 5f7ff 5%%&@Aqxx&@$GIi   tafgjkqgrast tt>'As C;.C; Dcdjjt}| |cxkr|ksnJd|d|d|dkr||z }D]"}|jjk(rJd|jDcgc]}|jc}|||j t fdttDScc}w)Nrz scan axis z must be < inputs rank (rz(all scan inputs must have the same shapec3K|]7}tj||jj9ywrrr)rcrqrscan_oprs rrez#associative_scan..ws4nVWW//2F1INN4I4I5Qnr)rkrrA create_scanrrrr7) rr)rreverser*rrrrs ` @@rassociative_scanrfs 1INN E u:D 5D 4 S:dV3KD6QR!SS  ax   Qvv||u$P&PP$Q!!V"<188"R>R>T)UV Hrct|jt|k7r td|jj dt j ||jj|S)NzDShape of input to max_contiguous does not match the length of valuesz tt.contiguityrArr#rrr rrrs rmax_contiguousrsS 177|s6{"_``HHor||FAHH.s)XDz$$?4)?%)??Xs,.zGExpected a list of constant integers (`int32_t` range) in `block_shape`z.sdis;3y>1ds!$zBExpected shape/strides/offsets/block_shape to have the same length)rrkrsrrr#r$rar|rrrrr^rirArrfrgr7rAcreate_make_block_ptrrr%r) baserstridesr rorderr*rrs ` rmake_block_ptrrs "'5 1E#GW5G#GW%HG 99   !5!5!>!>!@hii yyrww&D"//"''4993J3JKWU ; +"m VabdD",,!?4::TIbK XKX XRQR X 5* %PU V:dBLL9TZZt C VE V %=Ds5z!23 3s5ss 3 dE7T[]bCcd dMLM d  * *4;;wQ\^c dF 99VR__R]]499;O;OQ\-]^ __%c Ws ./H= /Ict||d}tj|j|j||j Sr)rr$r%create_advancerrk)rr r*s radvancers8#GW%HG 99W++DKKA499 MMr)r)rbr* ir.builderreturn tl.tensor)r6tl.dtyper7rrr) r6rrLr_r7rrMr_rNr_rr)T)rmr_)rrrrrur_rNone)FFTF)r~tl.tensor | numbers.Numberrrr*rrTuple[tl.tensor, tl.tensor])r~rrrr*rrcallable) rrrrrr_r*rrr)rrrrr*rrr) rrrrrr_r*rrr)rlrrrrtl.PropagateNanr*r) rlrrrrrrrr*r)rrrrr*rrr)rrrrr*rrr)rrr*r)rrrr)rrr*rrr)rrr*rrr)rrrz tl.block_type)r3rbr4rbr*rrr)r List[int]r2rr*rrr)rArrrr*rrr) rrrErrFr_r*rrr)rrr)rbr*rrr) r~rrrrFr_r*rrr)rVrrWrr*rrr)rVrr*rrr)rrriz Tuple[int]r*rrr)rrrrr*rrr)r~rrrr*rrr)r Optional[str])rrrrr*rrrrr) rrrrr*rrrrr)rrrOptional[tl.tensor]rrrrrrvrrvrrvrr_r*rrr) r rrrvrrvr*rrr)r rrArr*rrr)r rrrrList[tl.tensor]rrrrrrrrbrrbrrbrrbr*rrr)r rr*rrr)rrr"rrrrrvrrvr*rrr)rrr/rr"rrrvrrvr*rrr) rrr"rrrr4rvr*rrz&Tuple[tl.tensor, tl.tensor, tl.tensor])rrr"rrrrrvrrvr*rrr)r~rrrr|rrmrr}rbr~rr*rrr)rr)r~rrrrrrrr|ztl.tensor | Noner~rr*rrr) rrrlrrrr*rrr)rSequence[tl.tensor]r)rbr*rrTuple[tl.tensor, ...]) rrr)rbrr_r*rrr)rrrrbr*rrr)rlrrrrr)r*rrr) rrvrrrr_r*rrr)rrrrvr*rrr)rrr*rrr)u __future__rrtypingrrrrrrx _C.libtritonr r r$r r Exceptionrr+r.r<rQrjrvrrrrrrrrrrrrrrrrrrrrrrrrrrrrr"rr*r/r8rfr>rHrMrQr[r`rjrsr}rrr|rrrrrrrrrrr rrrr#r'r+r0r7rKrOrSrXr]rarernrrrr>rrrrrrrrrrrrrrrrXrrr s#";; CLF FD F @ /,&*/,/7/,d# V <]a,16QB&&:: ):644 )4$ 4 4 ) 4R4 :&&!*&485$5$ ]" +5 :U Q P Q '&" R R Q)$) 4 4 4 4 4 4"D"(.H[MK2 IL$2tpZ(04l5,l58Al5h       w,;| n n.1 nDG nVZ n n!* n ',5g   #  $ >\:)qXWW)3W8AW,j(27]6$)N$)No#(18EE&0E5>EP ? H 0 H=E HPZ H_h HPZ, u.o(o-Bo2n   8 VG BU0H $`NNr