bIg.nddlmZddlZddlZddlZddlZddlmZddlm Z m Z ddl m cm ZddlmZmZmZmZmZGddZddZdd Z d dd Z d dd Zdd Zdd Zd dZ d!dZ d"dZd dZ d"dZ d dZ!d#dZ"d$dZ#d%dZ$d&dZ%y)') annotationsN)chain)CallableIterable)AttributeProto FunctionProto GraphProto ModelProto TensorProtoceZdZddZy)ExternalDataInfocHd|_d|_d|_d|_d|_|j D]#}t ||j|j%|jrt|j|_|jrt|j|_yy)N) locationoffsetlengthchecksumbasepath external_datasetattrkeyvalueint)selftensorentrys V/var/www/html/answerous/venv/lib/python3.12/site-packages/onnx/external_data_helper.py__init__zExternalDataInfo.__init__s     )) 2E D%))U[[ 1 2 ;;dkk*DK ;;dkk*DK N)rr returnNone)__name__ __module__ __qualname__rrrr r s+rr ct|}tj||j|j}t |d5}|j r|j|j |jr!|j|j|_ n|j|_ dddy#1swYyxYw)zLoads data from an external file for tensor. Ideally TensorProto should not hold any raw data but if it does it will be ignored. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. rbN) r c_checker_resolve_external_data_locationrnameopenrseekrreadraw_data)rbase_dirinfoexternal_data_file_path data_files rload_external_data_for_tensorr3)s F #D'GG$-- %t ,/ ;; NN4;; ' ;;'nnT[[9FO'nn.FO///s A*B66B?ct|D]>}t|st||tj|_|j dd=@y)zLoads external tensors into model Arguments: model: ModelProto to load external data to base_dir: directory that contains external data N)_get_all_tensorsuses_external_datar3r DEFAULT data_locationr)modelr/rs rload_external_data_for_modelr:?sF#5)( f % )&( ;#.#6#6F $$Q' (rc||jdstd|jzdz|jdd=tj |_|| t|nd| t|nd||djD]9\}}| |jj}||_ t||_ ;y)Nr.zTensor zGdoes not have raw_data field. Cannot set external data for this tensor.)rrrrr) HasField ValueErrorr*rr EXTERNALr8ritemsaddrstrr) rrrrrrkvrs rset_external_datarDOs ??: & kk W X  Q&//F!'!3#f+!'!3#f+   eg !1 =((,,.EEIa&EK !rcXt|}|r t|}|rttj}|r,t j j|r td|}|D]C}|jdstj|j|k\s8t||Ey|D]w}|jdstj|j|k\s8|j}t|sttj}t||yy)a Call to set all tensors with raw data as external data. This call should precede 'save_model'. 'save_model' saves all the tensors data as external data after calling this function. Arguments: model (ModelProto): Model to be converted. all_tensors_to_one_file (bool): If true, save all tensors to one external file specified by location. If false, save each tensor to a file named with the tensor name. location: specify the external file relative to the model that all tensors to save to. Path is relative to the model path. If not specified, will use the model name. size_threshold: Threshold for size of data. Only when tensor's data is >= the size_threshold it will be converted to external data. To convert every tensor with raw data to external data set size_threshold=0. convert_attribute (bool): If true, convert all tensors to external data If false, convert only non-attribute tensors to external data zDlocation must be a relative path that is relative to the model path.r.N)_get_initializer_tensorsr5rAuuiduuid1ospathisabsr=r<sys getsizeofr.rDr*_is_valid_filename) r9all_tensors_to_one_filersize_thresholdconvert_attributetensors file_namertensor_locations rconvert_model_to_external_datarUms,'u-G"5) % ww}}X& Z!I 5F +MM&//2nD!&)4  5 ;F +MM&//2nD"(++)/:&)$**,&7O!&/: ;rct|D]N}t|s|jds td|jdd=t j |_Py)zCall to set all tensors which use external data as embedded data. save_model saves all the tensors data as embedded data after calling this function. Arguments: model (ModelProto): Model to be converted. r.raw_data field doesn't exist.N)r5r6r<r=rr r7r8)r9rs r convert_model_from_external_datarXsT#5)7 f %??:. !@AA$$Q'#.#6#6F 7rct|}tjj||j}|j ds t dtjj|st|d5 dddt|d5}|jdd|j[|j}|j|kDr!|jd|j|z z|j|j|j}|j|jt||j||j|z dddy#1swYxYw#1swYyxYw) zWrites tensor data to an external file according to information in the `external_data` field. Arguments: tensor (TensorProto): Tensor object to be serialized base_path: System path of a folder where tensor data is to be stored r.rWabNzr+br)r rIrJjoinrr<r=isfiler+r,rtellwriter.rD)r base_pathr0r1r2 file_sizers rsave_external_datarcs9 F #D ggll9dmmD ??: &899 77>>1 2 )4 0    %u - Tq! ;; "!(I{{Y&y)@ AB NN4;; '!(&$--9IF9RS T T    T Ts=E*CE6*E36E?c>tt|t|S)z=Scan an ONNX model for all tensors and return as an iterator.)rrF_get_attribute_tensors)onnx_model_protos rr5r5s!  !12/0 rc#K|jtjk(r||jEd{|jtjk(r"|j D]}||Ed{yy7D7 w)zICreate an iterator through processing ONNX model attributes with functor.N)typerGRAPHgGRAPHSgraphs) attributefuncgraphs r_recursive_attribute_processorrpsp~~--- $$$~~...%% #EE{ " " #/ % #s!3A>A:;A>1A<2 A><A>c#Kt|tr|jEd{|jD]+}|jD]}t |t Ed{-y7?7 w)zICreate an iterator of initializer tensors from ONNX model graph/function.N) isinstancer initializernodermrp#_get_initializer_tensors_from_graphgraph_or_functionrtrms rrurusm#Z0$0000!&& I5>    1 s! A&A"5A&A$ A&$A&c#Kt|jEd{|jD]}t|Ed{y7)7 w)z:Create an iterator of initializer tensors from ONNX model.N)ruro functions!_get_attribute_tensors_from_graphrffunctions rrFrFsH23C3I3IJJJ$..?4X>>>?K>A A!A AA A c#K|jD]^}|jD]M}|jdr|j|jEd{t |t Ed{O`y7!7 w)zSCreate an iterator of tensors from node attributes of an ONNX model graph/function.tN)rtrmr<rrRrprzrvs rrzrzsz"&& I!!#&kk! (( ( (5<     ) s$A A5A1A5'A3( A53A5c#Kt|jEd{|jD]}t|Ed{y7)7 w)zDCreate an iterator of tensors from node attributes of an ONNX model.N)rzroryr{s rreresH01A1G1GHHH$..?4X>>>?I>r}cdtjd}|j|}t|S)z8Utility to check whether the provided filename is valid.z^[^<>:;,?"*|/]+$)recompilematchbool)filenameexprs rrNrNs) **' (C IIh E ;rcb|jdxr|jtjk(S)z?Returns true if the tensor stores data in an external location.r8)r<r8r r>)rs rr6r6s. ( 9  K$8$8 8rcxt|jD]"\}}|j|k(s|j|=$y)aRemoves a field from a Tensor's external_data key-value store. Modifies tensor object in place. Arguments: tensor (TensorProto): Tensor object from which value will be removed field_key (string): The key of the field to be removed N) enumeraterr)r field_keyifields rremove_external_data_fieldrs<f223(5 99 !$$Q'(rct|D]=}t|s|jds!t|||j d?|S)aSerializes data for all the tensors which have data location set to TensorProto.External. Note: This function also strips basepath information from all tensors' external_data fields. Arguments: model (ModelProto): Model object which is the source of tensors to serialize. filepath: System path to the directory which should be treated as base path for external data. Returns: ModelProto: The modified model object. r.)r5r6r<rc ClearField)r9filepathrs rwrite_external_data_tensorsr,sL#5)* f %&//**E vx 0   j )* Lr)rr r/rAr r!)r9r r/rAr r!)NNNN)rr rrAr int | Nonerrr str | Nonerrr r!)TNiF) r9r rOrrrrPrrQrr r!)r9r r r!)rr rarAr r!)rfr r Iterable[TensorProto])rmrrnz-Callable[[GraphProto], Iterable[TensorProto]]r r)rwzGraphProto | FunctionProtor r)rrAr r)rr r r)rr rrAr r!)r9r rrAr r )& __future__rrIrrLrG itertoolsrtypingrronnx.onnx_cpp2py_export.checkeronnx_cpp2py_exportcheckerr( onnx.onnx_pbrrr r r r r3r:rDrUrXrcr5rprurFrzrerNr6rrr%rrrsX# %33++$/, (& ! !! !  !  !  ! !@%)# 1; 1;!1;1; 1;  1;  1;h 7 TD##%R## 1  ? 1  ? (r