Ig[,ddlmZddlmZmZmZddlZddlZddlm Z ddl m Z m Z ddl mZmZddlmZdd lmZdd lmZd d lmZej0eZGd de ZGdde ZGddeZeGddeZGddeZy)) dataclass)OptionalTupleUnionN)CrossEntropyLoss)InstructBlipQFormerConfigInstructBlipVisionConfig)$InstructBlipForConditionalGeneration/InstructBlipForConditionalGenerationModelOutput)PretrainedConfig)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES)logging)CONFIG_MAPPINGc eZdZy)InstructBlipVideoVisionConfigN__name__ __module__ __qualname__|/var/www/html/answerous/venv/lib/python3.12/site-packages/transformers/models/instructblipvideo/modular_instructblipvideo.pyrr)rrc eZdZy)InstructBlipVideoQFormerConfigNrrrrrr-rrrcPeZdZdZdZ dfd Zededede fdZ xZ S) InstructBlipVideoConfiga [`InstructBlipVideoConfig`] is the configuration class to store the configuration of a [`InstructBlipVideoForConditionalGeneration`]. It is used to instantiate a Instructblipvideo model according to the specified arguments, defining the vision model, Q-Former model and language model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the Instructblipvideo [Salesforce/instruct-blip-flan-t5](https://huggingface.co/Salesforce/instruct-blip-flan-t5) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vision_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`InstructBlipVideoVisionConfig`]. qformer_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`InstructBlipVideoQFormerConfig`]. text_config (`dict`, *optional*): Dictionary of configuration options used to initialize any [`PretrainedConfig`]. num_query_tokens (`int`, *optional*, defaults to 32): The number of query tokens passed through the Transformer. video_token_index (`int`, *optional*): Token index of special video token. kwargs (*optional*): Dictionary of keyword arguments. Example: ```python >>> from transformers import ( ... InstructBlipVideoVisionConfig, ... InstructBlipVideoQFormerConfig, ... OPTConfig, ... InstructBlipVideoConfig, ... InstructBlipVideoForConditionalGeneration, ... ) >>> # Initializing a InstructBlipVideoConfig with Salesforce/instruct-blip-flan-t5 style configuration >>> configuration = InstructBlipVideoConfig() >>> # Initializing a InstructBlipVideoForConditionalGeneration (with random weights) from the Salesforce/instruct-blip-flan-t5 style configuration >>> model = InstructBlipVideoForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> # We can also initialize a InstructBlipVideoConfig from a InstructBlipVideoVisionConfig, InstructBlipVideoQFormerConfig and any PretrainedConfig >>> # Initializing Instructblipvideo vision, Instructblipvideo Q-Former and language model configurations >>> vision_config = InstructBlipVideoVisionConfig() >>> qformer_config = InstructBlipVideoQFormerConfig() >>> text_config = OPTConfig() >>> config = InstructBlipVideoConfig.from_text_vision_configs(vision_config, qformer_config, text_config) ```instructblipvideoc ft|di||i}tjd|i}tjd|i}tjdt di||_t di||_d|vr|dnd}t|di||_ |jj|_ |jj|_ ||_ ||_ |j j|j_|jj t"v|_d|_d|_y) NzZvision_config is None. initializing the InstructBlipVideoVisionConfig with default values.z\qformer_config is None. Initializing the InstructBlipVideoQFormerConfig with default values.zTtext_config is None. Initializing the text config with default values (`OPTConfig`). model_typeoptg?g{Gz?r)super__init__loggerinfor vision_configrqformer_configr text_configtie_word_embeddingsis_encoder_decodernum_query_tokensvideo_token_index hidden_sizeencoder_hidden_sizer"ruse_decoder_only_language_modelinitializer_factorinitializer_range) selfr(r)r*r-r.kwargstext_model_type __class__s rr%z InstructBlipVideoConfig.__init__ks" "6"  M KKt u  !N KKv w  K KKn o:K]K<N~N7C{7R+l3X])/:I[I#'#3#3#G#G "&"2"2"E"E 0!2262D2D2P2P//3/?/?/J/JNo/o,"%!%rr(r)r*c n|d|j|j|jd|S)a  Instantiate a [`InstructBlipVideoConfig`] (or a derived class) from a InstructBlipVideo vision model, Q-Former and language model configurations. Returns: [`InstructBlipVideoConfig`]: An instance of a configuration object )r(r)r*r)to_dict)clsr(r)r*r5s r from_vision_qformer_text_configsz8InstructBlipVideoConfig.from_vision_qformer_text_configssD  '//1)113#++-    r)NNN N) rrr__doc__r"r% classmethodrrr r; __classcell__)r7s@rrr1sW5 n%J $&L 4 7 &   rrc eZdZy)4InstructBlipVideoForConditionalGenerationModelOutputNrrrrrArAsrrAc&eZdZ ddejdejdeej deejdeej deej deej d eed eed eej d eed edee e ffdZ ej ddejdeej deej deej deej d edej fdZ y))InstructBlipVideoForConditionalGenerationN pixel_valuesqformer_input_idsqformer_attention_mask input_idsattention_maskdecoder_input_idsdecoder_attention_maskoutput_attentionsoutput_hidden_stateslabels return_dictinterpolate_pos_encodingreturnc H | | n|jj} |j\} }}}}|j| |z|||}|j ||| | | }|d}t j |jddt j|j}|jj|jddd}t j |jddt j|j}|t j|}|j|d}|j|d}t j||gd}|j||||||| | }|dddd|jdddf}|j!|}|j| |jj"|zd}t j |jddt j|j}|j$j'|}|t j|}t)|jd dK||jj*k(j-dj/|}|j1||<nyt2j5d t j||j7|jgd}t j||j7|jgd}|jj8r|j%|||| | }| r |j:n|d}d}| | j7|j} |dd| jd dddf}|d ddddfj=} | d ddfj=j7|j}!t?d }"|"| jAd|jjBjD|!jAd}n?|j%|||||| | | }| r |jFn|d}| r |j:n|d}| s||||f}#||f|#zS|#StI|||||S)a0 ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```N)rDrKrLrNrOrdtypedevicedim)rGrH query_embedsencoder_hidden_statesencoder_attention_maskrKrLrNr.KExpanding inputs for video tokens in InstructBLIPVideo should be done in processing. Please follow instruction here (https://gist.github.com/zucchini-nlp/65f22892b054dc0d68228af56fbeaac2) to update your InstructBLIPVideo model. Using processors without these attributes in the config is deprecated and will throw an error in v4.47.) inputs_embedsrHrKrLrN.mean) reduction)r]rHrIrJrKrLrNrM)losslogitsvision_outputsqformer_outputslanguage_model_outputs)%configuse_return_dictshapereshape vision_modeltorchonessizelongrU query_tokensexpand ones_likerepeat_interleavecatqformerlanguage_projectionr-language_modelget_input_embeddingsgetattrr. unsqueeze expand_asflattenr& warning_oncetor1ra contiguousrviewr* vocab_sizer`rA)$r4rDrErFrGrHrIrJrKrLrMrNrO batch_sizeframeschannelheightwidthrb image_embedsimage_attention_maskrnquery_attention_mask query_outputs query_outputlanguage_model_inputslanguage_model_attention_maskr]special_image_maskoutputsrar` shift_logits shift_labelsloss_fctoutputs$ rforwardz1InstructBlipVideoForConditionalGeneration.forwardsN&1%->q-A+A1(DE !% 8 8 F!6 = =j$++JfJfioJoqs t(- ! & & (" -UZZH]HdHd) %C++@@B9M  !"__Y7N 4;; 3T : F"+t{{/L/L"L!W!WXZ![!e!efs!t 0E0M0M0OM, -   z  "II'N>NOdOkOk>l&mstuM"YY.0A0AB_BfBf0ghnoN ;; 6 6))+-"3%9' *G(3W^^ FD!6==1FKKN?#4a 78%c3B3k2==? %c12g99;>>v}}M ,f= 1 1"dkk6M6M6X6X Y[g[l[lmo[pq))+-"3'="3%9'* G$/7<jAd t j||j1|jgd }t j||j1|jgd }|jBj$jDsM|jGd d|jd zd z |d <|jGdd|jd z|d<|jBjHd||d|}|jBj$jDs|j$j*jJddk(rdn|j$j*j,}t j(|ggj/|d j1| j}tM|t jNs*t j||jPgd|_(|St j||gd}|S)a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width) or (batch_size, num_frames, num_channels, height, width)): Input images or videos to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. hf_device_mapT)rNrONrRrSrrVrX)rGrHrYrZr[rNr.r\ max_length min_length)r]rHLLaMAForCausalLMrr))hasattr_preprocess_acceleratergrhrilast_hidden_staterjrkrlrmrUrnrorprqrrrsrtrer- LongTensorr* bos_token_idrepeatr|rvrwr.rxryrzr&r{rur,getgenerate architectures isinstanceTensor sequences)r4rDrErFrGrHrOgenerate_kwargsrrrrrrrrnrrrrlanguage_attention_maskr]rrr bos_tokenss rrz2InstructBlipVideoForConditionalGeneration.generatels> 4 )  ' ' )6B5G5G2 FGVU#++J,?&RWX (( %=)     %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "-??A?N!7!I!I&VW!I!X!&,@BX+Y_`!a '1%".#7 % %66q:PL  4;; 3T : F"+t{{/L/L"L!W!WXZ![!e!efs!t 0E0M0M0OM, -   z  "II'N>NOdOkOk>l&mstuM"YY(.*;*;*:;BB:qQTTUaUhUhiJgu||4$)IIz7;L;L.MSU$V! ))Z$9rBr) NNNNNNNNNF)NNNNF)rrrrj FloatTensorrrboolrrrArno_gradrrrrrCrCs >B15598<=A,0/3-1&*).{ ''{ !,,{ !))9)9 : { E--. { !!1!12 { $E$4$45{ !))9)9 :{ $D>{ 'tn{ ))*{ d^{ #'{  uJJ K{ zU]]_9==A0459).A''A$E$4$45A!))9)9 : A E,,- A !!1!12 A#'A   AArrC) dataclassesrtypingrrrrjtorch.utils.checkpointtorch.nnr;transformers.models.instructblip.configuration_instructbliprr 6transformers.models.instructblip.modeling_instructblipr r configuration_utilsr models.auto.modeling_autorutilsrautor get_loggerrr&rrrrArCrrrrs ")) % 4J!   H % $<  %> u .u p  ;j   @0T@r