o Ng 2@s>ddlmZmZmZmZmZmZmZddlm Z m Z m Z ddl m Z ddlmZddlZddlZddlZddlZddlZddlmZddlmZddlmZdd lTdd lmZdd lmZddl m!Z"dd l#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+eZ,e Z-ee.d Z/e0e.ddZ1e0e.ddZ2eZ3e,4dedededededee-fdede5de0de0de0de f ddZ6e,7dee-fde5de)j8de fddZ9e,:d ee-fde5de fd!d"Z;e,je,sz!get_documents..z Documents retrieved successfullycSsg|]}|qSru)rrrururvrsr)r|r3)r6r7r!r8r9r?rrIrer@rrAralllenr=r_r`rrC)rr!rbrcrd documentsrrrsrururv get_documentss& 0 rz/task/{task_id}r1cs t|}|jt|j|jdS)N)rresult traceback)rrrCrr)r1 task_resultrururvget_task_statuss rurl save_path chunk_sizec CszU|ds tdWStj|ddd}|t|jdd}tjtj |ddt |d }|j |d D] }|r@| |q7Wd n1sKwYtd |WStjjys}ztd t|dWYd }~Sd }~wty}ztd|dWYd }~Sd }~ww)a Downloads a file from the specified URL and saves it to the given path. Args: url (str): The URL of the file to download. save_path (str): The local path where the file should be saved. chunk_size (int, optional): The chunk size for streaming data. Defaults to 1024. Returns: str: The path of the saved file. Raises: ValueError: If the URL is invalid or the response status is not successful. Exception: For other errors during download or file saving. )zhttp://zhttps://z9Invalid URL. URL must start with 'http://' or 'https://'.T )streamtimeoutzcontent-lengthrr"r%)rNryz"Failed to download file from URL: r(z)An error occurred while saving the file: r)) startswithr=requestsrUraise_for_statusintheadersrFrGrHdirnamerK iter_contentrLr_ exceptionsRequestExceptionrCr`)rrrresponse total_sizerNchunkrsrururvrS s,      rS)r)Afastapirrrrrrrfastapi.securityr r r fastapi.responsesr dotenvr rrFr:uuiddocx2pdfopenairdependencies.HybridSearchrdependencies.awsutilsrdependencies.helper celery_workerr celery.resultrdb_config.databasedatabaser6db_config.modelsrrrrdb_config.schemasr~db_config.authr8routersecurityrr]rC DB_FOLDERrD openai_clientpostrrwputDocumentrrrrUDocumentListResponserrrSrurururvsn$            v"&.