*#hddlZddlmZmZddlmZddlmZmZeje Z GddZ edZ d Zd Zeej d efd Zy) N)PoolRLock)tqdm) experimentalloggingceZdZdZy)ParallelBackendConfigN)__name__ __module__ __qualname__ backend_nameW/var/www/html/sandstorm/venv/lib/python3.12/site-packages/datasets/parallel/parallel.pyr r sLrr c ftjt|||||||St|||||||S)a **Experimental.** Apply a function to iterable elements in parallel, where the implementation uses either multiprocessing.Pool or joblib for parallelization. Args: function (`Callable[[Any], Any]`): Function to be applied to `iterable`. iterable (`list`, `tuple` or `np.ndarray`): Iterable elements to apply function to. num_proc (`int`): Number of processes (if no backend specified) or jobs (using joblib). types (`tuple`): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`): Whether to disable the tqdm progressbar. desc (`str`): Prefix for the tqdm progressbar. single_map_nested_func (`Callable`): Map function that applies `function` to an element from `iterable`. Takes a tuple of function, data_struct, types, rank, disable_tqdm, desc as input, where data_struct is an element of `iterable`, and `rank` is used for progress bar. )r r_map_with_multiprocessing_pool_map_with_joblib)functioniterablenum_proctypes disable_tqdmdescsingle_map_nested_funcs r parallel_maprsG"))1- h%tE[   Hh%tUk llrc|t|kr|n t|}g}t|D]Y}t||z} t||z} | |zt|| z} | | z|| krdndz} |j||| | ||||f[t|t d|Dk7r*t dt|dt d|Dt jd|dt|d |D cgc]} t| dc} d \}}|stftj}}t||| 5}|j||}dddt jd |d Dcgc] }|D]}| }}}t jdt|d|Scc} w#1swYcxYwcc}}w)Nrc38K|]}t|dywrNlen.0is r z1_map_with_multiprocessing_pool..3s:!C!I:zHError dividing inputs iterable among processes. Total number of objects z , length: c38K|]}t|dywr r!r#s rr&z1_map_with_multiprocessing_pool..7s93qt99r'z Spawning z processes for z objects in slices of )NN)initargs initializerz Finished z processesz Unpacked z objects) r"rangeminappendsum ValueErrorloggerinforrset_lockrmap)rrrrrrr split_kwdsindexdivmodstartendr%r)r*poolmappedproc_resobjs rrr)s#s8}4x#h-HJx](mx'(mh&e c%o-ck%#+Q158XeC%8% VZ[\ ] 8}:z:::''*8}o69j99: <   KK H:_S]O;QfpRqabSVWXYZW[S\RqQrs'Hk !& DMM+ h{ C>t0*=> KK)H:Z01"( =hH =Sc =c =F = KK)CK=12 MSr >>>s0F.>F37F?3F<cddljtj|5j fd|DcdddS#1swYyxYw)Nr)n_jobsc 3XK|]!}j|dddf#yw)NT)delayed)r$r=rjoblibrrs rr&z#_map_with_joblib..Os7! ad 2FNN1 2Hc5$PTVZ3[ \! s'*)rBparallel_backendr rParallel)rrrrrrrrBs` ` `@rrrIsU  !6!C!CH U  v ! hp!     s "AArc#K|t_|dk(r ddlm}| ddt_y#dt_wxYww)a **Experimental.** Configures the parallel backend for parallelized dataset loading, which uses the parallelization implemented by joblib. Args: backend_name (str): Name of backend for parallelization implementation, has to be supported by joblib. Example usage: ```py with parallel_backend('spark'): dataset = load_dataset(..., num_proc=2) ``` sparkr)register_sparkN)r r joblibsparkrG)rrGs rrCrCTs< *6&w. 2 -1*T*sA1 A >A) contextlibmultiprocessingrr tqdm.autorutilsrr get_loggerr r0r rrrcontextmanagerstrrCrrrrPsy')   H %mm0@  2322r