Ig'zddlmZmZmZmZmZmZmZmZddl m Z ddl m Z erddl mZmZddlmZGdde Zy) ) TYPE_CHECKINGAnyDict GeneratorListMappingOptionalUnion)CallbackManagerForLLMRun)LLM)RESTfulChatModelHandleRESTfulGenerateModelHandle)LlamaCppGenerateConfigc2eZdZUdZeed<eeed< eeed< eeefed< ddeedeedeffd Z e defd Z e de eeffd Z dd ed eeed eededef dZ ddedd ed eededdeeddff dZxZS) Xinferencea`Xinference` large-scale model inference service. To use, you should have the xinference library installed: .. code-block:: bash pip install "xinference[all]" If you're simply using the services provided by Xinference, you can utilize the xinference_client package: .. code-block:: bash pip install xinference_client Check out: https://github.com/xorbitsai/inference To run, you need to start a Xinference supervisor on one server and Xinference workers on the other servers Example: To start a local instance of Xinference, run .. code-block:: bash $ xinference You can also deploy Xinference in a distributed cluster. Here are the steps: Starting the supervisor: .. code-block:: bash $ xinference-supervisor Starting the worker: .. code-block:: bash $ xinference-worker Then, launch a model using command line interface (CLI). Example: .. code-block:: bash $ xinference launch -n orca -s 3 -q q4_0 It will return a model UID. Then, you can use Xinference with LangChain. Example: .. code-block:: python from langchain_community.llms import Xinference llm = Xinference( server_url="http://0.0.0.0:9997", model_uid = {model_uid} # replace model_uid with the model UID return from launching the model ) llm.invoke( prompt="Q: where can we visit in the capital of France? A:", generate_config={"max_tokens": 1024, "stream": True}, ) To view all the supported builtin models, run: .. code-block:: bash $ xinference list --all client server_url model_uid model_kwargsNc $ ddlm}|xsi}t |di|||d|j td|j td|||_ y#t$r( ddlm}n#t$r}td|d}~wwxYwYwxYw)Nr) RESTfulClientzCould not import RESTfulClient from xinference. Please install it with `pip install xinference` or `pip install xinference_client`.rrrzPlease provide server URLzPlease provide the model UID) xinference.clientr ImportErrorxinference_clientsuper__init__r ValueErrorrr)selfrrrre __class__s `/var/www/html/answerous/venv/lib/python3.12/site-packages/langchain_community/llms/xinference.pyrzXinference.__init__\s  7$)r   (& ,  ?? "89 9 >> !;< <#J/ 3  ; !Y < s5A B(A/.B/ B 8 BB  BBreturncy)zReturn type of llm. xinferencerr s r# _llm_typezXinference._llm_typescZid|jid|jid|jiS)zGet the identifying parameters.rrrrr's r#_identifying_paramszXinference._identifying_paramssC T__- DNN+ t001  r)promptstop run_managerkwargsc F|jj|j}|jdi}i|j|}|r||d<|r4|jdr#d}|j ||||D]}||z } |S|j ||} | ddd S) aqCall the xinference model and return the output. Args: prompt: The prompt to use for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Returns: The generated string by the model. generate_configr-stream)modelr,r.r1r,r1choicesrtext)r get_modelrgetr_stream_generategenerate) r r,r-r.r/r4r1combined_text_outputtoken completions r#_callzXinference._calls$ %%dnn54:JJ?PRT4UBT..B/B &*OF # 228<#% ..' / / . %-$  .( 'vWJi(+F3 3r)r4)rr r1rc#FK|j||}|D]}t|ts|jdg}|s)|d}t|ts?|jdd} |jd} |r|j | |j | | yw) a^ Args: prompt: The prompt to use for generation. model: The model used for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Yields: A string token. r5r6rr7r3logprobs)r=verbose log_probsN)r; isinstancedictr9on_llm_new_tokenrB) r r4r,r.r1streaming_responsechunkr6choicer=rCs r#r:zXinference._stream_generates$#^^?, ( $E%&))Ir2$QZF!&$/ & 62 6$*JJz$: &'88&+T\\Y9$  $s)B!B!B!A B!)NN)__name__ __module__ __qualname____doc__r__annotations__r strrrpropertyr(rr+rr r?r rr: __classcell__)r"s@r#rr sTFP K &}#sCx. :%)#'!0SM!0C=!0 !0F3 WS#X%6  %):> (4(4tCy!(467 (4  (4 (4\;?>B !$KL!$!$67 !$ "":; !$ 3d? # !$r)rN)typingrrrrrrr r langchain_core.callbacksr #langchain_core.language_models.llmsr rr rxinference.model.llm.corerrrr)r#rVs+VVV=3T@M$M$r)