> ## Documentation Index > Fetch the complete documentation index at: https://langchain-5e9cc07a-preview-ccdata-1770345743-a5afa16.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # ChatNVIDIA integration > Integrate with the ChatNVIDIA chat model using LangChain Python. This will help you get started with NVIDIA [chat models](/oss/python/langchain/models). For detailed documentation of all `ChatNVIDIA` features and configurations head to the [API reference](https://python.langchain.com/api_reference/nvidia_ai_endpoints/chat_models/langchain_nvidia_ai_endpoints.chat_models.ChatNVIDIA.html). ## Overview The `langchain-nvidia-ai-endpoints` package contains LangChain integrations for chat models and embeddings powered by [NVIDIA AI Foundation Models](https://www.nvidia.com/en-us/ai-data-science/foundation-models/), and hosted on the [NVIDIA API Catalog](https://build.nvidia.com/). NVIDIA AI Foundation models are community- and NVIDIA-built models that are optimized to deliver the best performance on NVIDIA-accelerated infrastructure. You can use the API to query live endpoints that are available on the NVIDIA API Catalog to get quick results from a DGX-hosted cloud compute environment, or you can download models from NVIDIA's API catalog with NVIDIA NIM, which is included with the NVIDIA AI Enterprise license. The ability to run models on-premises gives your enterprise ownership of your customizations and full control of your IP and AI application. NIM microservices are packaged as container images on a per model/model family basis and are distributed as NGC container images through the [NVIDIA NGC Catalog](https://catalog.ngc.nvidia.com/). At their core, NIM microservices are containers that provide interactive APIs for running inference on an AI Model. This example goes over how to use LangChain to interact with NVIDIA models via the `ChatNVIDIA` class. For more information on accessing embedding models through this API, refer to the [NVIDIAEmbeddings](/oss/python/integrations/text_embedding/nvidia_ai_endpoints) documentation. ### Integration details | Class | Package | Serializable | JS support | Downloads | Version | | :------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------- | :----------: | :--------: | :------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------: | | [ChatNVIDIA](https://python.langchain.com/api_reference/nvidia_ai_endpoints/chat_models/langchain_nvidia_ai_endpoints.chat_models.ChatNVIDIA.html) | [langchain-nvidia-ai-endpoints](https://python.langchain.com/api_reference/nvidia_ai_endpoints/index.html) | beta | ❌ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain_nvidia_ai_endpoints?style=flat-square\&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain_nvidia_ai_endpoints?style=flat-square\&label=%20) | ### Model features | [Tool calling](/oss/python/langchain/tools) | [Structured output](/oss/python/langchain/structured-output) | [Image input](/oss/python/langchain/messages#multimodal) | Audio input | Video input | [Token-level streaming](/oss/python/langchain/streaming/) | Native async | [Token usage](/oss/python/langchain/models#token-usage) | [Logprobs](/oss/python/langchain/models#log-probabilities) | | :-----------------------------------------: | :----------------------------------------------------------: | :------------------------------------------------------: | :---------: | :---------: | :-------------------------------------------------------: | :----------: | :-----------------------------------------------------: | :--------------------------------------------------------: | | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ## Install the package ```bash theme={null} pip install -qU langchain-nvidia-ai-endpoints ``` ## Access the NVIDIA API Catalog To get access to the NVIDIA API Catalog, do the following: 1. Create a free account on the [NVIDIA API Catalog](https://build.nvidia.com/) and log in. 2. Click your profile icon, and then click **API Keys**. The **API Keys** page appears. 3. Click **Generate API Key**. The **Generate API Key** window appears. 4. Click **Generate Key**. You should see **API Key Granted**, and your key appears. 5. Copy and save the key as `NVIDIA_API_KEY`. 6. To verify your key, use the following code. ```python theme={null} import getpass import os if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"): print("Valid NVIDIA_API_KEY already in environment. Delete to reset") else: nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ") assert nvapi_key.startswith( "nvapi-" ), f"{nvapi_key[:5]}... is not a valid key" os.environ["NVIDIA_API_KEY"] = nvapi_key ``` You can now use your key to access endpoints on the NVIDIA API Catalog. To enable automated tracing of your model calls, set your [LangSmith](https://docs.langchain.com/langsmith/home) API key: ```python theme={null} os.environ["LANGSMITH_TRACING"] = "true" os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ") ``` ## Instantiation Now we can access models in the NVIDIA API Catalog: ```python theme={null} ## Core LC Chat Interface from langchain_nvidia_ai_endpoints import ChatNVIDIA llm = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1") ``` ## Invocation ```python theme={null} result = llm.invoke("Write a ballad about LangChain.") print(result.content) ``` ## Self-host with NVIDIA NIM Microservices When you are ready to deploy your AI application, you can self-host models with NVIDIA NIM. For more information, refer to [NVIDIA NIM Microservices](https://www.nvidia.com/en-us/ai-data-science/products/nim-microservices/). The following code connects to locally hosted NIM Microservices. ```python theme={null} from langchain_nvidia_ai_endpoints import ChatNVIDIA, NVIDIAEmbeddings, NVIDIARerank # connect to a chat NIM running at localhost:8000, specifying a model llm = ChatNVIDIA(base_url="http://localhost:8000/v1", model="meta/llama3-8b-instruct") # connect to an embedding NIM running at localhost:8080 embedder = NVIDIAEmbeddings(base_url="http://localhost:8080/v1") # connect to a reranking NIM running at localhost:2016 ranker = NVIDIARerank(base_url="http://localhost:2016/v1") ``` ## Stream, batch, and async These models natively support streaming, and as is the case with all LangChain LLMs they expose a batch method to handle concurrent requests, as well as async methods for invoke, stream, and batch. Below are a few examples. ```python theme={null} print(llm.batch(["What's 2*3?", "What's 2*6?"])) # Or via the async API # await llm.abatch(["What's 2*3?", "What's 2*6?"]) ``` ```python theme={null} for chunk in llm.stream("How far can a seagull fly in one day?"): # Show the token separations print(chunk.content, end="|") ``` ```python theme={null} async for chunk in llm.astream( "How long does it take for monarch butterflies to migrate?" ): print(chunk.content, end="|") ``` ## Supported models Querying `available_models` will still give you all of the other models offered by your API credentials. The `playground_` prefix is optional. ```python theme={null} ChatNVIDIA.get_available_models() # llm.get_available_models() ``` ## Model types All of these models above are supported and can be accessed via `ChatNVIDIA`. Some model types support unique prompting techniques and chat messages. We will review a few important ones below. **To find out more about a specific model, please navigate to the API section of an AI Foundation model [as linked here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/codellama-13b/api).** ### General chat Models such as `meta/llama3-8b-instruct` and `mistralai/mixtral-8x22b-instruct-v0.1` are good all-around models that you can use for with any LangChain chat messages. Example below. ```python theme={null} from langchain_core.output_parsers import StrOutputParser from langchain_core.prompts import ChatPromptTemplate from langchain_nvidia_ai_endpoints import ChatNVIDIA prompt = ChatPromptTemplate.from_messages( [("system", "You are a helpful AI assistant named Fred."), ("user", "{input}")] ) chain = prompt | ChatNVIDIA(model="meta/llama3-8b-instruct") | StrOutputParser() for txt in chain.stream({"input": "What's your name?"}): print(txt, end="") ``` ### Code generation These models accept the same arguments and input structure as regular chat models, but they tend to perform better on code-generation and structured code tasks. An example of this is `meta/codellama-70b`. ```python theme={null} prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are an expert coding AI. Respond only in valid python; no narration whatsoever.", ), ("user", "{input}"), ] ) chain = prompt | ChatNVIDIA(model="meta/codellama-70b") | StrOutputParser() for txt in chain.stream({"input": "How do I solve this fizz buzz problem?"}): print(txt, end="") ``` ## Multimodal NVIDIA also supports multimodal inputs, meaning you can provide both images and text for the model to reason over. An example model supporting multimodal inputs is `nvidia/neva-22b`. Below is an example use: ```python theme={null} import IPython import requests image_url = "https://www.nvidia.com/content/dam/en-zz/Solutions/research/ai-playground/nvidia-picasso-3c33-p@2x.jpg" ## Large Image image_content = requests.get(image_url).content IPython.display.Image(image_content) ``` ```python theme={null} from langchain_nvidia_ai_endpoints import ChatNVIDIA llm = ChatNVIDIA(model="nvidia/neva-22b") ``` #### Passing an image as a URL ```python theme={null} from langchain.messages import HumanMessage llm.invoke( [ HumanMessage( content=[ {"type": "text", "text": "Describe this image:"}, {"type": "image_url", "image_url": {"url": image_url}}, ] ) ] ) ``` #### Passing an image as a base64 encoded string At the moment, some extra processing happens client-side to support larger images like the one above. But for smaller images (and to better illustrate the process going on under the hood), we can directly pass in the image as shown below: ```python theme={null} import IPython import requests image_url = "https://picsum.photos/seed/kitten/300/200" image_content = requests.get(image_url).content IPython.display.Image(image_content) ``` ```python theme={null} import base64 from langchain.messages import HumanMessage ## Works for simpler images. For larger images, see actual implementation b64_string = base64.b64encode(image_content).decode("utf-8") llm.invoke( [ HumanMessage( content=[ {"type": "text", "text": "Describe this image:"}, { "type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64_string}"}, }, ] ) ] ) ``` #### Directly within the string The NVIDIA API uniquely accepts images as base64 images inlined within `` HTML tags. While this isn't interoperable with other LLMs, you can directly prompt the model accordingly. ```python theme={null} base64_with_mime_type = f"data:image/png;base64,{b64_string}" llm.invoke(f'What\'s in this image?\n') ``` ## Example usage within a RunnableWithMessageHistory Like any other integration, ChatNVIDIA is fine to support chat utilities like RunnableWithMessageHistory which is analogous to using `ConversationChain`. Below, we show the [LangChain RunnableWithMessageHistory](https://python.langchain.com/api_reference/core/runnables/langchain_core.runnables.history.RunnableWithMessageHistory.html) example applied to the `mistralai/mixtral-8x22b-instruct-v0.1` model. ```python theme={null} pip install -qU langchain ``` ```python theme={null} from langchain_core.chat_history import InMemoryChatMessageHistory from langchain_core.runnables.history import RunnableWithMessageHistory # store is a dictionary that maps session IDs to their corresponding chat histories. store = {} # memory is maintained outside the chain # A function that returns the chat history for a given session ID. def get_session_history(session_id: str) -> InMemoryChatMessageHistory: if session_id not in store: store[session_id] = InMemoryChatMessageHistory() return store[session_id] chat = ChatNVIDIA( model="mistralai/mixtral-8x22b-instruct-v0.1", temperature=0.1, max_tokens=100, top_p=1.0, ) # Define a RunnableConfig object, with a `configurable` key. session_id determines thread config = {"configurable": {"session_id": "1"}} conversation = RunnableWithMessageHistory( chat, get_session_history, ) conversation.invoke( "Hi I'm Srijan Dubey.", # input or query config=config, ) ``` ```python theme={null} conversation.invoke( "I'm doing well! Just having a conversation with an AI.", config=config, ) ``` ```python theme={null} conversation.invoke( "Tell me about yourself.", config=config, ) ``` ## Tool calling Starting in v0.2, `ChatNVIDIA` supports [bind\_tools](https://python.langchain.com/api_reference/core/language_models/langchain_core.language_models.chat_models.BaseChatModel.html#langchain_core.language_models.chat_models.BaseChatModel.bind_tools). `ChatNVIDIA` provides integration with the variety of models on [build.nvidia.com](https://build.nvidia.com) as well as local NIMs. Not all these models are trained for tool calling. Be sure to select a model that does have tool calling for your experimention and applications. You can get a list of models that are known to support tool calling with, ```python theme={null} tool_models = [ model for model in ChatNVIDIA.get_available_models() if model.supports_tools ] tool_models ``` With a tool capable model, ```python theme={null} from langchain.tools import tool from pydantic import Field @tool def get_current_weather( location: str = Field(..., description="The location to get the weather for."), ): """Get the current weather for a location.""" ... llm = ChatNVIDIA(model=tool_models[0].id).bind_tools(tools=[get_current_weather]) response = llm.invoke("What is the weather in Boston?") response.tool_calls ``` See [How to use chat models to call tools](/oss/python/langchain/tools) for additional examples. *** ## API reference For detailed documentation of all `ChatNVIDIA` features and configurations head to the API reference: [python.langchain.com/api\_reference/nvidia\_ai\_endpoints/chat\_models/langchain\_nvidia\_ai\_endpoints.chat\_models.ChatNVIDIA.html](https://python.langchain.com/api_reference/nvidia_ai_endpoints/chat_models/langchain_nvidia_ai_endpoints.chat_models.ChatNVIDIA.html) ## Related topics * [`langchain-nvidia-ai-endpoints` package `README`](https://github.com/langchain-ai/langchain-nvidia/blob/main/libs/ai-endpoints/README.md) * [Overview of NVIDIA NIM for Large Language Models (LLMs)](https://docs.nvidia.com/nim/large-language-models/latest/introduction.html) * [Overview of NeMo Retriever Embedding NIM](https://docs.nvidia.com/nim/nemo-retriever/text-embedding/latest/overview.html) * [Overview of NeMo Retriever Reranking NIM](https://docs.nvidia.com/nim/nemo-retriever/text-reranking/latest/overview.html) * [`NVIDIAEmbeddings` Model for RAG Workflows](/oss/python/integrations/text_embedding/nvidia_ai_endpoints) * [NVIDIA Provider Page](/oss/python/integrations/providers/nvidia) *** [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/oss/python/integrations/chat/nvidia_ai_endpoints.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose). [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.