Skip to content

Ollama integration

To configure the use of Ollama models you don't usually need a valid API KEY but you will have to specify:

  • a base API URL if Ollama is not running on the same machine as Brevia.
  • a model from the Ollama model library

Embeddings

The basic configuration is as follows:

EMBEDDINGS='{"_type": "langchain_ollama.embeddings.OllamaEmbeddings" , "model": "nomic-embed-text"}'

Key variables you can add to this configuration include:

  • model: the name of Ollama model to use (string)
  • base_url: base url the model is hosted under, defaults to http://127.0.0.1:11434

For additional configuration options, refer to the LangChain API reference.

Conversational LLM

Conversational LLMs are configured using the variables QA_COMPLETION_LLM and QA_FOLLOWUP_LLM or SUMMARIZE_LLM for summarization and analysis that use the same JSON format.

An example JSON configuration using Meta's Llama 3.2 might look like this:

{"_type": "langchain_ollama.chat_models.ChatOllama", "model": "llama3.2",
"base_url":"http://localhost:11434", "temperature": 0}

The primary variables you can add to this configuration include:

  • model: name of the Ollama model to use (string)
  • base_url: base url the model is hosted under, defaults to http://127.0.0.1:11434
  • temperature: sampling temperature, ranges from 0.0 to 1.0. (float)
  • num_predict: max number of tokens to generate (int)

For further configuration options, refer to the LangChain API reference.