YaqeenDocs
Back to Yaqeen

How-to guide

Use your own model

Run the assistant on Hugging Face, Ollama, vLLM or any OpenAI style API by changing settings.

The assistant talks to one language model through one setting. Pick a hosted provider, a server you run, or a model on your own machine. Personal values are masked before any request leaves Yaqeen, whichever you pick. The setting that chooses the provider is ASSISTANT_LLM_PROVIDER: huggingface, endpoint, openai, openai_compatible, vllm, tgi, ollama or litellm.

Use a hosted provider#

A hosted provider runs the model for you. Personal values are masked before the request leaves Yaqeen, and your key stays in Yaqeen's settings. Plain http is refused for anything that is not a local address, so a hosted address must start with https://.

Hugging Face (open models, many providers behind one address)

  1. Make a Hugging Face account and create an access token at https://huggingface.co/settings/tokens. Give it the permission to make calls to Inference Providers. Treat it like a password and never commit it.
  2. Pick a model on the Hub that an inference provider serves, and note its id, for example <org>/<model>. Add a provider after a colon to choose who runs it (<org>/<model>:<provider>), or :fastest or :cheapest.
  3. Set these settings:

    ASSISTANT_LLM_PROVIDER=huggingface
    ASSISTANT_LLM_API_KEY=<your Hugging Face token>
    ASSISTANT_LLM_MODEL=<org>/<model>:<provider>
  4. Restart Yaqeen and ask the assistant something. If the reply is the plain "unavailable" message, read the log line for that call: it names the provider and the error class, and never the request text.
  5. Check the model can call tools. If it answers in text but never looks anything up, set ASSISTANT_LLM_TOOLS=false or pick a model that supports tools.

Hugging Face bills usage through its providers. Read the current pricing and your usage page before you rely on it.

A dedicated Hugging Face Inference Endpoint

Deploy the model from the Hub as an endpoint, then use its own address:

ASSISTANT_LLM_PROVIDER=endpoint
ASSISTANT_LLM_BASE_URL=https://<your endpoint address>/v1
ASSISTANT_LLM_API_KEY=<your Hugging Face token>
ASSISTANT_LLM_MODEL=<name the endpoint expects>

The model name depends on the container the endpoint runs, so read it from the endpoint's page.

OpenAI, or any other OpenAI style API

ASSISTANT_LLM_PROVIDER=openai
ASSISTANT_LLM_API_KEY=<your key>
ASSISTANT_LLM_MODEL=<model name>

For any other service that speaks the same format, use ASSISTANT_LLM_PROVIDER=openai_compatible and add ASSISTANT_LLM_BASE_URL=https://<its address>/v1. For a service that does not speak that format, use LiteLLM below.

Sending personal data to a provider outside the Kingdom can count as a transfer under the Saudi PDPL. Masking lowers what leaves. It does not settle the legal question, so ask counsel before you rely on it.

Run vLLM on your own GPU server#

vLLM serves open models fast on a GPU. It is the usual choice when a team hosts a model itself.

  1. Install it on the GPU machine, following https://docs.vllm.ai, then start a server for your model:

    vllm serve <org>/<model> --enable-auto-tool-choice --tool-call-parser <parser> --port 8001

    Without --enable-auto-tool-choice and a parser that matches the model's tool format, the server cannot return tool calls. The vLLM tool calling page lists the parser for each model family.

  2. Use port 8001 or another free one, so it does not clash with other services on the machine.
  3. Set these settings:

    ASSISTANT_LLM_PROVIDER=vllm
    ASSISTANT_LLM_BASE_URL=http://<server address>:8001/v1
    ASSISTANT_LLM_MODEL=<org>/<model>

    Add ASSISTANT_LLM_API_KEY if you started vLLM with --api-key. Plain http is accepted only for local and private addresses, so put the server on a private network or put TLS in front of it.

  4. Text Generation Inference works the same way with ASSISTANT_LLM_PROVIDER=tgi. Its tool support depends on the version, so test it.

Run a model on your own machine with Ollama#

Ollama runs open models on your computer and serves them on http://localhost:11434. Nothing leaves your network and no key is needed. These steps follow Ollama's own docs, so check each one against the version you install.

  1. Install Ollama.
    • macOS: download the app from https://ollama.com/download, or run brew install ollama.
    • Linux: curl -fsSL https://ollama.com/install.sh | sh.
    • Windows: run the installer from the same download page.
  2. Start the server. The macOS and Windows apps start it for you. On Linux, or with Homebrew, run ollama serve in its own terminal. Check it answers: curl http://localhost:11434/api/tags returns a list, empty at first.
  3. Pull a model that can call tools. The assistant needs tool calls to look things up, and not every model has them. On https://ollama.com/search filter by the tools label, then pull one that fits your memory, for example ollama pull llama3.1:8b. Run ollama list to see the exact name.
  4. Check the model answers over the OpenAI style route Yaqeen uses:

    curl http://localhost:11434/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "Say hello"}]}'
  5. Point Yaqeen at it and restart:

    ASSISTANT_LLM_PROVIDER=ollama
    ASSISTANT_LLM_MODEL=llama3.1:8b
    ASSISTANT_LLM_MAX_TOOLS=8

    ASSISTANT_LLM_MAX_TOOLS=8 matters on a small model: the full set of tool definitions is long, and Ollama's default context window is small enough to cut them off. If your Ollama version supports it, start the server with a larger window, for example OLLAMA_CONTEXT_LENGTH=8192 ollama serve.

  6. If Yaqeen runs in Docker, localhost is the container, not your computer. Set ASSISTANT_LLM_BASE_URL=http://host.docker.internal:11434/v1. Plain http is accepted for this name because it is a local address.

When something is wrong:

SymptomLikely cause and fix
Connection refusedThe server is not running. Start the app or run ollama serve.
Model not foundThe name must match ollama list exactly, including the tag.
The assistant answers but never looks anything upThe model cannot call tools. Pick one with the tools label, or set ASSISTANT_LLM_TOOLS=false and accept text only.
The first reply is slowOllama is loading the model into memory. Later replies are faster.
Tool calls look cut off or wrongThe context window is too small. Lower ASSISTANT_LLM_MAX_TOOLS or raise the window.

Yaqeen logs the provider, whether tools were sent, token counts and time for every call, and the error class if a call fails. It never logs the request text.

Small open models and tools#

"OpenAI compatible" describes the wire format, not the behaviour. Open models are weaker than hosted ones at calling several tools at once (Berkeley Function Calling Leaderboard), and a server may not parse a model's tool format at all. Three settings help:

  • ASSISTANT_LLM_TOOLS=false sends no tool definitions. The model then only writes text.
  • ASSISTANT_LLM_MAX_TOOLS=<n> offers only the n tools whose name and summary share the most words with the request. The full set of definitions is a lot for a small model. Default 0 offers all.
  • Reasoning models that put <think>...</think> in the reply are cleaned before the text is used. Tool arguments that arrive as an object instead of a string are accepted. Arguments that are not valid JSON become empty, the tool refuses them, and the model sees the refusal and can try again inside the same limits as any other call.

Up Down to moveEnter to openEsc to close