Danswer can also make requests to an arbitrary model server via REST requests. Optionally an access token can be passed in. To customize the request format and handling of the response, it may be necessary to update/rebuild the Danswer containers.

​
Extending Danswer to be compatible with your custom model server

There’s a very minimal interface to be implemented which can support any arbitrary LLM Model Server. Simply update the code here and rebuild. The default implementation is compatible with the blog demo shown below.

​
Danswer with self hosted Llama-2-13B-chat-GGML using a custom FastAPI Server.

  • See the Medium blog post.
  • This demo uses Google Colab to access a free GPU but this is not suitable for long term deployments

​
Setting Danswer to use the custom implementation:

GEN_AI_MODEL_PROVIDER=custom
GEN_AI_API_KEY=<your-model-server-api-key-or-leave-unset>
GEN_AI_API_ENDPOINT=<your-custom-model-endpoint-url>