Windows

Install Docker Desktop

Install Chocolatey

Open a command prompt: Ctrl + R

Install jq

choco install jq -y

Install curl

choco install curl -y

Install gît

Choco install git -y

Clone the repository

gît clone https://github.com/Smartappli/LLM_SERVER.git

Launch Docker Desktop

Docker volume creation

cd LLM_SERVER
cd Docker
create_docker_volume.bat

Build Docker image for Llama CPP Python Server - CPU with OpenBlast

cd cpu
docker build -t smartappli/llama-cpp-python-server-cpu:1.0 .

Build Docker image for Llama CPP Python Server - CUDA with OpenBlast

cd ..
cd cuda
docker build -t smartappli/llama-cpp-python-server-cuda:1.0 .

Run Llama cpp python server CPU

docker run -v LLM_SERVER:/models smartappli/llama-cpp-python-server-cpu

or

Run Llama cpp python server GPU

docker run -v LLM_SERVER:/models smartappli/llama-cpp-python-server-cuda

install dépendances

pip install -r requirements.txt

Launch tests

cd ..
python main.py

Ubuntu 24.04

Install Wget, jq, and git

apt install update
apt install wget jq git 

Clone the repository

gît clone https://github.com/Smartappli/LLM_SERVER.git

Docker volume creation

cd LLM_SERVER
cd Docker
sudo chmod +x create_docker_volume.sh
sudo ./create_docker_volume.sh

Build Docker image for Llama CPP Python Server - CPU with OpenBlast

cd cpu
docker build -t smartappli/llama-cpp-python-server-cpu:1.0 .

Build Docker image for Llama CPP Python Server - CUDA with OpenBlast

cd ..
cd cuda
docker build -t smartappli/llama-cpp-python-server-cuda:1.0 .

Run Llama cpp python server CPU

docker run -v LLM_SERVER:/models smartappli/llama-cpp-python-server-cpu

or

Run Llama cpp python server GPU

docker run -v LLM_SERVER:/models smartappli/llama-cpp-python-server-cuda

install dépendances

pip install -r requirements.txt

Launch tests

cd ..
python main.py