Skip to content

Commit 640d308

Browse files
guygirguygir
authored
Optimized chat completions library, build support and testing infrastructure (#79)
* Optimized library step and start of b.2.2 step * Completed step 2.2. Link python and install requirements in setup (Makefile for local runs, Dockerfile for building) * Added template caching, unit and e2e tests, thread-safe testing. Cleaned code. Fixed critical template_vars bug. * Added benchmarking, fixed lint errors, removed redundant global tracking in Go, used Logger instead of prints * cgo_functions.go changes didn't commit earlier * Fixed lint errors * Fixed lint errors * Last lint fix * Last lint fix...
1 parent 0adf094 commit 640d308

File tree

11 files changed

+1730
-268
lines changed

11 files changed

+1730
-268
lines changed

‎Dockerfile‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ RUN dnf install -y 'https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.
2626
dnf install -y gcc-c++ libstdc++ libstdc++-devel clang zeromq-devel pkgconfig && \
2727
dnf clean all
2828

29+
# Install Python 3.9+ for chat template functionality
30+
RUN dnf install -y python39 python39-pip python39-devel && \
31+
dnf clean all
32+
2933
# Copy the Go Modules manifests
3034
COPY go.mod go.mod
3135
COPY go.sum go.sum
@@ -37,6 +41,9 @@ RUN go mod download
3741
COPY examples/kv_events examples/kv_events
3842
COPY . .
3943

44+
# Install Python dependencies for chat template functionality
45+
RUN python3.9 -m pip install -r pkg/preprocessing/chat_completions_template/requirements.txt
46+
4047
# HuggingFace tokenizer bindings
4148
RUN mkdir -p lib
4249
ARG RELEASE_VERSION=v1.22.1
@@ -59,9 +66,14 @@ WORKDIR /
5966
# The final image is UBI9, so we need epel-release-9.
6067
USER root
6168
RUN dnf install -y 'https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm' && \
62-
dnf install -y zeromq
69+
dnf install -y zeromq python39 python39-pip
70+
71+
# Install Python dependencies in the final image
72+
COPY pkg/preprocessing/chat_completions_template/requirements.txt /tmp/requirements.txt
73+
RUN python3.9 -m pip install -r /tmp/requirements.txt
6374

6475
COPY --from=builder /workspace/bin/kv-cache-manager /app/kv-cache-manager
76+
COPY --from=builder /workspace/pkg/preprocessing/chat_completions_template/chat_template_wrapper.py /app/chat_template_wrapper.py
6577
USER 65532:65532
6678

6779
# Set the entrypoint to the kv-cache-manager binary

‎Makefile‎

Lines changed: 63 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -75,19 +75,70 @@ verify-boilerplate: $(TOOLS_DIR)/verify_boilerplate.py
7575
$(TOOLS_DIR)/verify_boilerplate.py --boilerplate-dir=hack/boilerplate --skip docs
7676

7777
.PHONY: unit-test
78-
unit-test: download-tokenizer download-zmq
78+
unit-test: download-tokenizer detect-python install-python-deps download-zmq
7979
@printf "\033[33;1m==== Running unit tests ====\033[0m\n"
8080
go test -ldflags="$(LDFLAGS)" ./pkg/...
81+
@printf "\033[33;1m==== Running chat template tests ====\033[0m\n"
82+
go test -tags=exclude -v -ldflags="$(LDFLAGS)" ./pkg/preprocessing/chat_completions_template/
83+
@printf "\033[33;1m==== Running chat template benchmarks ====\033[0m\n"
84+
go test -tags=exclude -bench=. -benchmem -ldflags="$(LDFLAGS)" ./pkg/preprocessing/chat_completions_template/
8185

8286
.PHONY: e2e-test
83-
e2e-test: download-tokenizer download-zmq
84-
@printf "\033[33;1m==== Running unit tests ====\033[0m\n"
87+
e2e-test: download-tokenizer detect-python install-python-deps download-zmq
88+
@printf "\033[33;1m==== Running e2e tests ====\033[0m\n"
8589
go test -v -ldflags="$(LDFLAGS)" ./tests/...
8690

8791
##@ Build
8892

93+
# Python detection and build configuration
94+
PYTHON_VERSION := 3.11.7
95+
PYTHON_DIR = build/python-$(PYTHON_VERSION)
96+
97+
.PHONY: detect-python
98+
detect-python:
99+
@printf "\033[33;1m==== Detecting Python installation ====\033[0m\n"
100+
@if python3 -c "import sys; print(sys.version)" >/dev/null 2>&1; then \
101+
echo "Using system Python"; \
102+
PYTHON_PATH=$$(python3 -c "import sys; print(sys.prefix)"); \
103+
PYTHON_VERSION=$$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')"); \
104+
sed -i.bak "s|{{PYTHON_PATH}}|$$PYTHON_PATH|g; s|{{PYTHON_VERSION}}|$$PYTHON_VERSION|g" \
105+
pkg/preprocessing/chat_completions_template/cgo_functions.go; \
106+
rm -f pkg/preprocessing/chat_completions_template/cgo_functions.go.bak; \
107+
else \
108+
echo "System Python not found, downloading..."; \
109+
$(MAKE) download-python; \
110+
fi
111+
112+
.PHONY: download-python
113+
download-python:
114+
@printf "\033[33;1m==== Downloading Python $(PYTHON_VERSION) ====\033[0m\n"
115+
@mkdir -p build
116+
@if [ ! -d "$(PYTHON_DIR)" ]; then \
117+
if [ "$(TARGETOS)" = "darwin" ]; then \
118+
curl -L https://www.python.org/ftp/python/$(PYTHON_VERSION)/python-$(PYTHON_VERSION)-macos11.pkg -o build/python.pkg; \
119+
sudo installer -pkg build/python.pkg -target /; \
120+
elif [ "$(TARGETOS)" = "linux" ]; then \
121+
curl -L https://www.python.org/ftp/python/$(PYTHON_VERSION)/Python-$(PYTHON_VERSION).tgz -o build/python.tgz; \
122+
tar -xzf build/python.tgz -C build/; \
123+
cd build/Python-$(PYTHON_VERSION) && ./configure --prefix=$(PWD)/$(PYTHON_DIR) && make && make install; \
124+
fi; \
125+
fi
126+
@# Update CGo flags with downloaded Python path
127+
@sed -i.bak "s|{{PYTHON_PATH}}|$(PWD)/$(PYTHON_DIR)|g; s|{{PYTHON_VERSION}}|$(PYTHON_VERSION)|g" \
128+
pkg/preprocessing/chat_completions_template/cgo_functions.go
129+
@rm -f pkg/preprocessing/chat_completions_template/cgo_functions.go.bak
130+
131+
.PHONY: install-python-deps
132+
install-python-deps: detect-python
133+
@printf "\033[33;1m==== Installing Python dependencies ====\033[0m\n"
134+
@if [ -d "$(PYTHON_DIR)" ]; then \
135+
$(PYTHON_DIR)/bin/pip install -r pkg/preprocessing/chat_completions_template/requirements.txt; \
136+
else \
137+
python3 -m pip install -r pkg/preprocessing/chat_completions_template/requirements.txt; \
138+
fi
139+
89140
.PHONY: build
90-
build: check-go download-tokenizer download-zmq
141+
build: check-go download-tokenizer detect-python install-python-deps download-zmq
91142
@printf "\033[33;1m==== Building ====\033[0m\n"
92143
go build -ldflags="$(LDFLAGS)" -o bin/$(PROJECT_NAME) examples/kv_cache_index/main.go
93144

@@ -351,6 +402,14 @@ print-namespace: ## Print the current namespace
351402
print-project-name: ## Print the current project name
352403
@echo "$(PROJECT_NAME)"
353404

405+
.PHONY: clean
406+
clean: ## Clean build artifacts and restore placeholders
407+
@printf "\033[33;1m==== Cleaning build artifacts ====\033[0m\n"
408+
@rm -rf build/
409+
@# Restore original placeholders
410+
@git checkout pkg/preprocessing/chat_completions_template/cgo_functions.go
411+
@echo "✅ Build artifacts cleaned and placeholders restored"
412+
354413
.PHONY: install-hooks
355414
install-hooks: ## Install git hooks
356415
git config core.hooksPath hooks

‎pkg/preprocessing/chat_completions_template/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -62,16 +62,16 @@ The templating process (steps 1.1-1.4) handles the conversion from structured re
6262
└── cgo_functions.go:NewChatTemplateCGoWrapper()
6363
└── Creates ChatTemplateCGoWrapper struct with initialized=false
6464
65-
1.2. **Template Fetching**: wrapper.GetModelChatTemplate(getReq)
66-
├── cgo_functions.go:GetModelChatTemplate(req)
65+
1.2. **Template Fetching**: wrapper.GetModelChatTemplate(ctx, getReq)
66+
├── cgo_functions.go:GetModelChatTemplate(ctx, req)
6767
│ ├── Initialize() Python interpreter via CGO
6868
│ ├── executePythonCode() - **CGO Binding** to Python
6969
│ └── **Python Wrapper**: chat_template_wrapper.py:get_model_chat_template()
7070
│ └── Uses Hugging Face AutoTokenizer to fetch model template
7171
└── Returns: (template, template_vars)
7272
73-
1.3. **Template Rendering**: wrapper.RenderChatTemplate(req)
74-
├── cgo_functions.go:RenderChatTemplate(req)
73+
1.3. **Template Rendering**: wrapper.RenderChatTemplate(ctx, req)
74+
├── cgo_functions.go:RenderChatTemplate(ctx, req)
7575
│ ├── Initialize() Python interpreter via CGO (if not already done)
7676
│ ├── executePythonCode() - **CGO Binding** to Python
7777
│ └── **Python Wrapper**: chat_template_wrapper.py:render_jinja_template()

0 commit comments

Comments
 (0)