Skip to content

Commit cbb8ca6

Browse files
committed
Update README to note latest recommended models, closes #14
1 parent 09b7e7c commit cbb8ca6

File tree

1 file changed

+14
-17
lines changed

1 file changed

+14
-17
lines changed

README.md

Lines changed: 14 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -63,25 +63,21 @@ This project also works well with papers from [PubMed](https://pubmed.ncbi.nlm.n
6363

6464
### Setup
6565

66-
Install the following.
67-
68-
```bash
69-
# Change autoawq[kernels] to "autoawq autoawq-kernels" if a flash-attn error is raised
70-
pip install annotateai autoawq[kernels]
71-
72-
# macOS users should run this instead
73-
pip install annotateai llama-cpp-python
74-
```
75-
7666
The primary input parameter is the path to the LLM. This project is backed by [txtai](https://github.com/neuml/txtai) and it supports any [txtai-supported LLM](https://neuml.github.io/txtai/pipeline/text/llm/).
7767

7868
```python
7969
from annotateai import Annotate
8070

81-
# This model works well with medical and scientific literature
71+
# Lightweight but powerful default model
72+
annotate = Annotate("Qwen/Qwen3-4B-Instruct-2507")
73+
74+
# The previous default model uses the now deprecated AutoAWQ library
75+
# Run pip install autoawq to enable
76+
# Note as time goes on, this may require pinning to older versions of transformers & torch
8277
annotate = Annotate("NeuML/Llama-3.1_OpenScholar-8B-AWQ")
8378

84-
# macOS users should run this instead
79+
# llama.cpp version of the above model
80+
# Run pip install llama-cpp-python to enable
8581
annotate = Annotate(
8682
"bartowski/Llama-3.1_OpenScholar-8B-GGUF/Llama-3.1_OpenScholar-8B-Q4_K_M.gguf"
8783
)
@@ -133,15 +129,16 @@ pip install txtai[pipeline-llm]
133129

134130
```python
135131
# LLM API services
136-
annotate = Annotate("gpt-4o")
137-
annotate = Annotate("claude-3-5-sonnet-20240620")
132+
annotate = Annotate("gpt-5.1")
133+
annotate = Annotate("claude-opus-4-5-20251101")
134+
annotate = Annotate("gemini/gemini-3-pro-preview")
138135

139136
# Ollama endpoint
140-
annotate = Annotate("ollama/llama3.1")
137+
annotate = Annotate("ollama/gpt-oss")
141138

142139
# llama.cpp GGUF from Hugging Face Hub
143140
annotate = Annotate(
144-
"bartowski/Llama-3.1_OpenScholar-8B-GGUF/Llama-3.1_OpenScholar-8B-Q4_K_M.gguf"
141+
"unsloth/gpt-oss-20b-GGUF/gpt-oss-20b-Q4_K_M.gguf"
145142
)
146143
```
147144

@@ -176,7 +173,7 @@ docker run -d --gpus=all -it -p 8501:8501 neuml/annotateai
176173
The LLM can also be set via ENV parameters.
177174

178175
```
179-
docker run -d --gpus=all -it -p 8501:8501 -e LLM=bartowski/Llama-3.2-1B-Instruct-GGUF/Llama-3.2-1B-Instruct-Q4_K_M.gguf neuml/annotateai
176+
docker run -d --gpus=all -it -p 8501:8501 -e LLM=unsloth/gpt-oss-20b-GGUF/gpt-oss-20b-Q4_K_M.gguf -e MAXLENGTH=10000 -e n_ctx=4096 neuml/annotateai
180177
```
181178

182179
The code for this application can be found in the [app folder](https://github.com/neuml/annotateai/tree/master/app).

0 commit comments

Comments
 (0)