sankya - Local AI Assistant

sankya — a modern, ChatGPT-style AI assistant that runs entirely locally using open-source Hugging Face models. No cloud APIs, no API keys required for the core model.

What is This?

This is sankya, a fully local AI assistant that runs on your own machine. It uses:

Qwen2.5-1.5B-Instruct - A powerful open-source language model
FastAPI - Modern Python backend
React - Beautiful frontend interface
Docker - Easy deployment with GPU support

No internet required after first download. No API keys. No subscriptions.

Features

Feature	Description
💬 Local LLM	Qwen2.5-1.5B-Instruct runs on your GPU/CPU
⚡ Streaming	Real-time token-by-token responses like ChatGPT
🛠️ AI Tools	Date/time, calculator, weather, web search
📄 RAG	Upload PDFs/DOCX and ask questions about them
💾 Memory	Persistent conversations and user preferences
🎨 Dark/Light Mode	Beautiful ChatGPT-style UI with theme toggle
🔍 Search	Search through your conversation history
📱 Responsive	Works on desktop, tablet, and mobile

Prerequisites

Minimum Requirements

OS: Windows 10/11, macOS 12+, or Linux
RAM: 8GB (16GB recommended)
GPU: NVIDIA GPU with 4GB+ VRAM (optional, but recommended)
Disk: 10GB free space
Docker: Docker Desktop with NVIDIA Container Toolkit (for GPU)

Without GPU

The app works on CPU only, but responses will be slower (~5-10 seconds per message).

Installation

Option A: Docker (Recommended)

This is the easiest method and works on Windows, Mac, and Linux.

Step 1: Install Docker

Windows:

Install Docker Desktop
Go to Settings → Resources → WSL Integration → Enable
Restart Docker Desktop

Mac:

brew install --cask docker

Linux:

sudo apt update
sudo apt install docker.io docker-compose
sudo systemctl start docker
sudo systemctl enable docker

Step 2: Enable NVIDIA GPU (Windows/Linux)

Install NVIDIA Container Toolkit
Verify GPU access:

docker run --rm --gpus all nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 nvidia-smi

Mac users: Skip this step. Apple Silicon uses different architecture.

Step 3: Clone the Repository

git clone https://github.com/Mohitverma-code7/sankya.git
cd ai-chatbot

Step 4: Configure Environment

Create a .env file in the backend folder:

# Windows (PowerShell)
Copy-Item backend\.env.example backend\.env

# Mac/Linux
cp backend/.env.example backend/.env

Edit backend/.env:

# Database
DATABASE_URL=sqlite:///./chatbot.db

# Optional: API keys (leave empty if not needed)
HF_TOKEN=
TAVILY_API_KEY=
SEARCH_API_KEY=
WEATHER_API_KEY=

# Model settings
MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct
MAX_NEW_TOKENS=256
TEMPERATURE=0.3

# Server
PORT=8000
ENVIRONMENT=production

Step 5: Start the Application

docker compose up --build

First run will take 10-15 minutes to:

Build the CUDA-enabled backend image
Build the frontend
Download the Qwen2.5-1.5B-Instruct model (~1.5GB)

Option B: Local Development (No Docker)

If you don't want to use Docker, you can run the backend and frontend directly.

Step 1: Install Python Dependencies

cd backend

# Create virtual environment
python -m venv venv

# Activate virtual environment
# Windows:
venv\Scripts\activate

# Mac/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

Step 2: Install Node.js Dependencies

cd frontend
npm install

Step 3: Start Backend

cd backend
uvicorn main:app --reload

Backend runs on http://localhost:8000

Step 4: Start Frontend (New Terminal)

cd frontend
npm run dev

Frontend runs on http://localhost:5173

Configuration

Environment Variables

All configuration is done through the .env file in the backend folder:

Variable	Required	Default	Description
`ENVIRONMENT`	No	`production`	App environment (`production` or `development`)
`PORT`	No	`8000`	Backend server port
`DATABASE_URL`	No	`sqlite:///./chatbot.db`	Database URL (SQLite or PostgreSQL)
`MODEL_NAME`	No	`Qwen/Qwen2.5-1.5B-Instruct`	Hugging Face model ID
`HF_TOKEN`	No	(empty)	Hugging Face token for private models
`TAVILY_API_KEY`	No	(empty)	Tavily search API key (better than DuckDuckGo)
`SEARCH_API_KEY`	No	(empty)	Alternative search API key
`WEATHER_API_KEY`	No	(empty)	Open-Meteo key (free, usually not needed)
`MAX_NEW_TOKENS`	No	`256`	Maximum tokens per response
`TEMPERATURE`	No	`0.3`	Response creativity (0.1 = focused, 1.0 = creative)
`TOP_P`	No	`0.85`	Nucleus sampling parameter
`REPETITION_PENALTY`	No	`1.1`	Penalty for repeating text

GPU Configuration

The app automatically detects and uses CUDA if available. No configuration needed.

CPU-only mode:

# Force CPU mode (not recommended)
CUDA_VISIBLE_DEVICES=""

Running the App

Docker Deployment

# Start in background
docker compose up -d

# View logs
docker compose logs -f backend
docker compose logs -f frontend

# Stop (preserves data)
docker compose down

# Stop and delete all data
docker compose down -v

Local Development

Terminal 1 (Backend):

cd backend
uvicorn main:app --reload

Terminal 2 (Frontend):

cd frontend
npm run dev

Verifying Installation

1. Check Backend Health

Open browser or curl:

curl http://localhost/api/health

Expected response:

{
  "status": "ok",
  "model_loaded": true,
  "model_name": "Qwen/Qwen2.5-1.5B-Instruct",
  "cuda_available": true,
  "cuda_device": "NVIDIA GeForce RTX 3050 Laptop GPU",
  "gpu_memory_allocated_mb": 2945.3,
  "gpu_memory_total_mb": 4095.5
}

2. Access Frontend

Open http://localhost in your browser.

3. Test Chat

Send a message: "Hello, how are you?"

4. Test Streaming

The response should appear token by token, like ChatGPT.

Using the App

Basic Chat

Type a message in the input box
Press Enter to send
Press Shift+Enter for new line
Click the stop button to cancel generation

Theme Toggle

Click the Sun/Moon icon in the navbar or sidebar
Theme preference is saved automatically

Conversation Management

New Chat: Click "New Chat" in sidebar
Rename: Hover over conversation → Click edit icon
Delete: Hover over conversation → Click trash icon
Search: Use the search box in sidebar

AI Tools

The chatbot automatically detects and uses tools:

Try saying...	What happens
"What is today's date?"	Returns exact date/time
"234 * 56"	Returns calculator result
"Weather in Delhi"	Returns current weather
"Latest AI news"	Searches web and summarizes
"Explain React hooks"	Normal AI response

Document Upload (RAG)

Click the upload button (if implemented in frontend)
Select a PDF, DOCX, or TXT file
Ask questions about the document

User Memory

The chatbot remembers:

Your name ("My name is Mohit")
Your preferences ("I prefer short answers")
Conversation history

Development Mode

Backend Development

cd backend

# Install dependencies
pip install -r requirements.txt

# Run with auto-reload
uvicorn main:app --reload

# API docs available at
# http://localhost:8000/api/docs

Frontend Development

cd frontend

# Install dependencies
npm install

# Run dev server
npm run dev

# Build for production
npm run build

# Preview production build
npm run preview

API Endpoints

Method	Endpoint	Description
GET	`/api/health`	Health check with GPU info
POST	`/api/chat`	Send message (non-streaming)
POST	`/api/chat/stream`	Send message (streaming)
POST	`/api/documents/upload`	Upload document for RAG
GET	`/api/documents`	List uploaded documents
DELETE	`/api/documents/{id}`	Delete document
POST	`/api/memory?user_id={id}`	Set user memory
GET	`/api/memory/{user_id}`	Get user memory
POST	`/api/users`	Create user
GET	`/api/users/{id}/profile`	Get user profile
PATCH	`/api/users/{id}/profile`	Update user profile

Project Structure

ai-chatbot/
├── backend/
│   ├── main.py                 # FastAPI routes & startup
│   ├── model.py                # Model loading & generation
│   ├── chatbot.py              # Intent routing & tool orchestration
│   ├── config.py               # Settings & environment variables
│   ├── database.py             # SQLAlchemy setup
│   ├── streaming.py            # SSE streaming handler
│   ├── tools/
│   │   ├── datetime_tool.py    # Python datetime (no LLM)
│   │   ├── calculator.py       # Safe AST math (no LLM)
│   │   ├── weather.py          # Open-Meteo weather API
│   │   ├── search.py           # DuckDuckGo / Tavily search
│   │   └── __init__.py
│   ├── router/
│   │   └── intent_router.py    # Regex-based intent detection
│   ├── services/
│   │   ├── tool_service.py     # Tool execution layer
│   │   ├── memory_service.py   # SQLAlchemy CRUD
│   │   └── streaming.py        # SSE streaming handler
│   ├── models/
│   │   └── __init__.py         # SQLAlchemy ORM models
│   ├── rag/
│   │   ├── __init__.py
│   │   ├── document_loader.py  # PDF/DOCX/TXT parsing
│   │   ├── embeddings.py       # Sentence-transformers wrapper
│   │   ├── vectorstore.py      # ChromaDB persistence
│   │   └── retriever.py        # Chunking + retrieval
│   ├── requirements.txt
│   ├── Dockerfile
│   ├── .dockerignore
│   └── .env.example
├── frontend/
│   ├── src/
│   │   ├── components/         # UI components
│   │   │   ├── Sidebar.tsx
│   │   │   ├── ChatArea.tsx
│   │   │   ├── InputArea.tsx
│   │   │   ├── MessageBubble.tsx
│   │   │   ├── WelcomeScreen.tsx
│   │   │   └── theme/          # Theme system
│   │   ├── pages/
│   │   │   └── ChatPage.tsx    # Main chat page
│   │   ├── hooks/
│   │   │   └── useChat.ts      # Chat state management
│   │   ├── services/
│   │   │   └── api.ts          # API client
│   │   ├── types/
│   │   │   └── index.ts        # TypeScript interfaces
│   │   ├── App.tsx
│   │   └── main.tsx
│   ├── package.json
│   ├── Dockerfile
│   ├── nginx.conf
│   ├── vite.config.ts
│   └── tailwind.config.js
├── docker-compose.yml
├── .gitignore
└── README.md

Troubleshooting

Docker Build Fails

Problem: docker build fails with CUDA or PyTorch errors

Solution:

# Ensure NVIDIA Container Toolkit is installed
docker run --rm --gpus all nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 nvidia-smi

# If it fails, install toolkit:
# Windows/Mac: Docker Desktop Settings → Enable GPU
# Linux: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Model Download Fails

Problem: First run gets stuck downloading model

Solution:

# Pre-download the model
docker compose run --rm backend python -c "from model import load_model; load_model()"

# Or manually download from Hugging Face:
# https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct

Out of Memory (OOM)

Problem: GPU runs out of memory

Solution:

Reduce MAX_NEW_TOKENS in .env:
```
MAX_NEW_TOKENS=128
```
Close other GPU applications
Restart Docker

Frontend Shows Blank Page

Problem: http://localhost shows blank page

Solution:

# Check frontend logs
docker compose logs frontend

# Rebuild frontend
docker compose up --build frontend

# Clear browser cache and reload

404 on `/api/health`

Problem: Frontend can't connect to backend

Solution:

# Check backend is running
curl http://localhost:8000/health

# Check docker compose
docker compose ps

# Restart services
docker compose restart

Slow Responses

Problem: AI takes too long to respond

Solution:

Ensure GPU is being used (check /api/health)
Reduce MAX_NEW_TOKENS to 128
Reduce MAX_HISTORY_MESSAGES in config.py
Use a smaller model (e.g., Qwen/Qwen2.5-0.5B-Instruct)

FAQ

Do I need an API key?

No. The app runs completely locally. Optional API keys (Tavily, etc.) are only for enhanced features.

Is my data private?

Yes. Everything runs on your machine. No data is sent to the cloud.

Can I use this without a GPU?

Yes, but it will be slower. CPU-only mode works fine for testing.

How much disk space do I need?

Model: ~1.5GB
Docker images: ~8GB
Database: ~100MB
Total: ~10GB recommended

Can I use a different model?

Yes. Change MODEL_NAME in .env:

MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct  # Smaller, faster
MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct  # Default
MODEL_NAME=Qwen/Qwen2.5-7B-Instruct    # Larger, smarter (needs 8GB+ VRAM)

How do I update the app?

git pull origin main
docker compose up --build -d

Can I backup my chats?

Yes. Copy these files:

backend/chatbot.db (conversations)
backend/.env (configuration)
Docker volume huggingface-cache (model cache)

Is this free?

Yes. 100% free and open source. No costs, no subscriptions.

Architecture

┌─────────────┐     /api/*      ┌──────────────┐
│   Browser   │ ◄──────────────► │   Nginx      │
│  (React)    │                 │  (Frontend)  │
└─────────────┘                 └──────┬───────┘
                                          │
                                          │ proxy
                                          ▼
┌─────────────────────────────────────────────────────┐
│                   Docker Compose                    │
│                                                     │
│  ┌──────────────┐    ┌───────────────────────────┐  │
│  │   Backend    │    │         Frontend          │  │
│  │  (FastAPI)   │    │      (Nginx + React)      │  │
│  │              │    │                           │  │
│  │  ┌────────┐  │    │  /api/chat  ──────────►  │  │
│  │  │ Model  │  │    │  /api/stream ─────────►  │  │
│  │  │ Qwen   │  │    │  /api/health ─────────►  │  │
│  │  │ 2.5B   │  │    │                           │  │
│  │  └────────┘  │    │                           │  │
│  │              │    │                           │  │
│  │  ┌────────┐  │    │                           │  │
│  │  │ RAG    │  │    │                           │  │
│  │  │ChromaDB│  │    │                           │  │
│  │  └────────┘  │    │                           │  │
│  │              │    │                           │  │
│  │  ┌────────┐  │    │                           │  │
│  │  │ Tools  │  │    │                           │  │
│  │  │Datetime │  │    │                           │  │
│  │  │Weather │  │    │                           │  │
│  │  │Search  │  │    │                           │  │
│  │  │Calc    │  │    │                           │  │
│  │  └────────┘  │    │                           │  │
│  │              │    │                           │  │
│  │  ┌────────┐  │    │                           │  │
│  │  │ Database│  │    │                           │  │
│  │  │ SQLite │  │    │                           │  │
│  │  └────────┘  │    │                           │  │
│  └──────────────┘    └───────────────────────────┘  │
│           ▲                                         │
│           │ persistent                              │
│           ▼                                         │
│  ┌─────────────────┐                               │
│  │ HuggingFace     │                               │
│  │ Model Cache     │                               │
│  │ (/cache/hf)     │                               │
│  └─────────────────┘                               │
└─────────────────────────────────────────────────────┘

Intent Routing

The chatbot uses a lightweight router to decide whether to use tools or the LLM:

Intent	Trigger	Action
`datetime`	"what date", "today", "current time"	Returns exact Python datetime, bypasses LLM
`calculator`	Pure math expression like `234 * 56`	Safe AST eval, bypasses LLM
`weather`	"weather in Delhi", "temperature"	Open-Meteo API + LLM summary
`search`	"latest news", "search for"	DuckDuckGo/Tavily + LLM summary
`normal`	Everything else	Qwen2.5-1.5B-Instruct

Testing

Docker

# Start services
docker compose up --build

# Health check
curl http://localhost/api/health

# Chat
curl -X POST http://localhost/api/chat \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}]}'

# Streaming
curl -X POST http://localhost/api/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}]}'

Local

# Backend health
curl http://localhost:8000/health

# Chat
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What is 234 * 56?"}]}'

License

MIT

Support

If you encounter issues:

Check Troubleshooting section
Search existing GitHub Issues
Create a new issue with:
- OS and version
- GPU model (if applicable)
- Docker version
- Full error logs

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Fork the project
Create your feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

Acknowledgments

Qwen2.5 - The local LLM
FastAPI - Backend framework
React - Frontend library
Tailwind CSS - Styling
Nginx - Frontend proxy & static serving
Docker - Containerization
ChromaDB - Vector storage
Sentence Transformers - Embeddings

Star History

If this project helps you, please give it a ⭐️ on GitHub!

Name		Name	Last commit message	Last commit date
backend		backend
frontend		frontend
.gitignore		.gitignore
INSTALL.txt		INSTALL.txt
README.md		README.md
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json

Latest commit

History

Folders and files

Repository files navigation

sankya - Local AI Assistant

Table of Contents

What is This?

Features

Prerequisites

Minimum Requirements

Without GPU

Installation

Option A: Docker (Recommended)

Step 1: Install Docker

Step 2: Enable NVIDIA GPU (Windows/Linux)

Step 3: Clone the Repository

Step 4: Configure Environment

Step 5: Start the Application

Option B: Local Development (No Docker)

Step 1: Install Python Dependencies

Step 2: Install Node.js Dependencies

Step 3: Start Backend

Step 4: Start Frontend (New Terminal)

Configuration

Environment Variables

GPU Configuration

Running the App

Docker Deployment

Local Development

Verifying Installation

1. Check Backend Health

2. Access Frontend

3. Test Chat

4. Test Streaming

Using the App

Basic Chat

Theme Toggle

Conversation Management

AI Tools

Document Upload (RAG)

User Memory

Development Mode

Backend Development

Frontend Development

API Endpoints

Project Structure

Troubleshooting

Docker Build Fails

Model Download Fails

Out of Memory (OOM)

Frontend Shows Blank Page

404 on /api/health

Slow Responses

FAQ

Do I need an API key?

Is my data private?

Can I use this without a GPU?

How much disk space do I need?

Can I use a different model?

How do I update the app?

Can I backup my chats?

Is this free?

Architecture

Intent Routing

Testing

Docker

Local

License

Support

Contributing

Acknowledgments

Star History

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

404 on `/api/health`