🤟 ISLCraft — Indian Sign Language & Gesture AI Translator

An intelligent, real-time Indian Sign Language (ISL) and hand gesture recognition platform built with PyTorch Deep CNN, OpenCV, and Flask. Features live browser webcam recognition with target ROI tracking, bilingual speech synthesis (English & Hindi), an interactive Sentence Builder, and a 1-Click Test Gallery for instant gesture recognition without requiring a webcam.

---

✨ Key Features

• 🧠 PyTorch Deep CNN Architecture: 4 convolutional blocks with Batch Normalization, Dropout regularization, and Cosine Annealing learning rate schedule trained on 3,600 gesture samples.
• 📹 Dual Webcam Modes:
  • Client-Side HTML5 Camera: Captures video frames directly in the browser via Canvas ROI extraction and posts to /predict_frame (zero lag and 100% cloud deployment compatibility on Render/Heroku).
  • Server-Side OpenCV Stream: Fallback MJPEG video stream with bounding ROI box.
• 🗣️ Bilingual Speech Synthesis (TTS):
  • English & Hindi audio voice output with one-click language toggle.
  • Hotkey support (S to speak detected gesture aloud, Space to append to sentence).
• 📝 Interactive Sentence Builder:
  • Real-time sentence composition bar accumulating sequential gestures.
  • Actions: Speak Sentence, Copy Text, Undo, and Clear.
• ⚡ 1-Click Sample Gesture Gallery: Instant one-click test cards for all 9 gesture classes for quick testing on devices without physical webcams.
• 📚 ISL Library & Reference Modal: Gesture meanings, cultural context, and bilingual translations.
• 🧪 100% Automated Test Coverage: Comprehensive PyTorch model & API test suite with pytest.
• 🐳 Containerized & Cloud Ready: Complete with Dockerfile, docker-compose.yml, Procfile, and .render.yaml.

---

🖐️ Supported 9 Gesture Classes

#	Gesture	Emoji	Meaning	English Response	Hindi Translation
1	Thumbs Up	👍	Approval, agreement, success	"You're doing great! Approved!"	सब बढ़िया है!
2	Thumbs Down	👎	Disapproval, negative	"Not cool! Try again!"	कुछ गड़बड़ है! 👎
3	Victory / Peace	✌️	Peace, victory, celebration	"Victory is yours! ✌️"	जीत आपकी है! ✌️
4	Okay Sign	👌	Perfect, confirmation	"Alright then, perfect! 👌"	सब ठीक है! 👌
5	Finger Snap	🫰	Snappy rhythm, recognition	"Snap magic! Smooth! ✨"	चुटकी में हो गया! ✨
6	Finger Gun	👉	Pointing, target locked	"Gotcha! Target locked! 🎯"	निशाना सटीक है! 🎯
7	Salute	🫡	Respect, duty, acknowledgment	"At your service, captain! 🫡"	सादर प्रणाम! 🫡
8	Crossed Fingers	🤞	Good luck, hope	"Fingers crossed! Best of luck! 🤞"	शुभकामनाएं! 🤞
9	Waving Hand	👋	Hello, greeting, farewell	"Hello! Hey there buddy! 👋"	नमस्ते मित्र! 👋

---

🏛️ Architecture Overview

 ┌────────────────────────────────────────────────────────────────────────┐
 │                           Client-Side Web UI                           │
 │                                                                        │
 │   ┌───────────────────────┐            ┌───────────────────────────┐   │
 │   │  HTML5 Webcam Canvas  │            │   Interactive Sentence    │   │
 │   │   Target Zone (ROI)   │            │         Builder           │   │
 │   └───────────┬───────────┘            └─────────────▲─────────────┘   │
 └───────────────┼──────────────────────────────────────┼─────────────────┘
                 │ (Base64 JPEG POST)                   │
                 ▼                                      │
 ┌──────────────────────────────────────────────────────┴─────────────────┐
 │                       Flask Backend (/predict_frame)                   │
 │                                                                        │
 │  ┌──────────────────────────────────────────────────────────────────┐  │
 │  │                Image Preprocessing Pipeline                      │  │
 │  │     Grayscale Conversion ➔ Resize (64x64) ➔ Normalized Float32   │  │
 │  └──────────────────────────────────┬───────────────────────────────┘  │
 │                                     │ (1, 1, 64, 64) Tensor            │
 │                                     ▼                                  │
 │  ┌──────────────────────────────────────────────────────────────────┐  │
 │  │              PyTorch 4-Block Deep CNN Classifier                 │  │
 │  │   Conv2D + BatchNorm + MaxPool + Dropout ➔ Linear Classification  │  │
 │  │                (Trained Weights: gesture_model.pt)               │  │
 │  └──────────────────────────────────┬───────────────────────────────┘  │
 │                                     │ Softmax Class Probabilities      │
 │                                     ▼                                  │
 │  ┌──────────────────────────────────────────────────────────────────┐  │
 │  │     Bilingual Translation & Confidence Score Metadata Engine     │  │
 │  └──────────────────────────────────────────────────────────────────┘  │
 └────────────────────────────────────────────────────────────────────────┘

---

🚀 Quick Start

1. Clone & Install Dependencies

# Clone the repository
git clone https://github.com/harshrameshnerkar/ISL-Translator.git
cd ISL-Translator

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate    # On Windows: .venv\Scripts\activate

# Install requirements
pip install -r requirements.txt

2. Run the Web Application

python app.py

Visit http://localhost:5000 in your browser to launch the web dashboard!

---

🧪 Running Automated Tests

Run the full automated test suite using pytest:

pytest -v tests/

All 9 tests verify PyTorch model tensor output dimensions, image normalization, confidence score calculations, and Flask REST endpoints.

---

🐳 Docker Deployment

Using Docker Compose

docker-compose up -d --build

Using Docker CLI

docker build -t isl-translator-app .
docker run -p 5000:5000 isl-translator-app

---

☁️ Deploy to Render / Railway

1. Push your code to your GitHub repository.
2. In Render, create a new Web Service and connect this repository.
3. Render will auto-detect .render.yaml and deploy with Gunicorn!

---

📂 Project Structure

7_ISL_translator/
├── app.py                      # Flask backend + PyTorch CNN inference & REST API
├── train_torch_model.py        # PyTorch training pipeline script
├── models/
│   └── gesture_model.pt        # Trained 4-block CNN weights (5.9MB)
├── templates/
│   └── index.html              # Glassmorphic Web Dashboard
├── static/
│   ├── css/
│   │   └── style.css           # Glassmorphism design system & responsiveness
│   ├── js/
│   │   └── script.js           # Client-side camera, Canvas ROI, TTS, & Sentence Builder
│   ├── samples/                # 9 pre-rendered sample gesture images
│   └── uploads/                # Directory for user-uploaded test photos
├── tests/
│   ├── __init__.py
│   └── test_model.py           # PyTorch & Flask automated test suite
├── .gitignore                  # Excludes large raw datasets (*.npz), uploads, caches
├── Dockerfile                  # Container definition
├── docker-compose.yml          # 1-command Docker setup
├── Procfile                    # Cloud process runner
├── .render.yaml                # Render cloud deployment blueprint
├── requirements.txt            # Dependencies
├── LICENSE                     # MIT License
└── README.md                   # Documentation

---

📜 License

This project is open source and available under the MIT License.

---

Made with ❤️ by Harsh Nerkar