Skip to content

CerebraLake is an autonomous, self-optimising data lake engine built with Rust, Java, AWS, and BigQuery, designed to learn, predict, and optimise data workflows automatically.

Notifications You must be signed in to change notification settings

Jitterx69/CerebraLake

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

7 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

CerebraLake

Rust Java React

A groundbreaking multi-language control-plane and data orchestration system that unifies Rust's performance, Java's reliability, and React's interactivity into one cohesive architecture.


πŸ“‹ Table of Contents


🌍 Overview

CerebraLake is a modular, real-time monitoring and control system designed for high-performance distributed environments. By integrating three powerful technologies, it delivers a seamless experience for data ingestion, orchestration, and visualization.

Why CerebraLake?

  • πŸ¦€ Rust Core: Lightning-fast data processing and low-level system control
  • β˜• Java Control Plane: Enterprise-grade orchestration and coordination
  • βš›οΈ React Dashboard: Beautiful, real-time system monitoring

Each layer operates independently while maintaining seamless integration through REST APIs and file-based discovery.


πŸ›οΈ Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                     React Dashboard                           β”‚
β”‚              (Real-time Monitoring Interface)                 β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                         β”‚ REST API / WebSocket
                         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚              Java Control Plane (Spring Boot)                 β”‚
β”‚  β€’ Orchestration Logic        β€’ State Management             β”‚
β”‚  β€’ Service Discovery          β€’ Health Monitoring            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                         β”‚ HTTP / File I/O
                         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   Rust Core (Axum Server)                     β”‚
β”‚  β€’ High-throughput Data Ingestion  β€’ Metrics Collection      β”‚
β”‚  β€’ Dynamic Port Allocation         β€’ Health Endpoints        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Communication Flow

Layer Technology Primary Responsibility
πŸ¦€ Data Plane Rust (Axum, Tokio) Low-level data ingestion, processing, and real-time metric emission
β˜• Control Plane Spring Boot Service coordination, orchestration, and state management
βš›οΈ Presentation React + Vite Real-time visualization and system monitoring

✨ Key Features

πŸ”„ Auto-Discovery System

  • Dynamic port allocation with automatic service discovery
  • Zero-configuration inter-service communication
  • File-based port registry (active_port.txt, java_active_port.txt)

⚑ Real-Time Monitoring

  • Live metrics dashboard with auto-refresh
  • Health status indicators for all services
  • Connection visualization and uptime tracking

πŸ”— Cross-Language Integration

  • Seamless Rust ↔ Java communication via REST
  • React consumes metrics from both backend services
  • Standard HTTP/REST patterns for universal compatibility

πŸš€ Stateless Architecture

  • Each component runs independently
  • Automatic reconnection and service recovery
  • No complex configuration required

πŸ“Š Clean Observability

  • Structured logging across all layers
  • Prometheus-compatible metrics endpoints
  • Centralized health monitoring

πŸ› οΈ Tech Stack

Component Technologies Purpose
Backend - Rust Axum, Tokio, Serde, Hyper High-performance async data plane
Backend - Java Spring Boot, WebFlux, Maven Reactive orchestration layer
Frontend React 18, TypeScript, Vite, TailwindCSS Modern, responsive UI
Integration REST APIs, File I/O Cross-language communication

πŸš€ Getting Started

Prerequisites

Ensure you have the following installed:

Installation

Clone the repository:

git clone https://github.com/Jitterx69/CerebraLake.git
cd CerebraLake

Running the Services

Option 1: Manual Startup

1. Start the Rust Core

cd rust-core
cargo run

The Rust service will:

  • Automatically select a free port
  • Write the port to active_port.txt
  • Expose endpoints at /metrics and /health

2. Start the Java Control Plane

cd java-controlplane
mvn spring-boot:run

The Java service will:

  • Read Rust's port from active_port.txt
  • Write its own port to java_active_port.txt
  • Start the orchestration layer

3. Launch the React Dashboard

cd react-dashboard
npm install
npm run dev

Open your browser at http://localhost:3000 to view the dashboard.

Option 2: One-Command Startup (Recommended)

Use the provided startup script:

chmod +x run_all.sh
./run_all.sh

This will start all three services in the correct order.


πŸ“ Project Structure

CerebraLake/
β”œβ”€β”€ rust-core/                 # πŸ¦€ Rust data plane
β”‚   β”œβ”€β”€ src/
β”‚   β”‚   └── main.rs
β”‚   β”œβ”€β”€ Cargo.toml
β”‚   └── active_port.txt       # Auto-generated port file
β”‚
β”œβ”€β”€ java-controlplane/         # β˜• Spring Boot orchestration
β”‚   β”œβ”€β”€ src/
β”‚   β”‚   └── main/
β”‚   β”œβ”€β”€ pom.xml
β”‚   └── java_active_port.txt  # Auto-generated port file
β”‚
β”œβ”€β”€ react-dashboard/           # βš›οΈ React monitoring UI
β”‚   β”œβ”€β”€ src/
β”‚   β”‚   β”œβ”€β”€ components/
β”‚   β”‚   β”œβ”€β”€ App.tsx
β”‚   β”‚   └── main.tsx
β”‚   β”œβ”€β”€ package.json
β”‚   └── vite.config.ts
β”‚
β”œβ”€β”€ docker-compose.yml         # Container orchestration
β”œβ”€β”€ run_all.sh                 # Startup script
β”œβ”€β”€ .gitignore
β”œβ”€β”€ LICENSE
└── README.md

πŸ’» Usage

Accessing the Dashboard

Once all services are running, navigate to:

http://localhost:3000

The dashboard displays:

  • βœ… Service health status
  • πŸ”Œ Active ports for Rust and Java services
  • πŸ“Š Real-time system metrics
  • πŸ•’ Last update timestamp

API Endpoints

Rust Core (Dynamic Port - check active_port.txt)

GET /health          # Health check
GET /metrics         # Prometheus metrics
GET /active_port     # Current port info

Java Control Plane (Dynamic Port - check java_active_port.txt)

GET /api/health      # Health check
GET /api/state       # Current system state
POST /api/control    # Control operations

🐳 Deployment

Docker Compose

Deploy the entire stack with Docker:

version: '3.8'
services:
  rust-core:
    build: ./rust-core
    ports:
      - "8000:8000"
    networks:
      - cerebralake-net

  java-controlplane:
    build: ./java-controlplane
    ports:
      - "8080:8080"
    depends_on:
      - rust-core
    networks:
      - cerebralake-net

  react-dashboard:
    build: ./react-dashboard
    ports:
      - "3000:3000"
    depends_on:
      - java-controlplane
    networks:
      - cerebralake-net

networks:
  cerebralake-net:
    driver: bridge

Run with:

docker-compose up --build

πŸ—ΊοΈ Roadmap

βœ… Current Features

  • Dynamic port allocation and discovery
  • Real-time health monitoring
  • Cross-language REST communication
  • React-based visualization dashboard

🚧 Upcoming Features

  • WebSocket support for real-time updates
  • PostgreSQL integration for persistent storage
  • Kafka-based message queue for telemetry
  • Prometheus metrics exporter
  • Grafana dashboard templates
  • Kubernetes deployment manifests
  • Authentication and authorization layer
  • Circuit breaker patterns
  • Distributed tracing with OpenTelemetry

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.


πŸ“§ Contact

Mohit (Jitterx69)


Built by JitterX

A cross-language symphony of Rust πŸ¦€ + Java β˜• + React βš›οΈ

About

CerebraLake is an autonomous, self-optimising data lake engine built with Rust, Java, AWS, and BigQuery, designed to learn, predict, and optimise data workflows automatically.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published