TrustyAI SDK

A Python toolkit for testing machine learning models. You can run tests on your computer or scale them up using Kubernetes.

✨ Key Features

• 🔄 Multiple Testing Tools: Works with LM Evaluation Harness, RAGAS, and other testing tools
• ☸️ Works with Kubernetes: Scale up tests on Kubernetes with TrustyAI Operator
• 🖥️ Run Anywhere: Test models on your computer or spread across cluster nodes
• 🛡️ Team Ready: Built-in checks, monitoring, and works with OpenDataHub
• 🎯 Easy to Use: Command line tools and Python code for different workflows

🚀 Quick Start

Local Testing

trustyai eval execute \
  --provider lm-eval-harness \
  --execution-mode local \
  --model "microsoft/DialoGPT-medium" \
  --tasks "hellaswag,arc_easy" \
  --limit 10

Kubernetes Testing

trustyai eval execute \
  --provider lm-eval-harness \
  --execution-mode kubernetes \
  --model "microsoft/DialoGPT-medium" \
  --tasks "hellaswag,arc_easy" \
  --namespace trustyai-eval \
  --cpu 4 \
  --memory 8Gi \
  --limit 50

📚 Documentation

📖 Complete Documentation

Quick links:

• Getting Started Guide
• Kubernetes Integration
• CLI Reference
• Examples

Installation

Standard Installation

Install the package with core functionality and CLI:

pip install .

After installation, you can use both the Python API and CLI:

trustyai --help
trustyai info
trustyai model list
trustyai eval list-providers

Full Installation

To install everything including evaluation support:

pip install .[all]

This includes all core, CLI, and evaluation dependencies.

Additional Optional Dependencies

Evaluation Support

For model evaluation capabilities:

pip install .[eval]

Development Dependencies

For development, testing, and linting:

pip install .[dev]

Usage

Python API

import numpy as np
from trustyai.core.model import TrustyModel
from trustyai.core.providers import ProviderRegistry

# Create a trusty model
model = TrustyModel(name="MyModel")

# Get explanations
X = np.random.rand(10, 5)
explanations = model.explain(X)
print(explanations)

Command Line Interface

The CLI is available by default after installation:

# Display help
trustyai --help

# Show version information
trustyai --version

# Show general information
trustyai info

# List available models
trustyai model list

# List evaluation providers
trustyai eval list-providers

# List available validators
trustyai validators list

# Run a validator
trustyai validators run python-version

Documentation

• API Documentation - Comprehensive API reference including validators

Development

This project uses:

• pytest for testing
• Ruff for linting and formatting
• mypy for type checking

# Install development dependencies
pip install .[dev]

# Run tests
make test

# Run linting
make lint

# Format code
make format

License

Apache License 2.0