Project Artifacts

Requirements

Functional Requirements

  • Accept pre-trained image classification models (e.g., ResNet50, EfficientNet, Vision Transformer (ViT))
  • Support dataset input (local folders / standard datasets like CIFAR-10, COCO)
  • Allow user to specify:
    • Target label (e.g., “dog”)
    • Number of test samples
  • Execute multiple testing strategies:
    • Random Testing
    • Stress Testing
    • AI-Guided Testing
    • Feature-Based Testing
  • Generate:
    • Metrics (Accuracy, Precision, Recall, F1-score)
    • Failure summaries
    • Visual comparisons (before vs after transformations)

Non-Functional Requirements

  • Scalability (handle large datasets)
  • Extensibility (add new testing strategies)
  • Interpretability (clear explanations via LLM)
  • Usability (simple UI + clear reports)

Prototype Design

System Architecture

  • Frontend (React + Vite)
    • User input (model, dataset, label)
    • Visualization (graphs, results, images)
  • Backend (Node.js + Express)
    • API endpoints:
      • /model-report
      • /random-test
      • /stress-test
      • /feature-test
  • ML Execution Layer (Python)
    • Model inference (PyTorch / TensorFlow)
    • Image transformations
  • LLM Integration
    • Summary generation
    • Failure analysis
    • Feature extraction

Data Flow

  1. User inputs model + dataset
  2. Backend initializes testing pipeline
  3. Python layer executes tests
  4. Results → sent to LLM
  5. LLM generates insights
  6. UI displays final report

Completed Work

  • Implemented:
    • Random Testing pipeline
    • Basic Stress Testing (background, occlusion)
    • Model summary using LLM
  • Integrated:
    • Pre-trained models (ResNet, VGG)

Observations

  • Accuracy alone is insufficient
  • Models fail under:
    • Occlusion
    • Background changes
  • Confidence scores do not reflect robustness

Challenges

  • Mapping dataset labels to model classes
  • Generating meaningful failure explanations
  • Handling API limits (LLM cost)

Final Design (Detailed)

Core Components

1. Model & Dataset Analyzer

  • Extract:
    • Architecture details
    • Training assumptions
  • LLM generates:
    • Model summary
    • Suggested testing strategy

2. Random Testing Module

  • Input: label + dataset
  • Process:
    • Sample images randomly
    • Run inference
  • Output:
    • Pass/Fail counts
    • Accuracy metrics

3. Stress Testing Module

  • Apply transformations:
    • Background change
    • Environment shift
    • Occlusion
    • Multi-object addition
  • Compare:
    • Before prediction
    • After prediction

4. AI-Guided Testing Module

  • LLM generates:
    • Edge cases
    • Rare scenarios
  • Examples:
    • Dog in unusual pose
    • Low-light conditions

5. Feature-Based Testing Module

  • LLM extracts:
    • Visual features (e.g., “fur texture”, “ear shape”)
  • Testing:
    • Evaluate model on feature combinations

6. Failure Analysis Engine

  • Input:
    • Failed test cases
  • LLM identifies:
    • Patterns
    • Root causes

Examples:

  • “Model fails when object is partially occluded”
  • “Confusion between visually similar classes”

7. Reporting Module

  • Generates:
    • Summary report
    • Graphs (performance trends)
    • Recommendations

 

A. Introduction for reporting:

Problem: Lack of robust testing frameworks

Motivation: Hidden failures in ML models

 

B. Related Work:

Uncertainty-based testing (FAST)

Traditional test selection methods

Limitations: No semantic understanding

No interpretability

 

C. Proposed Framework:

Unified testing system

Integration of:LLM

Multiple testing strategies

 

D. Methodology:

Testing Types

Random Testing → baseline

Stress Testing → robustness

Feature-Based Testing → interpretability

AI-Guided Testing → exploration

 

E. Experimental Setup:

Models: ResNet50, EfficientNet, ViT

Datasets:CIFAR-10, COCO (subset)

 

F. Evaluation Metrics:

Accuracy, Precision / Recall, F1-score

 

Failure Rate:

APFD (test prioritization)

 

G. Results & Analysis:

Stress testing reveals hidden failures

Feature-based testing improves interpretability

AI-guided testing finds edge cases

 

H. Contributions:

Unified AI-assisted testing framework

Integration of LLM for explanation

Test generation

Feature-level testing approach