Multi-Agent Orchestration for Compiler-Aware Semantic Evaluation
MASCoD-MLIRBench operates through a modular multi-agent architecture designed to separate mutation generation, execution validation, semantic reasoning, and benchmark evaluation into isolated execution stages.
Each agent performs a specialized task while communicating through structured intermediate representations and execution-derived metadata.
This separation enables deterministic evaluation, scalable benchmark generation, and controlled reasoning analysis across compiler intermediate representations.
Architecture Philosophy
The framework follows a strict execution-grounded evaluation design where:
- agents operate independently
- execution truth is isolated from reasoning agents
- semantic validation is compiler-driven
- inter-agent communication follows structured constraints
- reasoning quality is evaluated against deterministic execution behavior
The architecture minimizes information leakage between agents and ensures that semantic evaluation remains grounded in actual runtime behavior rather than textual similarity.
Multi-Agent Pipeline
The MASCoD-MLIRBench pipeline consists of five primary agents:
- Root Agent
- Mutation Agent
- Execution Sandbox Agent
- LLM Reasoning Agent
- Judge Agent
Root Agent
Responsibilities
The Root Agent coordinates the entire benchmark workflow and manages communication between all agents.
It acts as the orchestration layer responsible for:
- workflow scheduling
- task routing
- metadata management
- execution coordination
- benchmark state tracking
Inputs
- seed MLIR programs
- dialect configuration
- benchmark rules
- mutation configuration
Outputs
- structured agent tasks
- orchestration metadata
- benchmark execution states
Key Characteristics
- centralized orchestration
- deterministic workflow sequencing
- controlled inter-agent communication
- execution tracking and coordination
Mutation Agent
Responsibilities
The Mutation Agent generates controlled semantic mutations over MLIR programs while enforcing compiler-aware structural constraints.
The agent uses dialect-aware mutation strategies to produce semantically invariant, divergent, and failure-inducing transformations.
Mutation Levels
| Mutation Level | Title | Description |
| 1 | Syntactic Mutations | Structure-preserving textual or symbolic transformations |
| 2 | Parameter Mutations | Modifications to constants, dimensions, tensor parameters, or numerical values |
| 3 | Operator Mutations | Replacement or modification of semantic operators |
| 4 | Structural Mutations | Changes to loop structures, control flow, tensor layouts, or computation ordering |
| 5 | Failure-Inducing Mutations | Compiler-valid or near-valid transformations designed to trigger semantic or execution failures |
Key Characteristics
- compiler-aware mutation logic
- deterministic transformation constraints
- controlled semantic divergence generation
- MLIR dialect specialization
Execution Sandbox Agent
Responsibilities
The Execution Sandbox Agent establishes execution-grounded truth for all generated program mutations.
Programs are compiled, validated, and executed using compiler infrastructures and runtime execution backends.
Validation Pipeline
The framework uses:
- MLIR
- Torch-MLIR
- LLVM
- IREE
to validate legality and runtime behavior.
Outputs Generated
- execution outputs
- runtime traces
- compiler errors
- deterministic semantic labels
Key Characteristics
- isolated execution environment
- deterministic execution analysis
- execution-grounded validation
- compiler-driven semantic verification
LLM Reasoning Agent
Responsibilities
The LLM Reasoning Agent evaluates semantic differences between original and mutated MLIR programs without access to execution outputs.
The agent performs reasoning exclusively from program structure and mutation context.
Constraints
The agent does not receive:
- execution outputs
- runtime traces
- compiler validation results
- judge labels
This prevents execution leakage into reasoning evaluation.
Key Characteristics
- execution-blind reasoning
- semantic interpretation focus
- compiler IR analysis
- explanation-driven evaluation
Judge Agent
Responsibilities
The Judge Agent evaluates whether reasoning outputs produced by the LLM Reasoning Agent align with actual execution-grounded behavior.
The agent performs structured reasoning assessment using deterministic labels generated from the Execution Sandbox Agent.
Evaluation Dimensions
- semantic correctness
- reasoning faithfulness
- execution alignment
- hallucination detection
- divergence accuracy
Key Characteristics
- execution-grounded scoring
- deterministic evaluation
- structured reasoning assessment
- semantic consistency analysis
Inter-Agent Communication
Agents communicate through structured intermediate representations and controlled metadata exchange.
The framework follows a strict need-to-know communication model where agents receive only the information required for their assigned tasks.
This design prevents unintended information leakage and preserves the integrity of execution-grounded evaluation.
Communication Properties
- structured JSON-based communication
- isolated execution stages
- restricted metadata exposure
- deterministic workflow transitions
- controlled context propagation