MASCoD-MLIRBench
Multi-Agent Semantic Code Divergence Benchmark for Compiler IR Reasoning
Execution-grounded benchmark framework for evaluating whether large language models truly understand MLIR compiler semantics instead of relying on surface-level code patterns.
Quick Links
Team Members
Overview
MASCoD-MLIRBench is a research-driven benchmarking framework designed to evaluate semantic reasoning capabilities of large language models over compiler intermediate representations.
Unlike traditional AI code benchmarks that primarily evaluate source-code generation or compilation success, MASCoD-MLIRBench focuses on whether models can correctly reason about execution behavior, semantic divergence, compiler legality, and IR-level transformations.
The framework combines compiler infrastructures, controlled semantic mutation generation, execution-grounded validation, and multi-agent evaluation workflows to create a scalable benchmark ecosystem for compiler-aware AI systems.
Motivation
Modern large language models demonstrate strong performance on code generation tasks, yet their ability to understand program semantics remains unclear.
Most existing benchmarks evaluate:
- syntactic correctness
- generated outputs
- repository-level reasoning
- compilation success
However, these evaluations often fail to measure whether model explanations align with actual execution behavior.
Even small semantic mutations such as operator substitutions, boundary-condition changes, or tensor shape modifications can significantly alter runtime behavior while remaining syntactically valid.
MASCoD-MLIRBench addresses this gap through execution-grounded semantic evaluation over MLIR compiler intermediate representations.
System Architecture
MASCoD-MLIRBench operates as a modular multi-agent benchmark pipeline for mutation generation, execution-grounded validation, semantic reasoning evaluation, and benchmark scoring.
Core Components
- Root Agent
- Mutation Agent
- Execution Sandbox Agent
- LLM Reasoning Agent
- Judge Agent
→ View Detailed Multi-Agent Framework
Benchmark Workflow
The benchmark pipeline follows five major stages:
1. Input Program Processing
Programs are represented using MLIR compiler intermediate representations while preserving structural and semantic information across abstraction levels.
2. Controlled Mutation Generation
The framework generates structured semantic mutations across multiple transformation categories while maintaining compiler-aware constraints.
3. Execution-Grounded Validation
Programs are compiled and executed using MLIR validation pipelines and IREE execution backends to establish deterministic behavioral ground truth.
4. Semantic Reasoning Evaluation
Large language models analyze mutated program pairs without access to execution outputs or runtime behavior.
5. Judge-Based Benchmark Scoring
Model explanations are evaluated against execution-derived labels to measure semantic correctness, reasoning faithfulness, and hallucination behavior.
Execution-Grounded Evaluation
MASCoD-MLIRBench introduces execution-grounded evaluation to ensure that reasoning quality is measured against actual runtime behavior rather than surface-level textual similarity.
The framework validates semantic reasoning through:
- deterministic program execution
- compiler legality verification
- divergence-aware mutation analysis
- execution-derived ground truth generation
- structured reasoning evaluation
This enables reliable benchmarking of compiler-aware reasoning capabilities in modern AI systems.
Key Research Contributions
- Multi-agent benchmark framework for MLIR semantic reasoning
- Execution-grounded evaluation pipeline for compiler IR analysis
- Controlled semantic mutation generation across multiple mutation levels
- Benchmarking methodology for semantic divergence reasoning
- Structured judge-based evaluation for reasoning faithfulness
- Scalable framework for compiler-aware AI system evaluation