Throwly AI Accuracy Team: Sprint Reports
Senior Design Project Progress Summary
Team Members: Arslan Khan, Lucas Martins Sorge, Lucas Nerone Rillo, and Tanish Visanagiri
Project context. The team worked on the AI pricing portion of Throwly, an AI-assisted marketplace focused on producing fair price estimates for second-hand goods and services. Across the reported sprints, the work centered on improving pricing accuracy, reducing latency, expanding category coverage, cleaning and validating data, integrating external marketplace context, strengthening pipeline security and reliability, and documenting findings in a research-style format.
Sprint 1 Report
Milestone source: Lightning Talk 1
Overview. Sprint 1 established the project direction, clarified the team’s role within Throwly, and translated the product vision into concrete engineering and research requirements.
Work completed
- Defined the main problem as unreliable, slow, and difficult pricing for marketplace items, along with the broader platform friction caused by low-trust marketplace interactions.
- Framed the team’s scope around AI accuracy, latency, validation, and platform improvements for Throwly’s pricing system.
- Set a target of accurate pricing for common household and marketplace items with an end-to-end AI response time under three seconds.
- Identified the need for a validation dataset and a Python evaluation script to measure pricing quality quantitatively.
- Outlined additional goals including safer prompting, service pricing, serverless deployment knowledge, website and MCP server improvements, and a comprehensive report on findings.
Key results
- The sprint produced a clear requirements baseline that aligned technical work with business needs and academic deliverables.
- The team established the major research questions to investigate: how to improve model accuracy, how to reduce latency, and how to validate performance quantitatively over time.
Challenges
- Learning the Throwly codebase and previous work without duplicating effort from other teams.
- Understanding agentic AI concepts and the existing marketplace workflow well enough to make meaningful improvements.
- Balancing immediate implementation tasks with the longer-term research and reporting goals of the project.
Next steps. Begin building the data, evaluation, and integration foundations needed to benchmark the current AI pipeline and identify improvement opportunities.
Sprint 2 Report
Milestone source: Demo 1
Overview. Sprint 2 focused on building the project foundation: creating data pipelines, standing up evaluation workflows, and identifying the first concrete weaknesses in the production pricing system.
Work completed
- Built an eBay scraper and supporting data pipeline to collect external marketplace listings for evaluation and comparison.
- Established an evaluation framework that fed a validation dataset through the Throwly pricing model and recorded metrics such as accuracy and latency.
- Created a baseline validation dataset for future performance tracking and category-level testing.
- Set up CI/CD synchronization work and expanded the dataset effort with Facebook Marketplace data and additional category testing.
- Reviewed the existing category taxonomy and identified issues such as overlapping categories, missing categories, and overly broad mappings.
Key results
- Initial tests achieved a 100% pass rate for item test execution, produced no image URL errors, and successfully ran category-based evaluations.
- The evaluation exposed three major weaknesses in the current system: hit rate was low, latency was too high, and predicted prices often differed substantially from eBay market prices.
- The team also identified missing and overlapping categories, showing that taxonomy quality was affecting downstream pricing behavior.
Challenges
- Cost limits in testing alternative AI models.
- Roadblocks imposed by the currently deployed AI setup.
- Questions around legality and practical constraints of scraping-based data collection.
- Need to increase dataset size and diversity while keeping the evaluation pipeline manageable.
Next steps. Optimize the AI pipeline by benchmarking models and prompts, improving category definitions, reducing latency, and expanding the validation dataset for more reliable measurements.
Sprint 3 Report
Milestone source: Lightning Talk 2
Overview. Sprint 3 divided work across the team and translated the project foundation into specialized implementation tracks covering data collection, validation, infrastructure, prompting, and pipeline hardening.
Work completed
- Arslan scraped sales data, organized and formatted collected listings, updated older AI-related libraries, refactored code to match newer implementations, added token-count email monitoring, and began researching replacement AI models.
- Lucas Sorge built a benchmarking framework for Throwly pricing models, loaded listings from the dataset, ran them through the Throwly Pricing API, captured predictions, and created tools to process evaluation results and compute latency and accuracy metrics.
- Lucas Rillo created a CI/CD pipeline to push changes to the client repository, built a tool to query relevant products from the eBay API, improved image processing, compressed large images, removed redundant encoding steps, added prompt injection protections, and identified pipeline bottlenecks.
- Tanish tested eBay data against the Throwly model by category, compared eBay market prices to LLM prices, reviewed categories, cleaned the scraped dataset, and tested a newer prompt against the current prompt using accuracy metrics.
Key results
- The team moved from general planning to measurable experimentation and implementation.
- A reusable evaluation workflow was now in place, supported by cleaned data, automated metrics, and tools for comparing model outputs against real-market references.
- The sprint also produced early system hardening improvements, especially around image handling, prompt-safety, and efficiency.
Challenges
- Coordinating multiple technical streams while keeping the work integrated and aligned.
- Continuing to improve prompt quality and model selection without losing focus on latency and engineering constraints.
- Turning early experiments into user-facing or analytics-facing outputs that would be useful for the client long term.
Next steps. Test prompt and architecture variations more systematically, build analytics and dashboard views for pricing results, and continue improving both dataset quality and the AI pipeline itself.
Sprint 4 Report
Milestone source: Demo 2
Overview. Sprint 4 was the project’s strongest optimization phase, combining software engineering improvements with structured experiments on prompts, architectures, and retrieval strategies.
Work completed
- Strengthened project management workflows through branch handling, sprint board organization, CI/CD fixes, and pull request review.
- Implemented software improvements in four major areas: security, reliability, speed, and cost efficiency.
- Added defence-in-depth protections against prompt injection, standardized image processing and error handling, improved concurrency and async execution, and added targeted logging to isolate bottlenecks.
- Reduced API costs and generation time through prompt caching, payload reduction, and tighter caps on context size.
- Ran structured experiments on six prompt variants, four architecture types, and twenty-four total configurations across 120 test cases.
- Evaluated prompt-context strategies including a production baseline, price bands, embedding retrieval, and live eBay Browse API context.
Key results
- The structured prompt performed best overall among prompt variants, with roughly 22% better accuracy than baseline.
- The metadata-augmented architecture outperformed the baseline architecture with about 14% better accuracy, though with a modest latency increase.
- The best overall combination was metadata architecture plus structured prompt, achieving a mean absolute error of about $90 with latency around 9 seconds.
- Embedding retrieval produced the best raw accuracy among context strategies, while live eBay API retrieval produced the best hit rate at 30.5%.
- The sprint confirmed an important design conclusion: no single change solved the problem by itself; the best outcomes came from combining stronger prompting, better system design, and better external data.
Challenges
- Managing tradeoffs between speed and accuracy, since faster prompt styles tended to reduce pricing quality.
- Dealing with stale retrieval datasets and extraction failures that limited the usefulness of context-based methods.
- Improving accuracy substantially while still remaining mindful of the original latency targets.
Next steps. Use the experimental findings to guide final recommendations, continue refining retrieval quality, and package the results into a clear final analysis for the client and the research report.
Sprint 5 Report
Milestone source: Demo 3
Overview. Sprint 5 consolidated the team’s findings into a final analysis and shifted the project toward synthesis, communication, and future-ready feature direction.
Work completed
- Summarized the final testing campaign across prompt, architecture, and model variations.
- Prepared a visual results report showing item-by-item comparisons between AI prices and real eBay market prices.
- Produced a refined website design mock-up that could guide future improvements to the Throwly main site.
- Connected testing outputs to website presentation by showing how analytics and model results could be surfaced to users or internal stakeholders.
- Implemented a prototype for service pricing for marketplace items.
- Outlined additional forward-looking features such as AI try-on for marketplace items.
- Carried the validated findings into the final report.
Key results
- The final analysis showed a clear improvement in the AI’s reasoning and price-prediction performance compared with earlier iterations.
- Structured prompts improved accuracy, metadata improved system design, and real market data delivered the biggest performance boost.
- Among model options, Sonnet offered the best balance between speed and accuracy, while Opus was the most accurate but slower.
- The team’s overall conclusion was that pricing quality improves most when good prompting, strong architecture, and real-world marketplace data are combined.
Challenges
- Communicating experimental findings clearly enough that both technical and non-technical stakeholders could act on them.
- Converting prototype-level improvements into recommendations that fit the client’s product roadmap and engineering constraints.
- Balancing final analysis with exploratory feature work such as service pricing and virtual try-on.