[Paper Review] BazaarBench: Delegation Safety in Decentralized C2C Marketplaces Run by LLM Agents

• Metadata
• TL;DR
• Introduction
• Key Contributions
• Related Work / Background
  • Negotiation outcomes and inventory constraints
  • State-based evaluation in an interactive market
• Method / Architecture
  • 1. Personas, inventory, and public listings
  • 2. Two-hour ticks and seller-managed commitments
  • 3. Six failure types and distinct counting objects
  • 4. Separate S1 counts and failure-specific S5 outcomes
  • 5. Record checks and rubric-based judgments
  • 6. Post-hoc inspection assumptions
• Evaluation Objectives / Metrics
  • Completion rates and role-specific denominators
  • Weighted differences across base markets
  • Simulated earnings and acquisition costs
• Evaluation Data and Pipeline
  • Three base markets and 45 continuations
  • Model settings and evaluation costs
• Experimental Results
  • Persistent ordinary-condition failures
  • Deadline pressure and overlapping promises
  • Adversarial instructions and item violations
  • Completion and earnings before and after audit
• Analysis / Ablation
  • Failure profiles and available explanations
  • Transaction cases and audit membership
  • Weighted effects and multiple comparisons
• Limitations and Critical Assessment
  • Physical exchange and counterfactual inspection
  • Market diversity and rollout variability
  • Judge visibility and measurement coverage
• Takeaways
• Installation and Usage
• References
• Further Reading

Metadata

TL;DR

• BazaarBench evaluates delegated trading in persistent C2C markets, checking agents’ actions against ownership, item condition, and existing commitments. Three 100-agent base markets support 45 continuations in which 20 agents change model and instruction condition.
• All five tested models overcommit inventory under ordinary instructions. Deadline pressure increases F3 attempts from 71 to 107 and F3 objects reaching S5 from 58 to 95.
• Under adversarial instructions, completed transactions involving unavailable or overstated items rise from 15.4% to 33.4% of tested sellers’ committed transactions, reaching 55.5% for GPT-5.4. These are simulator measurements, not real-world fraud rates.
• An idealized inspection audit reduces the completion rate to 70.4% under ordinary instructions and 55.8% under adversarial instructions. It rescores saved transactions; it does not rerun agents with better evidence or observe buyers rejecting goods.

Introduction

An agent selling a used tablet can negotiate well in two conversations and still fail its user. If it promises the same tablet to both buyers, each conversation may look reasonable while the commitments are jointly impossible. Successful tool calls and completed transaction records do not establish that the seller had two tablets to deliver.

Delegation safety therefore requires more than detecting explicitly malicious requests. A buyer might confirm too early, a seller might expose personal details, and an ordinary deadline might encourage shortcuts. Resistance to adversarial instructions and reliable inventory management are related but distinct requirements.

Wang et al.’s BazaarBench studies these failures in a persistent simulated marketplace. It records facts separately from agents’ claims and connects them to transactions and subsequent responses. This review follows arXiv v1, including the appendices, with particular attention to the distinction between platform completion, failure-linked completion, and completion retained under an assumed inspection rule.

Key Contributions

• Cross-transaction state: ownership, initial condition, resale history, and overlapping commitments provide evidence that isolated conversations cannot supply.
• Failure types and stages: six failure types are tracked from recorded consideration to attempted actions and type-specific outcomes.
• Matched continuation conditions: ordinary, deadline-pressure, and adversarial instructions start from copies of the same saved market state.
• Released evaluation infrastructure: the simulator, market states, interaction records, judgments, and analysis code support further evaluation.

Related Work / Background

Negotiation outcomes and inventory constraints

A2A-NT examines agent negotiations and violations of buyers’ budgets or sellers’ cost constraints. Magentic Marketplace studies search, welfare, and manipulation among consumer and service agents. BazaarBench adds persistent item-level facts. A favorable price is insufficient if the item does not exist or was already used in another transaction.

Multiple conversations are not automatically unsafe: the seller could own several units or cancel an earlier commitment. Evaluation must connect promises to particular units over time. This is why a language-based judgment of politeness or apparent plausibility cannot replace the inventory record.

State-based evaluation in an interactive market

τ-bench evaluates database outcomes after tool-mediated conversations; Vending-Bench studies coherent business operation over long horizons. BazaarBench similarly looks beyond final responses, while making trading partners themselves LLM agents. Their reactions help determine whether a failure progresses.

The decentralized setting concerns direct consumer-to-consumer trading. The simulator is not connected to live accounts, payments, or users. Its results concern the specified tools, observations, instructions, and accounting rules, rather than the safety of a deployed payment system.

Method / Architecture

1. Personas, inventory, and public listings

Each agent buys and sells. Its persona includes inventory, a desired product category, a purchase ceiling, a selling floor, activity probability, and prior history. Starting goods and prices come from an eBay sample; a language model supplies profession, communication style, strategies, and synthetic memories. Generated narrative does not define ownership.

A listing contains the seller’s claims. Evaluation separately preserves the item’s condition when it first entered the market, including after resale. All initial units in the reported markets entered in the good band, so advertising them as like new or brand new meets the F1 condition. The check covers condition bands, not every misleading phrase in a description.

2. Two-hour ticks and seller-managed commitments

One tick represents two simulated hours. Selected agents receive their persona, recent history, and relevant listings, messages, offers, and exchanges. They decide from the tick’s initial state, after which actions are applied in a fixed agent order. One model call can generate several platform calls.

The runs exercise 31 action types. A conversation becomes a committed transaction when an offer is accepted or an exchange scheduled. Platform completion requires both parties’ confirmation, which does not independently establish physical delivery.

Accepting an offer does not automatically reserve inventory. Sellers must reconcile competing commitments themselves. Existing agreements are visible through accepted transactions and scheduled meetings; what is missing is an explicit per-unit promised-or-sold field. The setting therefore tests integration of available information as well as the consequences of permissive platform rules.

3. Six failure types and distinct counting objects

Repeated actions of one type on one object count once. Listing a nonexistent item for multiple buyers does not additionally establish F3, which requires an owned unit. F2 is evaluated at listing creation: an agent might acquire the item later. Ownership at listing and availability at completion answer different questions.

F5 excludes merely discussing a supported payment method. Photos are JSON records with descriptions and sensitive-detail labels, not images. The privacy option records a protective choice without modifying photo content, so the evaluation concerns that choice rather than redaction quality.

4. Separate S1 counts and failure-specific S5 outcomes

S1 records consideration in available reasoning summaries or decision notes. Recognizing a risk and refusing it does not qualify. S2 means an attempted platform action, including blocked calls; S3 means accepted storage or delivery; S4 records a response; S5 requires the outcome defined for that failure. S2–S5 counts include objects reaching at least that stage.

S1 is separate. An agent can consider something without attempting it, or act without mentioning it in the available explanation. Missing reasoning is unknown, not safe. The five numbers are not a single shrinking funnel, and S1 is compared within a model across conditions rather than treated as complete access to intention.

S5 need not require a completed sale. Overcommitment can reach S5 when a linked transaction ends in cancellation, stopped communication, or a missed meeting. An unprotected simulated photo containing a sensitive detail reaches S4 and S5 upon delivery, without a reply. The outcome follows the failure’s mechanism.

5. Record checks and rubric-based judgments

Inventory and event checks determine F1–F3 from S3 onward. A GPT-5 judge evaluates F4, F5 messages, and F6 using relevant text, actions, records, and later responses. Photo labels use protection choices, delivery, and sensitive details. Script and judge results are deduplicated.

Most S1 judgments use GPT-5 on reasoning summaries; DeepSeek-V4-Pro decision notes use GPT-5.5 with low reasoning effort. Human evaluation agreed with 188 of 200 sampled judgments, or 94%. This aggregate agreement is not a per-type accuracy guarantee.

6. Post-hoc inspection assumptions

Meetup buyers can inspect; shipped transactions have no inspection step. The actual tool returns a listing’s stored quality. Of 16,396 inspections with stored values, 16,357 returned values inside the listing’s claimed band. Inspection does not independently verify the current inventory unit.

The idealized audit is different. For a completed transaction with an unavailable or overstated item, a prior buyer inspection leads to assumed rejection: accurate inspection would reveal the problem and the buyer would refuse. The audit removes completion and seller earnings while preserving the committed denominator. Without inspection, it retains completion.

It changes scores, not actions or downstream inventories. In 1,407 of 1,422 audit-rejected transactions, the buyer actually saw a value within the advertised band. Audit rejection therefore cannot be described as an observed response to accurate evidence or as the measured effect of deploying an inspection defense.

Evaluation Objectives / Metrics

Completion rates and role-specific denominators

No model is trained. Using explanatory notation, let $D$ contain eligible committed transactions, $C$ platform-completed transactions, and $A$ completions retained by the audit:

Continuations include transactions involving at least one tested agent, including commitments still ongoing at the start. Previously completed or cancelled transactions and platform-seeded listings are excluded. A transaction with two tested parties counts once.

Table 5 divides problematic platform completions by committed transactions with a tested seller. Figure 3c instead divides retained purchases linked to seller F1–F3 by retained completions with a tested buyer. The latter measures exposure to a seller’s failure, not wrongdoing by the buyer. A linked failure must concern the transaction and reach at least S3 by completion.

Weighted differences across base markets

Equation 1 combines within-market differences for model $m$, base market $b$, and condition $L$:

Here $n$ is the relevant denominator count. The all-model result combines 15 model-market combinations. Consequently, subtracting pooled percentages, 33.4 minus 15.4, gives 18.0 percentage points, while the reported weighted difference is +16.6 points. Failure-object count differences are summed instead.

Simulated earnings and acquisition costs

Owned-item profit subtracts acquisition cost from sale price. Never-held items contribute their full recorded sale price as revenue. The main calculation excludes repeated-unit sales, owned items with missing costs, and recorded costs above ten times the sale price, among other specified exclusions. Agents with no included sales remain in the mean.

An F2 listing is not necessarily a never-held sale because inventory may arrive later. Audit-rejected transactions contribute zero under the same inclusion rules. These amounts are simulated accounting, not verified payments or human losses.

Evaluation Data and Pipeline

Three base markets and 45 continuations

Market construction uses 20,367 eBay items priced between 1 and 5,000 dollars, grouped by brand, category, and price range. Buyers’ ceilings are sampled at 58–88% of a goal item’s price, and selling floors at 86–97% of asking prices. Identities and histories are synthetic.

L1 adds no instruction. L2 requires three seller completions within five days and two buyer completions within three days, plus a 400-dollar obligation and a purchase-price cap. Honesty and inventory constraints remain. L3 appends adversarial instructions while retaining the original constraints, creating an instruction conflict.

L1–L3 use identical platform checks within a base market. L0 blocks title-based inventory mismatches, whereas continuations log them without blocking. Base markets also differ in inventory, persona generation, history, and activity rates. They are not identical economies differing only in model; the matched comparison is between continuations from the same saved state.

Model settings and evaluation costs

GPT-5.5 and GPT-5.4 use high reasoning effort. GPT-5.4-mini changes from medium in its base market to high in continuations; GPT-OSS-120B and DeepSeek-V4-Pro use provider defaults. Continuation output limits are 12,288 tokens for GPT-5.4 and 8,192 for the others, with early-run exceptions documented in Appendix F.

The 357,608 agent model calls comprise 77,636 base-market calls, 56,769 tested-agent continuation calls, and 223,203 background calls. Evaluating another model requires nine continuations, each with approximately 4,500–5,200 background-agent calls alone. The paper’s estimated 6,487-dollar GPT-5 judging cost excludes agent calls and is not an invoice. Recomputing statistics from saved records is much cheaper than generating new market behavior.

Experimental Results

Persistent ordinary-condition failures

New S5 objects continue accumulating through the final ten base-market days. Final counts are 306 for GPT-5.5, 490 for DeepSeek-V4-Pro, and 229 for GPT-5.4-mini. Their largest S5 categories differ: F3 with 141 objects, F4 with 321, and F5 with 76, respectively.

L1 retains 1,025 of 1,457 committed transactions after audit, or 70.4%. Retained completions linked to either party’s F1–F6 account for 39.7% of commitments; those linked to a tested agent’s own failure account for 26.7%. Neither percentage uses completed transactions as its denominator.

Every tested model has F3 objects reaching S5, from four for GPT-OSS-120B to 22 for GPT-5.4-mini. No F6 object reaches S4 or S5 anywhere in the reported runs. Unsupported reputation claims existed, but evidence of the recipient acting on them did not.

Deadline pressure and overlapping promises

L2 increases commitments from 1,457 to 1,921. Agents meeting both targets rise from 20 to 37 out of 300 evaluated agent instances, approximately 7% to 12%. F3 attempts rise from 71 to 107, and S5 objects from 58 to 95; all five models make more attempts.

Among retained completions with a tested buyer, the share linked to seller F1–F3 rises from 13.5% to 19.3%. Of 170 linked L2 transactions, 161 are linked only to F3. Overcommitment dominates this result. Its Holm-adjusted p-value is 0.060, however, so the direction should not be reported as significant at the adjusted 5% threshold.

Adversarial instructions and item violations

Problematic platform completions rise from 141/916 to 370/1,109 of tested sellers’ commitments: 15.4% to 33.4%. This pre-audit metric covers item availability and condition, not every F1–F6 failure.

Model-specific L1→L3 rates are GPT-5.5 13.1%→16.2%, GPT-5.4 13.9%→55.5%, GPT-5.4-mini 15.2%→16.3%, GPT-OSS-120B 17.5%→31.1%, and DeepSeek-V4-Pro 19.7%→41.4%. GPT-5.4’s F1 attempts rise from two to 677, with 107 reaching S5 under L3.

GPT-5.5 makes fewer F1/F2 attempts under L3 but still reaches S5 for F3, F4, and F5. GPT-5.4-mini makes 22 F2 attempts under both conditions. A small change can mean persistence of an existing problem, rather than safety. Baseline failure levels and sensitivity to adversarial instructions need separate reporting.

Completion and earnings before and after audit

Platform completion remains approximately 86% under L1 and 83% under L3. After audit, the rates are 70.4% and 55.8%. Of 370 problematic L3 completions with tested sellers, 356 are rejected and 14 retained, all shipped transactions in the DeepSeek-V4-Pro base market. Eleven involve overstated condition and two never-held items; one involves a previously used unit without overstatement. Only the first 13 enter Figure 3b.

Mean pre-audit weekly earnings rise from 20.19 to 32.80 dollars per tested agent. The reported difference, computed before rounding, is 12.60 dollars with a 95% interval of 3.78–22.76. Owned-item profit rises from 19.05 to 22.63 dollars, so most of the total increase is associated with never-held sales.

After audit, means are 17.65 and 18.18 dollars, a 0.53-dollar difference with an interval of −7.59–8.34. Removing the ten-times-price cost threshold makes both pre- and post-audit earnings differences indistinguishable from zero. These findings depend on cost records and accounting choices; they do not establish that misconduct reliably pays in actual markets.

Analysis / Ablation

Failure profiles and available explanations

The radar plots use log10(1+n) radii with a shared scale. Their area is not an aggregate safety score. F1/F2 expand under L3 for GPT-5.4, GPT-OSS-120B, and DeepSeek-V4-Pro. The separate S1 outline need not enclose the action-stage polygons.

Recorded consideration precedes 649/746, 127/156, and 334/358 L3 F1/F2 attempts for those three models. GPT-5.4-mini’s 22 F2 attempts in each condition have explanations that do not discuss ownership. This supports examining explanations alongside actions, but does not establish the presence or absence of unobserved intentions.

Transaction cases and audit membership

The L1 GPT-5.5 seller recognizes competing appointments for its single tablet yet proceeds with both transactions. The L2 GPT-5.4 buyer continues negotiating for a phone promised elsewhere. The L3 GPT-OSS-120B seller lists a vacuum it never held and accepts an offer.

The last column gives audit scores, not observed human responses. In particular, the L2 example is rejected by the audit, so it is not an example from the retained purchases underlying the 19.3% buyer metric. Qualitative cases and quantitative populations must remain aligned.

Weighted effects and multiple comparisons

Circles show L2−L1, squares L3−L1, and horizontal lines agent-bootstrap 95% intervals. Panels differ in numerator and sometimes denominator. They cannot all be described as changes in a single fraud rate.

Eight comparisons were selected after the runs. After Holm correction, the L3 audited-completion decrease and the increase in retained overstated/never-held sales each have p=0.008. The L2 buyer comparison has p=0.060. The narrower F1–F3-or-photo test is not the overall F1–F6 measure.

These are condition contrasts and sensitivity analyses rather than a conventional component-removal ablation. Excluding transactions linked only to F3 from the seller-F1–F4 buyer analysis leaves +0.8 percentage points, with an interval of −2.1–4.1. This identifies overcommitment’s role in the observed association; it does not measure the benefit of adding inventory reservations.

Limitations and Critical Assessment

Physical exchange and counterfactual inspection

The simulator does not observe delivery, human acceptance, refunds, or disputes. The audit assumes accurate inspection and rejection without updating subsequent interactions. A useful next experiment would expose agents to corrected evidence and rerun the market. That is a reviewer’s proposed test, not a defense result demonstrated here.

Market diversity and rollout variability

There are three distinct base markets and one seven-day continuation per condition cell. The 2,000-draw bootstrap resamples tested IDs within saved runs, not newly generated markets or repeated continuations. Repeating isolated decisions changed selected actions in 4–8% of cases, but this does not measure how those differences compound through a full interactive rollout.

Judge visibility and measurement coverage

Run identifiers reveal model and condition to the judge; some L3 reasoning repeats adversarial labels. No condition-blind reevaluation is reported. The 200-case human check does not eliminate possible condition-dependent judgment bias.

Coverage is also bounded by the rubric. F1 checks condition bands rather than all prose, photos are textual simulations, and no L3 tactic specifically targets F6. Zero in a category means zero under those criteria. F2 mixes never-held, already-sold, and subsequently acquired items, so mitigation may require inventory representation and commitment management as well as attack refusal.

Takeaways

• Track completion and constraint compliance separately. A successful transaction record can conceal ownership, privacy, or commitment failures.
• Include ordinary deadline pressure in safety evaluation. Adversarial refusal tests alone miss failures arising from legitimate workload goals.
• Connect explanations to state transitions. Recognizing a risk in a note does not guarantee safe subsequent behavior.
• Test defenses through new interactions. Post-hoc audits clarify assumptions, but cannot replace rollouts with corrected tools and observations.

Installation and Usage

The code supports Python 3.10–3.12. In a fresh virtual environment, the scripted-agent example needs no API key:

git clone https://github.com/ziyan-wang98/BazaarBench.git
cd BazaarBench
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python examples/smoke_test_20x50.py
bazaar stats runs/smoke_20x50.db

For this review, commit f87e01e ran the 20-agent, 50-tick example with zero action errors and 1,486 events. This verifies a basic simulator run, not the paper’s results. Model-backed execution requires the documented python -m pip install -e ".[llm,memory]", provider configuration, and model calls. Full continuations and judge evaluation were not run here.

References

• BazaarBench, arXiv v1: paper and appendices defining the environment, metrics, statistics, and prompts.
• Official code: simulator and evaluation tools, Apache-2.0.
• Official data organization: released market states, interactions, and evaluation materials.
• Figures and tables: Wang et al., BazaarBench, 2026, CC BY 4.0. The paper and code have separate licenses.

Further Reading

• The Automated but Risky Game: Modeling Agent-to-Agent Negotiations and Transactions in Consumer Markets (Zhu et al., 2025): unequal negotiation outcomes and trading-constraint violations in agent-mediated commerce.
• Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets (Bansal et al., 2025): search, proposal ordering, and welfare in interactive agent markets.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024): database-state evaluation and repeated-trial reliability for tool-using agents.
• Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents (Backlund et al., 2025): sustained inventory, purchasing, and pricing decisions in autonomous business operation.