A benchmark for visual artifact authoring

VisualArtifactSpec 1.0Evaluating Visual Artifacts
under Task Specifications

Shuyi Ouyang1,2*Xiaojun Tang2*Zijian Zou2Xintong Zhang2
Mingli Song1Guannan Zhang2†Hongwei Xue2†
1 Zhejiang University2 Accio Team, Alibaba Group

* Equal contribution† Corresponding author

Evaluating authoring capabilities across artifact families.
Assessing requirement fulfillment and how authoring choices serve the audience, purpose, and medium.

Explore the benchmark
211Source-grounded tasks
4Artifact families
14Task types
8Application purposes
Overview of 47 presentation tasks, 72 static graphics tasks, 32 animation and video tasks, and 60 web interface tasks.
One benchmark across slides, space, time, and interface states. Representative illustrations from the report; not evaluated agent outputs. Click figures to enlarge ↗

01 / Overview

Beyond a
finished artifact.

Visual artifact tasks ask agents to turn task specifications and source materials into useful outputs. Comparing artifact families reveals whether authoring strengths generalize across media, while assessing how content, organization, and presentation serve the intended audience and purpose.

VisualArtifactSpec 1.0 (VAS) comprises 211 creation and modification tasks across four artifact families. Evaluation separates Task Fulfillment from Artifact Performance, using shared criteria, task-specific rating anchors, and inspectable evidence.

02 / Leaderboard

VisualArtifactSpec leaderboard

Across nine model–harness configurations, high Task Fulfillment can coexist with deficiencies in Artifact Performance. Compare dimensions and artifact families to examine strengths and weaknesses that overall scores can obscure.

Total score across nine configurations

Nine model–harness configurations ranked by total score S, with brand logos. GPT-6 Astra leads at 70.5, followed by GPT-6.1 Sol at 69.4 and Claude Opus 5.5 at 65.0.
Overall performance. Total score S = B + Δ, where B is the Task Fulfillment base score and Δ is the Artifact Performance adjustment. The dashed line marks a reference score of 50. Configurations are ordered by total score; the tables below show the score breakdown. Click to enlarge.

Task-level performance distribution

Artifact Performance adjustments for nine configurations on 38 matched tasks. Filled circles indicate full fulfillment, open circles incomplete fulfillment, and diamonds mark each configuration's mean.
Task-level performance. Nine configurations on the same fixed 38-task sample. Filled circles indicate B = 50; open circles indicate B < 50. Diamonds mark means, horizontal lines show observed ranges, and the dashed line marks Δ = 0.
Experimental results

Configurations are ordered by total score across all three views.

Score ranges: B, 0–50; Δ, −50 to +50; S, −50 to 100. Valid tasks reports scored-task coverage out of 211.

How are these scores calculated?

03 / Benchmark

Different media. Shared authoring demands.

Task categories follow the primary objective, rather than the file extension. HTML may encode slides, an animation, or a website.

47 tasks · 22.3%

Presentations

Narrative, layout, and editable structure across slides.

  • Original decks 31
  • Template adaptation 14
  • Existing-deck revision 2
72 tasks · 34.1%

Static graphics

Hierarchy and coherent expression across space and pages.

  • Posters 8
  • Infographics & visual summaries 24
  • Banners 10
  • Multi-page series 30
32 tasks · 15.2%

Animations & videos

Processes, key states, and visual continuity across time.

  • Animated graphics 8
  • Animated videos 22
  • Edited videos 2
60 tasks · 28.4%

Web interfaces

Content, interaction, and workflows across interface states.

  • Content browsing 22
  • Data exploration 16
  • Tools & simulations 13
  • Stateful workflows 9

Designed for a purpose.

Each task has one primary application purpose, independently of its artifact family.

Conceptual understanding 61Decision support 35Promotion & engagement 27Research communication 24Practical guidance 24Public information 18Production & operations 14Culture & leisure 8

04 / Context matters

The same information.
Different contexts.

Required information is retained while audience, purpose, and medium shape content emphasis, information organization, and visual presentation.

Three paired examples show content emphasis in technical and product promotion slides, information organization in a service overview card and web setup guide, and visual presentation in a coffee tasting card and cafe promotional banner.
Illustrative designs from the report, not official materials or experimental results. Coffee photo: Nathan Dumlao / Unsplash.

05 / Evaluation

Two complementary assessments.

Task Fulfillment and Artifact Performance are assessed separately. Judgments link criteria to source materials and native, rendered, temporal, or interaction evidence, as appropriate to the artifact.

Base score · 0 to 50

Task Fulfillment

Does the artifact meet the specified obligations?

CP Compliance
Required facts, formats, modification boundaries, and native editability.
FN Functionality
Required operations, state transitions, and consistency after interaction, where applicable.

Equal weight across acceptance groups. A group passes only when all its requirements pass.

Adjustment · −50 to +50

Artifact Performance

How well do its authoring choices serve the intended audience, purpose, and medium?

C Content25%
Meaning, selection, qualifications, and explanation.
O Organization15%
Relationships, reading order, narrative, and continuity.
V Visuals50%
Legibility, hierarchy, encoding, and contextual fit.
D Delivery10%
Direct use, native structure, and traceability.

18 shared criteria, instantiated through task-specific rating anchors for the intended audience, purpose, and medium. Ratings range from −3 to +3; zero denotes complete basic satisfaction without substantive defects, but ordinary quality. Ratings are mapped to points before weighted aggregation.

Map each rating before averaging

Each criterion is mapped to points individually. Within each dimension, mapped points are averaged using the criterion weights.

Nonlinear mapping from criterion ratings to points
Rating r−3−2−10+1+2+3
Points φ(r)−50−25−100+10+25+50

pd = Σ wk φ(rk) / Σ wk for criteria k within dimension d

Final score

S = B + Δ

B = 50F
Δ = 0.25pC + 0.15pO + 0.50pV + 0.10pD

F is the fraction of passed acceptance groups.
pd is the dimension point score (−50 to +50).
Negative totals are retained; theoretical range: −50 to 100.

06 / Construction

From source materials to verifiable tasks.

211 tasks selected from 322 candidates, considering coverage, evaluability, and source diversity.

Three construction stages: sources and feasibility, task and reference construction, execution and selection.
Agent-visible inputs and evaluator-only references are kept separate. Rubrics and input versions are frozen before comparison.

07 / Case studies

Same task.
Different authoring choices.

Inspect eight evaluated outputs across four complex tasks: research evidence, mobile reading, scientific motion, and interactive state. Compare the original artifacts with their recorded assessments.

Four tasks · Eight outputs English artifacts · Hugging Face release IDs · Current v3 scoring

presentations-001 · English

Keep the conditions with the claim.

View released task ↗

Create an eight-slide ResNet research-group presentation: explain the residual mechanism, reproduce experimental comparisons, distinguish evaluation protocols, and discuss limitations in an editable deck.

Both outputs satisfy the acceptance groups. The comparison exposes a subtler difference: whether experimental conditions stay attached to the numbers they qualify.

GPT-6.1 Sol

Above baseline performance
73.20Total S50.00Base B23.20Adjustment Δ25.00Content25.00Organization22.90Visuals17.50Delivery

CP: 10/10 groups passed. C/O/V/D are current-protocol points (−50 to +50).

O2 · Rating +2

Single-model validation and ensemble test results occupy separate cards with their own protocols. The deck still has small tick labels in an earlier reproduced plot (V3 +1).

DeepSeek V4.1 Flash

Above baseline performance
63.65Total S50.00Base B13.65Adjustment Δ14.20Content13.33Organization14.20Visuals10.00Delivery

CP: 10/10 groups passed. C/O/V/D are current-protocol points (−50 to +50).

O2 · Rating -1

The final slide groups different experimental settings under “Evidence at equal budget.” Its shared qualifier requires the reader to recover the intended scope from an earlier slide.

Selected comparisons illustrate strengths and limitations, not average model quality. Scores and criterion judgments come from the saved evaluations; no new grading is implied. Slides and cards show full original previews; videos and labs are embedded. Inspect scores, judgments, and artifact hashes ↗

08 / Citation

Cite VisualArtifactSpec.

If you use VisualArtifactSpec in your research, please cite our report.

BIBTEX
@misc{ouyang2026visualartifactspec10evaluatingvisual,
  title={VisualArtifactSpec 1.0: Evaluating Visual Artifacts under Task Specifications},
  author={Shuyi Ouyang and Xiaojun Tang and Zijian Zou and Xintong Zhang and Mingli Song and Guannan Zhang and Hongwei Xue},
  year={2026},
  eprint={2611.21382},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2611.21382},
}