AdaVDR — Adaptive Video Deep Research

In brief

Abstract

Video deep research answers complex questions by jointly understanding video content and retrieving external knowledge from the open Web. Yet, different questions demand substantially different reasoning trajectories, while uncertain grounding and retrieval results can trigger redundant tool calls and compound errors.

We introduce AdaVDR, an agent that adaptively selects tools and revises unreliable intermediate results. We develop these capabilities through two complementary components:

Data Construction. We construct task-specific evidence-acquisition trajectories and apply model-conditioned tool necessity filtering to remove calls the base model can bypass. SFT on 2.9K verified trajectories initializes adaptive tool use and reflection.
RL Training — NASA-GRPO. Our algorithm combines segment-guided evidence rewards, model-conditioned necessity weighting, and tool-type reward normalization to reward useful, necessary evidence acquisition. Beyond SFT, it improves VideoDR accuracy from 52.00% to 66.00% (+14.00 percentage points) and VDR-EE accuracy from 48.00% to 55.20% (+7.20 percentage points).

AdaVDR overview showing adaptive tool invocation and reflection — Figure 1. AdaVDR adaptively invokes tools and reflection for video deep research.

Training curves comparing GRPO, the variant without necessity weighting, and NASA-GRPO — Training curves for GRPO, the variant without necessity weighting, and NASA-GRPO. Pale and dark lines indicate raw and smoothed rollout rewards; the curve without necessity weighting is approximately digitized from a training-dashboard screenshot.

Method

Adapt to the evidence, not a fixed workflow

AdaVDR constructs a variable-length trajectory for every question and only invokes another tool when the current evidence is insufficient.

01

Adaptive Tool Invocation

Select temporal, timestamp, spatial, image, or web tools on demand. Skip a step whenever the answer is already supported.

02

Adaptive Reflection

Detect irrelevant or inconsistent evidence, identify the faulty step, and resume reasoning from a revised decision.

03

NASA-GRPO

Our RL algorithm rewards necessary evidence acquisition with segment-guided credit and tool-type normalization. Beyond SFT, it adds 14.00 percentage points on VideoDR and 7.20 on VDR-EE.

Videoraw temporal evidence

→

Groundtemporal · timestamp · spatial

→

Retrieveimage · web · page visit

→

Answerevidence synthesis

Data Engine

From videos to verified trajectories

Starting from diverse videos and their ASR, our pipeline discovers retrieval-relevant entities and events and follows two task-specific evidence-acquisition paths.

Entity-centric

Recognize who or what appears

Locate the target with temporal, timestamp, and spatial grounding; identify it through image search; then retrieve the external attributes needed to answer the question.

Event-centric

Reason over temporal events

Discover retrieval-relevant temporal events encoded in the video, localize their time spans, select representative visual evidence, recognize the event or scenario, and retrieve related external knowledge.

Only QA pairs requiring both video evidence and external knowledge are retained. Their evidence paths are converted into executable trajectories, checked and retried when necessary, then simplified using Qwen3.5-9B, the base model used to initialize AdaVDR-9B, for model-conditioned tool necessity filtering.

AdaVDR data construction pipeline — Figure 2. QA and trajectory generation, refinement, and model-conditioned tool filtering.

VDR-EE Benchmark

Video evidence meets external knowledge

VDR-EE contains 250 manually verified questions. Every question refers to an identifiable entity or event in the video, requires both video evidence and external knowledge, is unambiguous, and has a reference answer supported by retrieved evidence.

🤗 Get VDR-EE Benchmark on Hugging Face ↗

Dimension 01

Semantic domain

Questions span seven domains: culture, entertainment, industry, news, scene understanding, science, and sports.

Dimension 02

Video-centric categories

Questions are grouped as entity-centric or event-centric, then analyzed by entity multiplicity, grounded-event duration, and full-video duration.

VDR-EE distribution and benchmark statistics — VDR-EE spans seven semantic domains, entity multiplicities, event durations, and full-video durations.

VDR-EE question examples — Representative short-, medium-, and long-event questions and single- and multi-entity questions.

Training

Learn to acquire the evidence that matters

Supervised Fine-Tuning (SFT). We fine-tune Qwen3.5-9B on 2.9K verified trajectories tailored to task requirements and the base model’s capabilities. These trajectories initialize adaptive tool use and reflection, teaching the agent to acquire relevant evidence and revise unreliable intermediate results.
Reinforcement Learning (RL): NASA-GRPO. We further refine the SFT policy with outcome feedback and necessity-aware process credit:
- Outcome reward. Evaluates final-answer correctness and penalizes invalid rollouts or those that reach the turn limit, providing a rollout-level success signal.
- Segment-guided evidence reward. Rewards reasoning segments that recover correct target entities or external facts needed to answer the question. A clue receives credit only when it has not already been obtained in earlier segments.
- Model-conditioned necessity weighting. Tests whether the current policy can recover the same target information while bypassing a tool. Bypassable calls receive less credit; the policy snapshot is held fixed during reward evaluation and refreshed at each RL iteration.
- Tool-type reward normalization. Normalizes process rewards within each tool type so that tools with higher positive-reward rates do not dominate learning. The process signal is also scaled to match the average magnitude of the outcome advantage.
- Tool-level credit assignment. Combines the outcome advantage applied to all generated tokens with local process credit applied only to tokens of the credited tool action. This links successful evidence acquisition to the specific tool calls responsible for it.

NASA-GRPO training: evidence segments, necessity weighting, tool-type normalization, and token-level credit assignment — NASA-GRPO provides local feedback for useful evidence acquisition while reducing credit for tool calls the policy can bypass.

Training Curves

Compare the learning trajectories

GRPO shows limited sustained improvement in later steps. The variant without necessity weighting improves early, while NASA-GRPO maintains a higher smoothed reward toward the end of the displayed training window. Held-out accuracy provides complementary evidence of policy improvement.

Three training curves: GRPO, without necessity weight, and NASA-GRPO — Pale and dark lines show raw and smoothed rollout rewards. The curve without necessity weighting is approximately digitized from a training-dashboard screenshot.

Main Results

From base model to adaptive video research

AdaVDR-9B improves over its agentic Qwen3.5-9B base model on both benchmarks. Gains are absolute percentage-point differences.

VideoDR · AdaVDR-9B

66.00%+29.00 pp over base

VDR-EE · AdaVDR-9B

55.20%+15.20 pp over base

VideoDR · SFT

52.00%+15.00 pp over base

VDR-EE · SFT

48.00%+8.00 pp over base

Qualitative Analysis

Skip when confident. Reflect when needed.

Entity questions can omit unnecessary grounding; event questions can recover from mismatched retrieval by selecting stronger visual evidence.

AdaVDR qualitative reasoning cases — Adaptive trajectories for entity- and event-oriented questions.

Citation

BibTeX

@article{zhang2026adavdr,
  title   = {AdaVDR: Adaptive Tool Use and Reflection for Video Deep Research},
  author  = {Zhang, Xintong and Fan, Xiaomeng and Yan, Shilin and He, Ekko and Liu, Zicheng and Zou, Zijian and Zhang, Guannan and Wu, Yuwei and Gao, Zhi and Xue, Hongwei},
  journal = {arXiv preprint arXiv:2608.25559},
  year    = {2026}
}