Sign in

Category: Other

  • SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

    SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

    SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

    arXiv:2610.16412v1 Announce Type: new Abstract: As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizing large log streams into human-readable reports. Further progress requires meaningfully evaluating this function, yet existing benchmarks leave end-to-end summarization from large log streams underexplored due to the prohibitive cost and complexity of collecting real-world data. We introduce SLS-Bench, a benchmark for incident log summarization featuring 90 tasks grounded in real-world incident reports, each with a synthetic log stream and a reference summary. To construct SLS-Bench, we developed a data generation pipeline that converts incident reports into structured incident models, which guide the generation of synthetic logs and reference summaries. This pipeline combines narrative grounding, explicit incident modeling, generator-verifier loops, and statistical comparisons with real-world log streams to produce realistic and controllable observability data. Our evaluation of 14 LLMs on SLS-Bench shows that while frontier models can produce useful summaries, they do so at substantial cost and latency, and open-weight models lag significantly. SLS-Bench provides a framework for systematic evaluation of log analysis agents and a scalable blueprint for future synthetic observability benchmarks.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16412

  • Chain Dimension Scaling: Communication-Efficient Loop-Style Scaling for Industrial Short-Video Ranking

    Chain Dimension Scaling: Communication-Efficient Loop-Style Scaling for Industrial Short-Video Ranking

    arXiv:2610.16415v1 Announce Type: new Abstract: Industrial recommendation rankers keep scaling in width and depth to billions of parameters, but this increasingly incurs costs in parameter growth, distributed communication overhead, and training inefficiency. We introduce \underline{C}hain \underline{D}imension \underline{S}caling (CDS), a scaling strategy orthogonal to width and depth. CDS repeatedly applies a single feature-interaction block with parameters shared across iterations, so the number of iterations (the chain length) becomes a new scaling dimension that adds no parameters. A lightweight intra-chain residual connection stabilizes optimization and improves quality. CDS enables two regimes. In the \emph{compression regime}, shallow blocks with a longer chain match deeper baselines with fewer parameters and faster training; in the \emph{quality-scaling regime}, increasing chain length under a fixed parameter budget yields consistent offline and online gains, suggesting the chain dimension adds modeling capacity beyond parameter reuse. We evaluate CDS in a large-scale multi-task short-video ranking system in production. In the compression regime, CDS stays within $0.03\%$ of baseline offline quality while reducing trainable parameters by 26.6\% and improving training throughput by 6.4\%, with neutral-to-positive online participation. In the quality-scaling regime, quality improves monotonically with chain length. Wall-clock analysis shows the efficiency gain comes mainly not from reduced FLOPs but from lower cross-device synchronization overhead and reduced optimizer update cost. These results position parameter-shared iterative computation as a practical, communication-efficient scaling axis for industrial ranking systems.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16415

  • AuthGateBench: Benchmarking LLMs as Authorization Gate

    AuthGateBench: Benchmarking LLMs as Authorization Gate

    arXiv:2610.16426v1 Announce Type: new Abstract: Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classifier, typically a large language model (LLM). Such systems, adopted in tools like Claude Code and Codex, remain vulnerable to failure modes such as scope escalation, where the agent takes actions exceeding what the user authorized. Existing benchmarks focus mainly on malicious requests, evaluating refusal under an implicit policy. We introduce AuthGateBench, the first policy-guided benchmark for authorization gating under benign requests. We implement a novel pipeline where an LLM generates contrast pairs with identical action and trajectory but differing request, flipping the label between block and allow. This allows us to assess whether models reason over context rather than the action alone. We further examine whether auxiliary environment signals can help models make better decisions. The resulting benchmark comprises 1,400 instances spanning five failure modes drawn from real-world Claude Code incidents. Evaluations show accuracy degrades unevenly across failure modes as models over-rely on internal safety alignment, and models still block most actions even when explicitly authorized, revealing unstable rather than deterministic decisions; retrospectively retrieved environment signals, however, substantially improve authorization accuracy.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16426

  • World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models

    World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models

    arXiv:2610.16365v1 Announce Type: new Abstract: Video world models must generate physically plausible futures, not just realistic frames. Existing approaches typically use semantic, physical, or predictive information to guide the visual generation stream, leaving a fundamental question: where inside the generator should predictions of scene evolution live? Standard video tokens jointly encode appearance and dynamics, but provide no distinct state for learning and reading predictive dynamics. We answer this question with the Physical Registry: temporally grounded registry slots inside a video DiT that are initialized from an observation-conditioned forecast, co-evolve with video tokens, and receive future-representation supervision, making predictive dynamics an explicit, readable state of the generator. In a controlled comparison of predictive-state interfaces on Physics-IQ Verified, Physical Registry improves vanilla fine-tuning by 9.9/8.7 points on I2V/V2V, the largest gain among the interfaces we compare. Across all Cosmos and WAN backbones, Physical Registry improves Physics-IQ Verified scores. With only 0.45% additional parameters, it enables Cosmos 3 Nano 16B to surpass the published I2V score of Cosmos 3 Super 64B (43.56 vs. 42.7). On WorldModelBench, it also improves physical adherence and the total score for all four evaluated backbones. Because the registry becomes candidate-specific while remaining readable, the Physical Registry Selector (PRS) ranks emerging futures by how far their intermediate registry readouts depart from the shared forecast before full denoising is complete. Across four backbones, this selection improves I2V scores by 5.6-15.9% without completed-video external scoring. Together, these results support a broader principle: dynamics prediction should live as an explicit model state that co-evolves with the video representation. Project page: https://grainsack.github.io/physical-registry/

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16365

  • RetroPlanner: Multi-Route Expansion and Multi-Objective Reasoning for LLM-Based Retrosynthesis Planning

    RetroPlanner: Multi-Route Expansion and Multi-Objective Reasoning for LLM-Based Retrosynthesis Planning

    arXiv:2610.16391v1 Announce Type: new Abstract: Multi-step retrosynthesis planning is often considered solved once a route to purchasable starting materials is found, but such a route should also be practical in real-world chemistry. Recent LLM-based planners formulate this process as sequential decision-making over an AND-OR graph, yet typically expand a single molecule per turn and are optimized for solvability alone, leaving alternative routes uncompared across the multiple objectives that determine practicality before the search budget is committed. We introduce RetroPlanner, an LLM-based planner that jointly compares multiple routes to steer the search toward higher synthesizability, as measured by reaction plausibility, round-trip consistency, and starting-material price. RetroPlanner is trained to reason over chemical evidence: structural progress from the target molecule, bond changes and reaction classes, where candidate routes diverge from one another, and how they rank across multiple objectives. Experiments on multi-step retrosynthesis benchmarks demonstrate that RetroPlanner recovers more alternative routes than prior LLM-based planners while improving synthetic route quality and maintaining a high success rate.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16391

  • Recursive Environment Scaling for General Long-Horizon Agents

    Recursive Environment Scaling for General Long-Horizon Agents

    arXiv:2610.16397v1 Announce Type: new Abstract: We study environment scaling as a way to increase agent-benchmark difficulty while holding core task objectives fixed. Our framework, Recursive Environment Scaling (RES), transforms existing benchmark environments while preserving their objectives and completion requirements. A proposer changes information access, resource relations, and execution dependencies; execution-grounded checks validate the resulting candidate, and accepted changes are inherited across rounds. This produces related environments for paired evaluation and training-trajectory collection. Experiments on 294 initial tasks from three benchmarks show lower task success for all six evaluation models, with decreases of 57.14–77.14 percentage points on Claw-Eval-Live. With equal trajectory counts and identical optimization settings, fine-tuning on RES trajectories outperforms original-trajectory SFT in all nine model–benchmark comparisons by 7.00–14.61 percentage points. Ablations identify recursive inheritance and dependency composition as important contributors to increased difficulty. Together, these results support environment scaling for both agent evaluation and training.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16397

  • EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

    EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

    arXiv:2610.16402v1 Announce Type: new Abstract: Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily evaluate development toward fixed specifications. We introduce EvoWeb, a novel benchmark comprising 320 cumulative requests over 32 web applications, grounded in real-world product features and structured by functional dependency graphs. Agents iteratively modify a persistent codebase, while a multimodal web agent tests each intermediate application in a live browser, assessing newly requested functionality, cross-request integration, and previously requested functionality. To characterize reliability throughout development, we integrate trajectory performance across user-tolerance thresholds rather than impose a fixed stopping criterion. Experiments across open- and closed-source frontier models show that EvoWeb reveals clear performance differences between models, with cross-request integration emerging as a key challenge. Further analysis shows that later requests with deeper or more numerous dependencies are solved less accurately, while runtime errors become more frequent over successive requests. EvoWeb provides a testbed for evaluating and improving coding agents in long-horizon, multi-request development.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16402

  • TRAP: Tool-result Replay Attack for Probing Misbinding in LLM Agents

    TRAP: Tool-result Replay Attack for Probing Misbinding in LLM Agents

    TRAP: Tool-result Replay Attack for Probing Misbinding in LLM Agents

    arXiv:2610.16333v1 Announce Type: new Abstract: As AI agents increasingly act through external tools, tool results are no longer simple outputs but execution artifacts that must remain correctly associated with the context that produced them. Yet as results persist across caches, recovery mechanisms, and shared runtime state, this association can break, causing a real and unmodified result to be consumed in an incompatible execution context and lead the agent to an unsafe action. We characterize this failure as tool-result misbinding. To systematically study this failure, we introduce ReplayBench, a benchmark that uses controlled tool-result replay to evaluate misbinding across 19 representative agent scenarios, 4 failure properties, and 3 replay mechanisms. Across eight LLMs, three agent frameworks, and diverse stateful workflows, we find widespread replay-induced failures, with a mean Final-State Error Rate of 0.649 under cross-task replay. Stage-wise analysis localizes these failures to model-side action generation. Further analyses show that replay susceptibility varies with task similarity, shows little dependence on object similarity, and remains high under result staleness and implicit environment changes. These findings motivate runtime validity mechanisms that eliminate final-state errors in our evaluated settings while preserving legitimate recovery. The results call for explicit contextual binding in tool-using agent runtimes. Our work is public at https://github.com/Phitooo/Replaybench.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16333

  • Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate

    Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate

    arXiv:2610.16343v1 Announce Type: new Abstract: Multi-agent debate has shown promise for improving the reasoning of large language models, yet recent theory suggests its benefits are highly regime-dependent. While interaction can amplify informative signals under corrective conditions, symmetric debate dynamics are neutral in expectation, often making majority voting preferable. We reconcile these views by arguing that debate is effective only when invoked at the right time and with appropriate structure. Based on this insight, we propose LASE: Leader-Adaptive Structured Engagement, a leader-centric multi-agent debate framework that selectively engages interaction only in non-neutral regimes. LASE introduces an asymmetric leader-supporter architecture that enables directed information flow and selective signal amplification, while defaulting to simple aggregation otherwise. Experiments across diverse reasoning benchmarks show that LASE achieves multi-agent-level performance with near single-agent token cost, substantially improving efficiency over static debate and voting baselines.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16343

  • ChronoScribe: Semantic-Numerical Specialization for Instruction-Driven Time Series Forecasting

    ChronoScribe: Semantic-Numerical Specialization for Instruction-Driven Time Series Forecasting

    arXiv:2610.16345v1 Announce Type: new Abstract: Recent advances in instruction-driven time series forecasting highlight a fundamental tension: systems must simultaneously master natural-language instruction following and precise numerical prediction. Current approaches often force a compromise. Large language models (LLMs) can interpret unstructured requests but may be inaccurate when asked to produce forecast values directly, whereas time series foundation models (TSFMs) provide numerical forecasts but typically require structured inputs specifying the target series, context window, and horizon. We present ChronoScribe, which applies semantic-numerical specialization to this workflow. A LoRA-adapted LLM proposes an execution path and context length; deterministic rules resolve the executable configuration. On the specialist path, a frozen TSFM produces quantile trajectories for the task-specified horizon. Deterministic trajectory summaries condition the response, while the LLM neither generates nor revises predicted values. We evaluate a Qwen3.5-9B–Chronos-2 implementation across 23 evaluation dimensions and 30 public datasets against eleven proprietary direct LLMs and three open TSFMs. ChronoScribe has the lowest error on sMAPE, MASE, WAPE, RMSE, and MAE among all evaluated systems, including fixed-context controls. Across these metrics, even the lowest direct-LLM error remains 2.0–2.7 times the corresponding ChronoScribe error. At the system level, ChronoScribe achieves a Task-Level Score (TLS) of 65.5, 10.6 points above the strongest direct-LLM baseline. The results support ChronoScribe's division of labor, in which the LLM handles interpretation, configuration, and communication while the TSFM handles numerical forecasting.

    Fallback brief based on the available RSS feed; no unsupported details have been added.

    Source: arXiv
    Original article: https://arxiv.org/abs/2610.16345