AI Infra Bench

Tasks

17 offline tasks with execution-based behavioral and e2e tests.

Bug fix

Anthropic Inline System Template

Adapt Anthropic inline system messages to chat-template ordering constraints.

Serving API
Bug fix

ASR Chunk Spacing

Preserve word boundaries when merging multi-chunk ASR output.

Serving API, Model implementation
Bug fix

Async KV Token Accounting

Preserve exact computed-token accounting across asynchronous external KV loads.

Scheduler, KV cache data movement
Bug fix

Async Spec Placeholder Discard

Prevent stale speculative frames from underflowing async scheduler placeholder accounting.

Scheduler
Bug fix

Concurrent Config Refresh

Tolerate transient configuration-file read failures during concurrent cache refreshes.

Model, Frontend API
Feature

CPU Offload Reset Inflight

Reset CPU-offload cache state without releasing blocks still used by asynchronous transfers.

KV cache data movement
Feature

Implement Anthropic Rust Serving

Implement Anthropic SDK 1.3 Messages compatibility in vLLM's Rust frontend.

Serving API, Input processing, Streaming
Bug fix

KV Admission Thrashing

Keep mixed decoding and long-prompt workloads making progress without over-serializing KV-cache admission.

Scheduler, Memory KV

17 matching tasks. Page 1 of 3. Showing tasks 1 to 8.