Multimodal Encoder Cache Accounting
Account for multimodal encoder-cache usage in embedding rows, not placeholder tokens.
We use multimodal prompts in vLLM where an image or other media item can occupy many prompt positions but produce far fewer embedding rows. For example, an item spanning 100 prompt positions may supply only eight rows, yet it can be rejected by a cache with room for those eight rows. Please fix the accounting so cache capacity, allocation, eviction, capacity planning and encoder scheduling all reflect the actual encoder output, while ordinary placeholders without an embedding mask keep working as before. For the same encoder output, adding timestamps or other non-embedding prompt tokens must not make the cached embedding payload grow with the placeholder span.
Chunked prefill must select the embeddings that belong to the current prompt window, in the right order, even when prompt offsets and embedding offsets differ. When neither the current window nor speculative-decoding lookahead needs an embedding, the request must still advance through its prompt tokens without scheduling the encoder or consuming encoder budget, even if the budget or available cache capacity is zero. When a later window first needs an uncached item, encoding that item must reserve space and budget for its entire output, not just the rows consumed by that window. Previously encoded items should be reused correctly across subsequent windows.
Work in /workspace/vllm. Keep request handling, scheduling, profiling and the model runner consistent through encoding, caching and partial reads. Sparse and mask-free items, all-false masks, empty placeholders and multiple media items must all remain well-defined. You can choose the internal representation and interfaces, but existing public behavior must remain compatible.