Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models

This paper presents a hardware-software co-design methodology for accelerating multimodal foundation models. It combines transformer optimization through mixed-precision quantization and structural pruning with inference techniques like speculative decoding and model cascading. A specialized transformer accelerator is designed to efficiently execute the optimized pipeline while meeting bandwidth and latency constraints.