ZIP Lab

(Selected Publications. * equal contribution, # corresponding author)

Preprint

ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsification and KV Cache Compression

Yefei He, Feng Chen, Jing Liu, Wenqi Shao, Hong Zhou, Kaipeng Zhang, Bohan Zhuang

[Paper]

ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models

Jing Liu, Ruihao Gong, Mingyang Zhang, Yefei He, Jianfei Cai, Bohan Zhuang#

[Paper]

T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching

Zizheng Pan, Bohan Zhuang#, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, Anima Anandkumar

[Paper][Project][Code]

2024

MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, Bohan Zhuang#

[Paper] NeurIPS 2024

ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, Bohan Zhuang#

[Paper][Code] NeurIPS 2024

MVSplat360: Feed‑Forward 360° Scene Synthesis from Sparse Views

Yuedong Chen, Chuanxia Zheng, Haofei Xu, Bohan Zhuang, Andrea Vedaldi, Tat-Jen Cham, Jianfei Cai

[Paper][Project Page][Code] NeurIPS 2024

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Pengcheng Chen*, Jin Ye*#, Guoan Wang*, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Junjun He, Yu Qiao

[Paper][Project Page][HuggingFace] NeurIPS 2024 Datasets and Benchmarks Track

MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images

Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, Jianfei Cai

[Paper][Project][Code] ECCV 2024 (Oral, Top 2%)

LongVLM: Efficient Long Video Understanding via Large Language Models

Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, Bohan Zhuang#

[Paper] ECCV 2024 (Oral, Top 2%)

Stitched ViTs are Flexible Vision Backbones (SN-Net v2)

Zizheng Pan, Jing Liu, Haoyu He, Jianfei Cai, Bohan Zhuang#

[Paper][HuggingFace][Code] ECCV 2024

Motion Mamba: Efficient and Long Sequence Motion Generation

Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, Hao Tang

[Paper][Project] ECCV 2024

QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models

Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, Bohan Zhuang#

[OpenReview][Code-1][Code-2] ICLR 2024

EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models

Yefei He, Jing Liu, Weijia Wu, Hong Zhou#, Bohan Zhuang#

[OpenReview][Code] ICLR 2024 (Spotlight, Top 5%)

Object-Aware Inversion and Reassembly for Image Editing

Zhen Yang, Ganggui Ding, Wen Wang, Hao Chen#, Bohan Zhuang#, Chunhua Shen

[OpenReview][Project] ICLR 2024

Efficient Stitchable Task Adaptation

Haoyu He, Zizheng Pan, Jing Liu, Jianfei Cai, Bohan Zhuang#

[Paper][Code] CVPR 2024

ModaVerse: Efficiently Transforming Modalities with LLMs

Xinyu Wang, Bohan Zhuang, Qi Wu

[Paper][Code] CVPR 2024

LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning

Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang

[Paper] ACL 2024 Findings

SAM-Med3D-MoE: Towards a Non-Forgetting Segment Anything Model via Mixture of Experts for 3D Medical Image Segmentation

Guoan Wang*, Jin Ye*, Junlong Cheng, Tianbin Li, Zhaolin Chen, Jianfei Cai, Junjun He, Bohan Zhuang#

[Paper] MICCAI 2024

2023

Stitchable Neural Networks

Zizheng Pan, Jianfei Cai, Bohan Zhuang#

[Paper][Project][Code] CVPR 2023 (Highlight, Top 2.5%)

PTQD: Accurate Post-Training Quantization for Diffusion Models

Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou#, Bohan Zhuang#

[OpenReview][Code] NeurIPS 2023

Mask Propagation for Efficient Video Semantic Segmentation

Yuetian Weng, Mingfei Han, Haoyu He, Mingjie Li, Xiaojun Chang, Bohan Zhuang#

[OpenReview][Code] NeurIPS 2023

Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction

Zeshuai Deng, Zhuokun Chen, Shuaicheng Niu, Thomas H. Li, Bohan Zhuang#, Mingkui Tan#

[OpenReview][Code] NeurIPS 2023

Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning

Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, Bohan Zhuang#

[Paper][Code] ICCV 2023 (Oral, Top 2.4%)

BiViT: Extremely Compressed Binary Vision Transformer

Yefei He, Zhenyu Lou, Luoming Zhang, Jing Liu, Weijia Wu, Hong Zhou#, Bohan Zhuang#

[Paper] ICCV 2023

Dynamic Focus-aware Positional Queries for Semantic Segmentation

Haoyu He, Jianfei Cai, Zizheng Pan, Jing Liu, Jing Zhang, Dacheng Tao, Bohan Zhuang#

[Paper][Code] CVPR 2023

Pruning Self-attentions into Convolutional Layers in Single Path

Haoyu He, Jing Liu, Zizheng Pan, Jianfei Cai, Jing Zhang, Dacheng Tao, Bohan Zhuang#

[Paper][Code] TPAMI 2023

Single-path Bit Sharing for Automatic Loss-aware Model Compression

Jing Liu, Bohan Zhuang, Peng Chen, Chunhua Shen, Jianfei Cai, Mingkui Tan

[Paper] TPAMI 2023

End-to-end One-shot Human Parsing

Haoyu He, Jing Zhang, Bohan Zhuang, Jianfei Cai, Dacheng Tao

[Paper][Code] TPAMI 2023

A Survey on Efficient Training of Transformers

Bohan Zhuang#, Jing Liu, Zizheng Pan, Haoyu He, Yuetian Weng, Chunhua Shen

[Paper] IJCAI 2023 Survey Track

SwitchGPT: Adapting Large Language Models for Non-Text Outputs

Xinyu Wang, Bohan Zhuang, Qi Wu

[Paper]

2022

EcoFormer: Energy-Saving Attention with Linear Complexity

Jing Liu*, Zizheng Pan*, Haoyu He, Jianfei Cai, Bohan Zhuang#

[OpenReview][Code] NeurIPS 2022 (Spotlight, Top 3%)

Fast Vision Transformers with HiLo Attention

Zizheng Pan, Jianfei Cai, Bohan Zhuang#

[OpenReview][Code] NeurIPS 2022 (Spotlight, Top 3%)

An Efficient Spatio-Temporal Pyramid Transformer for Action Detection

Yuetian Weng, Zizheng Pan, Mingfei Han, Xiaojun Chang, Bohan Zhuang#

[Paper][Code] ECCV 2022

Automated Progressive Learning for Efficient Training of Vision Transformers

Changlin Li, Bohan Zhuang#, Guangrun Wang, Xiaodan Liang, Xiaojun Chang, Yi Yang

[Paper][Code] CVPR 2022

Less is More: Pay Less Attention in Vision Transformers

Zizheng Pan, Bohan Zhuang#, Haoyu He, Jing Liu, Jianfei Cai

[paper] [Code] AAAI 2022

Structured Binary Neural Networks for Image Recognition

Bohan Zhuang, Chunhua Shen, Mingkui Tan, Peng Chen, Lingqiao Liu, Ian Reid

[paper] IJCV 2022

Mesa: A Memory-saving Training Framework for Transformers

Zizheng Pan, Peng Chen, Haoyu He, Jing Liu, Jianfei Cai, Bohan Zhuang#

[Paper][Code]

Sharpness-aware Quantization for Deep Neural Networks

Jing Liu, Jianfei Cai, Bohan Zhuang#

[Paper][Code]

FocusFormer: Focusing on What We Need via Architecture Sampler

Jing Liu, Jianfei Cai, Bohan Zhuang#

[Paper]

Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints

Jing Liu, Bohan Zhuang#, Mingkui Tan, Xu Liu, Dinh Phung, Yuanqing Li, Jianfei Cai

[Paper]

2021

Scalable Vision Transformers with Hierarchical Pooling

Zizheng Pan, Bohan Zhuang#, Jing Liu, Haoyu He, Jianfei Cai

[Paper][Code] ICCV 2021

FATNN: Fast and Accurate Ternary Neural Networks

Peng Chen*, Bohan Zhuang*, Chunhua Shen

[Paper][Code] ICCV 2021

Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations

Bohan Zhuang*, Mingkui Tan*, Jing Liu*, Ian Reid, Chunhua Shen#

[Paper][Code] TPAMI

Discrimination-aware Network Pruning for Deep Model Compression

Jing Liu*, Bohan Zhuang*, Zhuangwei Zhuang*, Yong Guo, Junzhou Huang, Jinhui Zhu, Mingkui Tan#

[Paper][Code] TPAMI

AQD: Towards Accurate Fully-Quantized Object Detection

Peng Chen*, Jing Liu*, Bohan Zhuang#, Mingkui Tan, Chunhua Shen

[Paper][Code] CVPR 2021 (Oral)

SA-BNN: State-Aware Binary Neural Network

Chunlei Liu*, Peng Chen*, Bohan Zhuang*, Chunhua Shen, Baochang Zhang, Wenrui Ding

[Paper] AAAI 2021