近日，实验室博士生张博关于大模型对话生成的研究成果被计算机领域顶级期刊Information Fusion录用。Information Fusion期刊属于中科院一区期刊，影响因子14.8。

题目：Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation

Abstract: Integrating multimodal knowledge into large language models (LLMs) represents a significant advancement in dialogue generation capabilities. However, the effective incorporation of such knowledge in zero-resource scenarios remains a substantial challenge due to the scarcity of diverse, high-quality dialogue datasets. To address this, we propose the Visual Implicit Knowledge Distillation Framework (VIKDF), an innovative approach aimed at enhancing LLMs for enriched dialogue generation in zero-resource contexts by leveraging implicit multimodal knowledge. VIKDF comprises two main stages: knowledge distillation, using an Implicit Query Transformer to extract and encode visual implicit knowledge from image-text pairs into knowledge vectors; and knowledge integration, employing a novel Bidirectional Variational Information Fusion technique to seamlessly integrate these distilled vectors into LLMs. This enables the LLMs to generate dialogues that are not only coherent and engaging but also exhibit a deep understanding of the context through implicit multimodal cues, effectively overcoming the limitations of zero-resource scenarios. Our extensive experimentation across two dialogue datasets shows that VIKDF outperforms existing state-of-the-art models in generating high-quality dialogues.

中文摘要：摘要：将多模态知识整合到大语言模型（LLMs）中能够显著提升了对话生成能力。然而，由于缺乏多样化、高质量的对话数据集，在零资源场景下有效地融合这些知识仍是一大挑战。为了解决这一问题，我们提出了一种隐式视觉知识蒸馏框架（VIKDF），该框架创新性地利用隐式多模态知识，以在零资源条件下提升LLMs的对话生成能力。VIKDF包括两个主要阶段：（1）知识蒸馏阶段，通过一个隐式查询Transformer，从图文对中提取并编码隐式视觉知识为知识向量；（2）知识融合阶段，提出一种新颖的双向变分信息融合技术，将这些蒸馏出的知识向量无缝地整合到LLMs中。这使得LLMs生成的对话不仅连贯且富有吸引力，还能通过隐式的多模态线索体现出对上下文的深入理解，从而有效地克服零资源情境下的限制。在两种对话数据集上的广泛实验表明，VIKDF在生成高质量对话方面优于现有的最先进模型。