BLIP: 统一视觉-语言理解与生成的引导性语言图像预训练

模型卡片：基于大型架构的BLIP模型在视觉问答任务上进行了训练（使用了ViT大型骨干）。

简介

摘要中来自 paper 的作者写道：

视觉-语言预训练（VLP）已经提高了许多视觉-语言任务的性能。然而，现有的大多数预训练模型只在理解型任务或生成型任务中表现出色。此外，通过扩大使用从网络中收集到的含有噪音的图像文本对的数据集，实现了性能的大幅提升，但这种方式并不是最优的监督来源。在本文中，我们提出了BLIP，这是一种新的VLP框架，能够灵活应用于视觉-语言理解和生成任务。BLIP通过引导生成标题，有效地利用了网络数据中的噪音，其中一个标题生成器生成合成标题，然后通过一个过滤器来去除噪音标题。我们在广泛的视觉-语言任务中取得了最先进的结果，如图像文本检索（平均召回率@1提高了2.7%）、图像字幕（CIDEr提高了2.8%）和视觉问答（VQA评分提高了1.6%）。BLIP还展示了强大的泛化能力，可以直接以零-shot方式转移到视频-语言任务中。代码、模型和数据集已经发布。

使用方法

您可以使用此模型进行有条件和无条件的图像字幕生成。

使用Pytorch模型

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForQuestionAnswering

processor = BlipProcessor.from_pretrained("Salesforce/blip-vqa-capfilt-large ")
model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-capfilt-large ")

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

question = "how many dogs are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> 1

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForQuestionAnswering

processor = BlipProcessor.from_pretrained("Salesforce/blip-vqa-capfilt-large")
model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-capfilt-large").to("cuda")

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

question = "how many dogs are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt").to("cuda")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> 1

import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForQuestionAnswering

processor = BlipProcessor.from_pretrained("ybelkada/blip-vqa-capfilt-large")
model = BlipForQuestionAnswering.from_pretrained("ybelkada/blip-vqa-capfilt-large", torch_dtype=torch.float16).to("cuda")

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

question = "how many dogs are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt").to("cuda", torch.float16)

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> 1

BibTex和引用信息

@misc{https://doi.org/10.48550/arxiv.2201.12086,
  doi = {10.48550/ARXIV.2201.12086},
  
  url = {https://arxiv.org/abs/2201.12086},
  
  author = {Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
  
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {Creative Commons Attribution 4.0 International}
}