HindSBERT-STS

这是在STS数据集上微调的HindSBERT模型（ l3cube-pune/hindi-sentence-bert-nli ）。作为MahaNLP项目的一部分发布： https://github.com/l3cube-pune/MarathiNLP 在此处共享支持主要印度语言和跨语言句子相似性的多语言版本的模型 indic-sentence-similarity-sbert

关于数据集、模型和基准结果的更多详细信息可以在我们的[论文]（ https://arxiv.org/abs/2211.11187 ）中找到

@article{joshi2022l3cubemahasbert,
  title={L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi},
  author={Joshi, Ananya and Kajale, Aditi and Gadre, Janhavi and Deode, Samruddhi and Joshi, Raviraj},
  journal={arXiv preprint arXiv:2211.11187},
  year={2022}
}

monolingual Indic SBERT paper multilingual Indic SBERT paper

下面列出了其他单语相似性模型： Marathi Similarity Hindi Similarity Kannada Similarity Telugu Similarity Malayalam Similarity Tamil Similarity Gujarati Similarity Oriya Similarity Bengali Similarity Punjabi Similarity Indic Similarity (multilingual)

下面列出了其他单语Indic句子BERT模型： Marathi SBERT Hindi SBERT Kannada SBERT Telugu SBERT Malayalam SBERT Tamil SBERT Gujarati SBERT Oriya SBERT Bengali SBERT Punjabi SBERT Indic SBERT (multilingual)

这是一个 sentence-transformers 模型：它将句子和段落映射到一个768维稠密向量空间，并可用于聚类或语义搜索等任务。

使用（Sentence-Transformers）

当您安装了 sentence-transformers 之后，使用这个模型变得很容易：

pip install -U sentence-transformers

然后您可以像这样使用模型：

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('{MODEL_NAME}')
embeddings = model.encode(sentences)
print(embeddings)

使用（HuggingFace Transformers）

如果没有 sentence-transformers ，您可以像这样使用模型：首先，将输入通过变换器模型，然后必须在上下文化的词嵌入之上应用正确的汇集操作。

from transformers import AutoTokenizer, AutoModel
import torch


#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
model = AutoModel.from_pretrained('{MODEL_NAME}')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

作者:

L3Cube

数据集大小:

915.41 MB