Bg2Vec

Bg2Vec is a Bulgarian text encoder. It is BgGPT-7B-Instruct-v0.2 after the LLM2Vec procedure:

1. Bidirectional attention. Each token attends to all tokens, before and after it.
2. Masked next token prediction (MNTP). LoRA fine-tuning on Bulgarian Wikipedia. The model recovers masked tokens from the state of the previous token, so it learns to use the new attention.
3. Unsupervised SimCSE. LoRA fine-tuning on Bulgarian Wikipedia. Two dropout masks give two vectors for the same passage. The model learns to make them similar, and to make them different from the other passages in the batch.

Both LoRA adapters are merged into the weights in this repository. To get a text vector, use bidirectional attention and take the mean of the last hidden states.

• Developed by: Martin Boyanov
• Model type: Mistral-7B decoder used as a bidirectional encoder, 7.1B parameters, 4096-dimension vectors
• Language: Bulgarian
• License: Apache 2.0 (same as the base model)
• Fine-tuned from: INSAIT-Institute/BgGPT-7B-Instruct-v0.2
• Training code: github.com/mboyanov/bg2vec
• Blog series: Let's build Bg2Vec, in five parts
• Demo: Bg2Vec search, BgGPT before and after LLM2Vec on Bulgarian Wikipedia search

How to use

With plain transformers

You do not need the llm2vec package. At inference time, LLM2Vec is two things: attention that is not causal, and mean pooling.

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

name = "mboyanov/bg2vec"
tokenizer = AutoTokenizer.from_pretrained(name)
tokenizer.padding_side = "right"
model = AutoModel.from_pretrained(name, dtype=torch.bfloat16, attn_implementation="sdpa").eval()
for layer in model.layers:
    layer.self_attn.is_causal = False  # a guard; the 4D mask below does the real work


@torch.no_grad()
def encode(texts: list[str]) -> torch.Tensor:
    batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt").to(model.device)
    mask = batch["attention_mask"]
    # transformers uses a 4D mask as it is. This mask hides only the padding,
    # so every token attends to every other token.
    attention = mask[:, None, None, :].bool()
    hidden = model(input_ids=batch["input_ids"], attention_mask=attention).last_hidden_state.float()
    weights = mask.unsqueeze(-1).float()
    vectors = (hidden * weights).sum(dim=1) / weights.sum(dim=1)  # mean over real tokens
    return F.normalize(vectors, dim=-1)


original, positive, negative = encode([
    "Тя обича да пътува през лятото, особено до студени дестинации, като избягва горещи и пренаселени места",
    "Тя обича да пътува през лятото, специално към хладни места, избягвайки топли и многолюдни райони",
    "Тя обича да пътува през лятото, но предпочита да посещава горещи и оживени туристически места",
])
print((original @ positive).item(), (original @ negative).item())
# ≈ 0.787 0.728: the positive is closer than the negative

We tested this code with transformers 5.22. Its vectors agree with the llm2vec package (cosine similarity > 0.999).

Do not add a query prefix or an instruction. The model was trained without them.

With llm2vec

from llm2vec import LLM2Vec

model = LLM2Vec.from_pretrained("mboyanov/bg2vec")
model.encode(["Това е примерно изречение."])

The llm2vec package needs an older transformers (4.44 or lower).

Training

The training follows the LLM2Vec paper for Mistral-7B. Each step trains a LoRA adapter (r=16, alpha=32, dropout 0.05) on all attention and MLP projections: 0.58% of the parameters. Both steps run for 1000 steps in bf16.

Data

mboyanov/bgwiki: a dump of Bulgarian Wikipedia (May 2024), with text extracted by wikiextractor and empty redirect pages removed.

Step 1: MNTP

• Data: the articles are tokenized with the BgGPT tokenizer, joined, and cut into chunks of 512 tokens. At most 50,000 chunks are used for training.
• Masking: all_mask (as the paper uses for Mistral), mask probability 0.8, _ as the mask token.
• Batch size 32.
• Adapter: mboyanov/bggpt-mntp. The validation loss goes down from 5.53 at step 100 to 4.89 at step 1000.

Step 2: SimCSE

• Start point: BgGPT with the MNTP adapter merged in, bidirectional attention.
• Data: each article is a pair with itself. Texts are truncated to 128 tokens.
• Loss: HardNegativeNLLLoss from llm2vec, with in-batch negatives, scale 20.
• Batch size 128, learning rate 3e-5, attention dropout 0.3, mean pooling.

The full configurations are in model_configurations/.

Hardware

One NVIDIA A100 80GB or A6000 48GB GPU on Paperspace Gradient.

Evaluation

All the results come from Part 5 of the blog series.

MTEB, Bulgarian tasks

The table shows the MTEB main score of each task. BgGPT uses mean pooling with its original causal attention.

Task	Bg2Vec	BgGPT-7B-Instruct-v0.2	bge-multilingual-gemma2
BulgarianStoreReviewSentimentClassfication	0.609	0.570	0.696
LanguageClassification	0.983	0.992	0.920
MultilingualSentimentClassification	0.810	0.720	0.905
SIB200Classification	0.718	0.758	0.769
SIB200ClusteringS2S	0.262	0.319	0.391
BelebeleRetrieval	0.910	0.348	0.764
WikipediaRetrievalMultilingual	0.780	0.392	0.689
XNLI	0.621	0.578	0.792
WikipediaRerankingMultilingual	0.752	0.656	0.813

Bg2Vec is better than BgGPT on 6 of the 9 tasks. The largest gain is on retrieval: BelebeleRetrieval goes from 0.348 to 0.910. Bg2Vec has the best retrieval scores of the three models. bge-multilingual-gemma2 has the best score on 6 tasks, and BgGPT on LanguageClassification. MultiEURLEXMultilabelClassification is not in the table, because it used too much memory.

Hard triplets

A set of sentence triplets, translated into Bulgarian from the Echo Embeddings paper. The three sentences in a triplet start with the same words. The first two sentences have the same meaning, and the third sentence has a different meaning. The score is the mean of cos(original, positive) - cos(original, negative). A higher score is better.

Model	Score
Bg2Vec	0.236
bge-multilingual-gemma2	0.164
e5-mistral-7b-instruct	0.133
multilingual-e5-large	0.039
BgGPT-7B-Instruct-v0.2	0.033
gte-Qwen2-7B-instruct	0.026

Demo queries

The demo Space searches 29,912 passages of Bulgarian Wikipedia. Each of its 10 showcase queries has one target article. The MRR@10 of the target article is 0.777 for Bg2Vec, 0.725 for multilingual-e5-large, 0.283 for BM25 and 0.025 for BgGPT. Ten queries are a showcase, not a benchmark.

Limitations

• Size. The model has 7.1B parameters (14 GB in bf16) and gives 4096-dimension vectors. On one L40S, it encodes approximately 48 passages per second at 256 tokens or less. A smaller multilingual encoder can be a better choice for many tasks.
• Text length. MNTP used chunks of 512 tokens and SimCSE used texts of 128 tokens. We did not test texts longer than 512 tokens.
• No text generation. The repository has no language-model head. Use the model only to encode text.
• Unsupervised training. The training used only Wikipedia text, with no labelled pairs and no instructions. The model is weaker on clustering and topic classification than BgGPT, and weaker than bge-multilingual-gemma2 on most tasks that are not retrieval.
• Rare tokens. The model can miss exact matches on codes and rare terms (for example, "ISO 8601"). For such queries, use it together with lexical search such as BM25.
• Bias. The model can have the biases of Bulgarian Wikipedia and of BgGPT.

Citation

@misc{boyanov2024bg2vec,
  author = {Martin Boyanov},
  title  = {Let's Build Bg2Vec: Apply LLM2Vec on BgGPT},
  year   = {2024},
  url    = {https://mboyanov.github.io/2024/05/18/Bg2Vec.html}
}

@article{behnamghader2024llm2vec,
  title   = {LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders},
  author  = {BehnamGhader, Parishad and Adlakha, Vaibhav and Mosbach, Marius and Bahdanau, Dzmitry and Chapados, Nicolas and Reddy, Siva},
  journal = {arXiv preprint arXiv:2404.05961},
  year    = {2024}
}