Lately, I’ve been spending quite a lot of time working with vector embeddings and Cosmos DB Vector Search. Since I live in Finland, I work a lot with information in another language than English. That’s why I set out to compare a handful of the vector embedding models available in Azure AI Foundry.

Comparison Focus

As I wrote in a previous article, embedding models can be language agnostic. In this comparison, I also wanted to get some kind of understanding of how language agnostic they can. I also wanted to see whether there is any difference between various embedding models.

Another area I wanted to focus on was to see how good different embedding models are to capture the semantic context of an input instead of literally capturing the meaning. What I mean by this is that I wanted to check how similar input is when it’s written differently, but essentially means the same thing. So for instance “the weather is nice” should be quite similar to “enjoyable weather” or “the sun is shining from a clear blue sky”.

Vector Embedding Model List

In this comparison, I used the following embedding models available in Azure AI Foundry.

• embed-v-4-0
• Cohere-embed-v3-english
• Cohere-embed-v3-multilingual
• text-embedding-3-large
• text-embedding-3-small
• text-embedding-ada-002

Test Rig for Vector Embedding Model Comparison

I created a simple console application for running my comparisons. It takes two strings, creates a vector for both strings using the embedding models I included in the comparison, and then calculates a similarity score for the produced vectors. The similarity score is calculated using the following function that calculates the similarity using cosine similarity.

static double CosineSimilarity(float[] v1, float[] v2)
{
    if (v1.Length != v2.Length)
        throw new ArgumentException("Vectors must be of same length.");

    double dot = 0.0, mag1 = 0.0, mag2 = 0.0;
    for (int i = 0; i < v1.Length; i++)
    {
        dot += v1[i] * v2[i];
        mag1 += v1[i] * v1[i];
        mag2 += v2[i] * v2[i];
    }
    if (mag1 == 0 || mag2 == 0)
        return 0; // Avoid division by zero

    return dot / (Math.Sqrt(mag1) * Math.Sqrt(mag2));
}

Vector Embedding Model Comparison

So let’s start testing. I’ve added each test case as a sub chapter. Each test case defines the two texts used, and a table representing the similarity produced by the embedding models included in this comparison.

The similarity score using cosine similarity metrics can vary between -1 (least similar) to +1 (most similar). In the tables, similarity score is rounded to 2 decimals.

Test #1

Let’s start with a very simple test with two short, very similar texts.

1. Apples are red. Bananas are yellow.
2. Apples have a red color. Bananas are colored yellow.

Embedding Model	Similarity Score
embed-v-4-0	0.86
Cohere-embed-v3-english	0.97
Cohere-embed-v3-multilingual	0.97
text-embedding-3-large	0.87
text-embedding-3-small	0.91
text-embedding-ada-002	0.97

Test #2

Now let’s use the same text from the previous test, but translate the other text into Finnish, as precisely as possible.

1. Apples are red. Bananas are yellow.
2. Omenat ovat punaisia. Banaanit ovat keltaisia.

Embedding Model	Similarity Score
embed-v-4-0	0.53
Cohere-embed-v3-english	0.37
Cohere-embed-v3-multilingual	0.66
text-embedding-3-large	0.57
text-embedding-3-small	0.49
text-embedding-ada-002	0.84

Test #3

Let’s move on to describing the sun setting.

1. The sun sets beautifully over the calm ocean.
2. The sun descends gracefully above the tranquil sea.

Embedding Model	Similarity Score
embed-v-4-0	0.80
Cohere-embed-v3-english	0.82
Cohere-embed-v3-multilingual	0.92
text-embedding-3-large	0.76
text-embedding-3-small	0.78
text-embedding-ada-002	0.96

Test #4

Now let’s do the same as in the previous test, but translate the first language into Swedish.

1. The sun sets beautifully over the calm ocean.
2. Solen går vackert ner över det lugna havet.

Embedding Model	Similarity Score
embed-v-4-0	0.68
Cohere-embed-v3-english	0.26
Cohere-embed-v3-multilingual	0.81
text-embedding-3-large	0.63
text-embedding-3-small	0.61
text-embedding-ada-002	0.87

Test #5

Now we move on to a bit longer texts. First, two texts in English written in different ways, still keeping the semantic meaning intact.

1. In the south-western Finnish archipelago, a summer sunset paints the sky in hues of gold, pink, and deep orange, reflecting over the calm sea. The warm evening air carries the scent of saltwater and blooming wildflowers, while gentle waves lap against the rocky shoreline. As the sun dips below the horizon, the archipelago’s countless small islands cast long, soft shadows over the tranquil waters, creating a breathtakingly serene scene.
2. In the south-western archipelago of Finland, a summer evening sky is illuminated with shades of gold, pink, and rich orange, mirrored on the still surface of the sea. The mild night air is filled with the aroma of sea spray and blossoming wildflowers, as soft waves brush against the rocky coast. When the sun sinks beneath the horizon, the many tiny islands of the archipelago stretch their gentle shadows across the peaceful waters, forming an extraordinarily calm and beautiful landscape.

Embedding Model	Similarity Score
embed-v-4-0	0.86
Cohere-embed-v3-english	0.94
Cohere-embed-v3-multilingual	0.94
text-embedding-3-large	0.95
text-embedding-3-small	0.92
text-embedding-ada-002	0.98

Test #6

Now we take the text from the previous test and translate it to Finnish.

1. In the south-western Finnish archipelago, a summer sunset paints the sky in hues of gold, pink, and deep orange, reflecting over the calm sea. The warm evening air carries the scent of saltwater and blooming wildflowers, while gentle waves lap against the rocky shoreline. As the sun dips below the horizon, the archipelago’s countless small islands cast long, soft shadows over the tranquil waters, creating a breathtakingly serene scene.
2. Kesäinen auringonlasku Lounais-Suomen saaristossa maalaa taivaan kultaisiin, pinkkeihin ja syvänoransseihin sävyihin, jotka heijastuvat tyyneen mereen. Lämmin iltailma kantaa mukanaan suolaveden ja kukkivien luonnonkukkien tuoksua, kun lempeät aallot huuhtovat kivikkoista rantaa. Auringon vajotessa horisontin taakse lukuisat pikkusaaret heittävät pitkiä, pehmeitä varjoja rauhallisen veden ylle, luoden henkeäsalpaavan seesteisen maiseman.

Embedding Model	Similarity Score
embed-v-4-0	0.34
Cohere-embed-v3-english	0.41
Cohere-embed-v3-multilingual	0.74
text-embedding-3-large	0.74
text-embedding-3-small	0.66
text-embedding-ada-002	0.91

Test #7

Let’s twist up things a little more and take the Finnish text from the previous test and translate that into Swedish.

1. Kesäinen auringonlasku Lounais-Suomen saaristossa maalaa taivaan kultaisiin, pinkkeihin ja syvänoransseihin sävyihin, jotka heijastuvat tyyneen mereen. Lämmin iltailma kantaa mukanaan suolaveden ja kukkivien luonnonkukkien tuoksua, kun lempeät aallot huuhtovat kivikkoista rantaa. Auringon vajotessa horisontin taakse lukuisat pikkusaaret heittävät pitkiä, pehmeitä varjoja rauhallisen veden ylle, luoden henkeäsalpaavan seesteisen maiseman.
2. En somrig solnedgång i sydvästra Finlands skärgård målar himlen i gyllene, rosa och djupt orange nyanser som speglas i det stilla havet. Den varma kvällsluften bär med sig doften av saltvatten och blommande vilda blommor, medan mjuka vågor sköljer mot den steniga stranden. När solen sjunker bakom horisonten kastar de otaliga små öarna långa, mjuka skuggor över det lugna vattnet och skapar ett andlöst fridfullt landskap.

Embedding Model	Similarity Score
embed-v-4-0	0.41
Cohere-embed-v3-english	0.49
Cohere-embed-v3-multilingual	0.75
text-embedding-3-large	0.83
text-embedding-3-small	0.66
text-embedding-ada-002	0.90

Test #8

Now, let’s twist this up even further. We create two texts in different language, and try to make them as different as possible. The first text is the same text we’ve used in the previous tests. The second text is a product description for the Kalevala Kosmos Pedant in bronze in Finnish.

1. In the south-western Finnish archipelago, a summer sunset paints the sky in hues of gold, pink, and deep orange, reflecting over the calm sea. The warm evening air carries the scent of saltwater and blooming wildflowers, while gentle waves lap against the rocky shoreline. As the sun dips below the horizon, the archipelago’s countless small islands cast long, soft shadows over the tranquil waters, creating a breathtakingly serene scene.
2. Kosmoksen kirkas pronssi hehkuu lämpimissä sävyissä. Kullanhohtoisen pronssin ja mustan tai turkoosin korukiven yhdistelmä luo näyttäviin Kosmos-koruihin jännitettä ja dramatiikkaa. Papuketjussa säädettävä pituus 45/42 cm. Papukaijalukko. Jos ketju on sinulle liian lyhyt, saat siihen kätevästi lisää mittaa erikseen myytävällä jatkopalalla. Jatkopala kiinnitetään siinä olevalla papukaijalukolla korun ketjuun jatkeeksi. Jatkopalan pituus 5 cm.

Embedding Model	Similarity Score
embed-v-4-0	0.21
Cohere-embed-v3-english	0.35
Cohere-embed-v3-multilingual	0.33
text-embedding-3-large	0.17
text-embedding-3-small	0.19
text-embedding-ada-002	0.79

Test #9

Let’s do one final test. Use two texts in English and try to make them as different as possible.

1. The impressive Kosmos pendant is embellished with a gleaming black onyx or a tranquil turquoise. Made of bright bronze, the pendant glows like gold and is a true eye-catcher. Adjustable length 42/45 cm. Lobster clasp. If the chain is too short for you, you can easily add length to it with a 5 cm extension piece sold separately.
2. To be, or not to be, that is the question: Whether ’tis nobler in the mind to suffer the slings and arrows of outrageous fortune, or to take arms against a sea of troubles and by opposing end them.

Embedding Model	Similarity Score
embed-v-4-0	0.19
Cohere-embed-v3-english	0.10
Cohere-embed-v3-multilingual	0.27
text-embedding-3-large	0.07
text-embedding-3-small	0.07
text-embedding-ada-002	0.71

Summary

At first I was surprised how good the text-embedding-ada-002 was performing to find similarity between different texts. However, in the last two tests, where the two texts are very different, that model still thought the texts were quite similar. Because of this, I would not use the text-embedding-ada-002 model for anything.

The second observation is that the text-embedding-3-large model performs as well, or even better than the Cohere-embed-v3-multilingual model, except for with very short input. However, when comparing vectors for short text in different languages, the Cohere-embed-v3-multilingual model is not performing significantly better than for instance text-embedding-3-large, or even the text-embedding-3-small model.

So which one should you pick for your purpose? As with many architectural decisions – It depends, and I would say it depends on whether your inputs (the texts that you vectorize) are all in the same language, or if you need to be able to find similarities (or differences) across different languages.

If your inputs are all in the same language, the text-embedding-3-large seems to perform quite well. The text-embedding-3-small works also well, and is a much cheaper option.

But, if you are processing inputs in different languages, and processing texts that are anything than short sentences, I would go with either the text-embedding-3-large or Cohere-embed-v3-multilingual model. Since the cost for the Cohere-embed-v3-multilingual is $0.0001 / 1000 tokens or roughly 0.000093 EUR / 1000 tokens, it is about the same as text-embedding-3-large, which is around 0.000112 EUR / 1000 tokens.

To make those figures a bit more tangible, the cost for 10 000 000 tokens for each of these two models is:

• text-embedding-3-large: 1.12 EUR / 10 000 000 tokens
• Cohere-embed-v3-multilingual: 0.93 EUR / 10 000 000 tokens

Because the text-embedding-3-large embedding model is better at recognizing differences and on par with the Cohere-embed-v3-multilingual model, I would go with the text-embedding-3-large model in multi-lingual scenarios, even if that model is just a tad more expensive. The difference is still quite negligible.