Learn / Search quality
How does multilingual RAG work?
Short answer
Multilingual RAG maps text from many languages into one shared meaning space, so a question in English can retrieve a passage written in Hindi, Arabic or Russian. It also needs OCR that reads non-Latin scripts and keyword search that understands each language.
With troveGEN
troveGEN is multilingual end to end in 18 verified languages: a shared meaning space, OCR for non-Latin scripts, per-script keyword search and right-to-left display.
The shared meaning space
A multilingual embedding model is trained so that sentences with the same meaning land near each other whatever the language. That makes cross-language retrieval possible: the question and the passage never need to share a word.
The other parts that must be multilingual
- OCR: scanned documents need recognition models for the scripts involved, such as Devanagari, Tamil or Arabic, and the OCR language has to be chosen to match.
- Keyword search: stemming and stop words differ by language, and some scripts have no spaces between words.
- Reranking: use a reranker that also handles the languages you need.
- Display: right-to-left languages must render correctly in answers and citations.
Which languages work well today
The honest test is whether a document written in a language can be read, split, indexed and found again with a question in that language, and with a question in English. troveGEN checks this on real documents rather than assuming it from the model's claimed coverage.
In that test, questions asked in the document's own language found the right document every time in 18 languages, across nine scripts. Questions asked in English found documents in other languages less consistently with the embedding model alone, and consistently once reranking was switched on.
- Verified: Hindi, Marathi, Tamil, Telugu, Bengali, Urdu, Arabic, Hebrew, Russian, Greek, Spanish, French, German, Polish, Turkish, Indonesian, Vietnamese and English.
- Not yet: Chinese, Japanese, Korean and Thai, because their words are not separated by spaces and keyword search needs a word splitter first.
- Scans: set the OCR language to match, otherwise a Hindi or Tamil page is read as English and comes out as nonsense.
Common pitfalls
Quality varies by language, and lower-resource languages are usually weaker. Mixed-language documents can confuse language detection. Scores from cross-language matches are often lower than same-language ones, so a threshold tuned in English may refuse good results. Test with real questions in each language you support.
Key takeaways
- A multilingual embedding model makes cross-language retrieval possible.
- OCR, keyword search, reranking and display all need language support too.
- Test every language you promise; quality is uneven.
How troveGEN helps with multilingual RAG
troveGEN uses a multilingual embedding model that understands about 100 languages, and has verified search end to end in 18: Hindi, Marathi, Tamil, Telugu, Bengali, Urdu, Arabic, Hebrew, Russian, Greek, Spanish, French, German, Polish, Turkish, Indonesian, Vietnamese and English. Keyword search is handled per script, OCR reads the language you set, and the console renders right-to-left text. Chinese, Japanese, Korean and Thai documents are not accepted yet. Cross-language questions are most reliable with reranking switched on. You can bring your own embedding model if you need a specialised one.
What troveGEN provides
- Cross-language search: ask in English, find answers in Hindi, Arabic, Russian and more (best with reranking)
- OCR for scanned pages in the language you set
- Keyword handling that respects each script
- Right-to-left rendering in answers and citations
- Bring your own embedding model if you need a specialised one
Frequently asked questions
Can I ask in English and get answers from Hindi documents?
Yes, with a multilingual embedding model. Results are best when documents are cleanly read, so scanned pages need good OCR.
Do I need a separate index per language?
Not with a multilingual model. One index can hold all languages, and you can filter by language metadata when you need to.
Which languages work best?
Widely used languages tend to perform best. Always evaluate on your own documents.
How does troveGEN help with multilingual RAG?
troveGEN is multilingual end to end in 18 verified languages: a shared meaning space, OCR for non-Latin scripts, per-script keyword search and right-to-left display. It provides: Cross-language search: ask in English, find answers in Hindi, Arabic, Russian and more (best with reranking); OCR for scanned pages in the language you set; Keyword handling that respects each script; Right-to-left rendering in answers and citations; Bring your own embedding model if you need a specialised one.