Localization in MarkLogic - localization

Question

I'm having trouble with localization in MarkLogic (testing on 7.0-1) and wildcard searches.
Example:
let $x :=
<root>
<el xml:lang="en">hello</el>
<el xml:lang="fr">hello</el>
</root>
return
$x//el[cts:contains(., cts:word-query("hello*", ("wildcarded", "lang=fr")))]
Why does it return both elements el and not only one with xml:lang="fr"? When I remove asterisk from "hello*" it returns just one element as expected.
How to use localization in wildcard searches?

Answer 1

MarkLogic uses language-dependent indexes for stemmed searches, but not for unstemmed searches. And wildcarded searches are performed against the language-independent unstemmed indexes unfortunately.
The section 'Language-Aware Searches' of the Search Dev Guide explains how language-awareness works in MarkLogic and states:
All searches use the language setting in the cts:query constructor to determine how to tokenize the search terms. Stemmed searches also use the language setting to derive stems. Unstemmed searches use the specified language for tokenization but use the unstemmed (word searches) indexes, which are language-independent.
And the section 'Interaction with Other Search Features' directly relates wildcarded and stemmed searches, and states:
The system will not perform a stemmed search on words that are wildcarded.
I think you have two options basically:
You can either filter manually afterwards, but that would likely result in too high estimates because of false positives from wrong languages.
Alternatively, you could use a word-lexicon to lookup explicit values, and pass that as a sequence to your cts:word-query.
Something like:
let $x :=
<root>
<el xml:lang="en">hello</el>
<el xml:lang="fr">hello</el>
</root>
return
$x//el[cts:contains(., cts:word-query(cts:words("hell"), ("lang=fr")))]
Note that the latter does require you to enable a word lexicon, and values returned by cts:words are fed from documents in the database.
HTH!

Answer 2

The supported languages for MarkLogic are:
English
French
Italian
German
Russian
Spanish
Arabic
Chinese (Simplified and Traditional)
Korean
Persian (Farsi)
Dutch
Japanese
Portuguese
Norwegian (Nynorsk and Bokmål)
Swedish
With these languages, stemmed searches and indexes work as you would expect.