Keyword lists

What are Keyword lists?

Keyword lists are corpora that contain key terminology entries. You can use keyword lists to dynamically change the translations of words/terms based on the entries in the lists. Ensure that only real keywords are used and avoid generic words and words that may have different meanings.

What format can be used for a keyword list?

You can use corpora in CSV, TBX, and TSV format (UTF-8 encoding). The keyword list can contain words or phrases.

Format

Globalese reads the older TBX 2.x / TBX-Basic / MARTIF style (ISO 30042:2008). Expected structure:

<martif> → <text> → <body> → <termEntry> → <langSet xml:lang=".."> → <tig> → <term>

The newer TBX v3 structure (<tbx>, <langSec>, <termSec>, ISO 30042:2019) isn't supported. The parser only looks for <termEntry> elements and doesn't check the root element or the type attribute.

Languages

The file may contain more than two languages, but only two are used:

• The first and second <langSet> element of the first <termEntry> become the source and target language.

• In every later <termEntry>, only <langSet>s in those two languages are read; other languages are ignored.

• Language codes are normalised (e.g. "en-US" and "en" are treated as the same).

• If the first entry has fewer than two language sets, the import fails.

Fields

Only these are used:

• xml:lang on <langSet>: the language

• <tig><term>: the term text (only the first tig/term per language; synonyms are dropped)

• id on <termEntry>: optional, stored as the pair ID

Entries missing a <tig> or <term> in either of the two languages are skipped. All other fields (<descrip>, <note>, <admin>, <transacGrp>, part of speech, definition, subject field, etc.) are ignored without error. Inline markup inside <term> is stripped and whitespace is normalised.

Sample

<?xml version="1.0" encoding="UTF-8"?>
<martif type="TBX" xml:lang="en">
<text><body>
<termEntry id="1">
<langSet xml:lang="en"><tig><term>Dog</term></tig></langSet>
<langSet xml:lang="hu"><tig><term>Kutya</term></tig></langSet>
</termEntry>
</body></text>
</martif>

What happens in the case of changes or updates to keyword lists?

Keyword lists can be changed and updated when updating an engine. If the only change made to an engine is in the keyword list(s) and/or the custom prompt, no retraining is required. The changes will take effect immediately once the engine has been saved. Please note that in this scenario, a new version of the engine will be created.

What kind of engines are supported?

Currently, keyword lists can be used in domain-adapted, AI-boosted engines.

Recommendations

The purpose of the keyword list is to change the priority of translations for a specific term or phrase, even if it appears differently in the training data.

However, you should avoid adding large, complex term bases or terminology lists. You should also avoid using homonyms or very basic words.

The best sources for terminology, especially in the case of homonyms and basic words, are still clean and consistent master corpora. Keyword lists should only be used to fine-tune translation results for critical terms and phrases.