Tokenization Configuration

Tokenization (also known as autolinking) sets object names as keywords (= tokens). When these keywords appear in the content of other objects (such as descriptions or summaries), a relation is created between them automatically.

Tokenization is especially useful for applications like the Business Glossary as definitions or terms are frequently referenced across different data objects.

Setting up tokenization involves the following steps:

1. Enable tokenization for your space or application.
   • Optional: Specify which spaces or applications the tokens should originate from.
2. Select object type attributes that will scan text content and detect tokens.
   • Optional: Specify which additional attributes should act as tokens themselves.
3. Create relation types that will be used for autolinking and assign them to relevant object types.

1. Enable Tokens in Application

In your application’s settings, enable tokenization using the following template:

"applications": [
    {
        "key": "app",
        ...
        "settings": [
            {
                "key": "has_token_keywords",
                "value": true
            },
            {
                "key": "has_token_texts",
                "value": true
            },
            {
                "key": "token_keyword_spaces",
                "value": [ ]
            },
            {
                "key": "token_keyword_apps",
                "value": [ ]
            }
        ]
    }
],

Setting	Purpose
has_token_keywords	Allows object types in this application to be set as keywords.
has_token_texts	Allows object type attributes in this application to identify and link keyword mentions.
token_keyword_spaces	[Optional] Specifies which spaces the keywords originate from. Provide the space’s ID, which can be found in your instance’s URL after space/ (e.g., in the https://example.dawiso.cloud/.../space/123/... URL, the space ID is 123).
token_keyword_apps	[Optional] Specifies what apps the keywords originate from. Provide the app’s ID, which can be found in your instance’s URL after app/ (e.g., in the https://example.dawiso.cloud/.../app/345/... URL, the app ID is 345).

2. Allow Attribute Types to Mention Tokens

Next, decide which attribute types can detect keywords and link them. To do so, add the is_token_text feature to your attribute type(s).

"attributeTypes": [
  {
    "key": "",
    ...
    "features": [
      {
        "key": "is_token_text",
        "value": true
      }
    ]
  },
  ...
],

Optional: Assign Attributes as Tokens

The is_token_keyword attribute feature allows you to assign specific attributes as keywords. This is particularly useful when an object has an attribute with alternative names. Any text in this attribute will become a keyword linking to its object.

For example, an Alternative Name attribute will be configured like this:

"attributeTypes": [
  {
    "key": "alternative_name",
    "name": "Alternative Name",
    "description": "Alternative Name for Tokenization purpose",
    "features": [
      {
        "key": "is_token_keyword"
      }
    ]
  },
  ...
],

3. Create Relations That Generate Keywords

Finally, to specify which object types become related when a keyword is detected, we need to:

• Create the relation type which is generated by keywords.
• Specify between which object types this relation can be created.

In our example, we will create the is mentioned and is mentioned by relation types. The most important part is the is_generated_by_keyword relation type feature, which links the two object types when a token is found in one object’s attributes.

"relationTypes": [
  {
    "key": "isMentioning",
    "name": "is mentioning",
    "description": "Relation from Text to Keyword",
    "reverseKey": "isMentionedBy",
    "isOriented": true,
    "features": [
      {
        "key": "is_generated_by_keyword",
        "value": true
      }
    ]
  },
  {
    "key": "isMentionedBy",
    "name": "is mentioned by",
    "description": "Relation from Keyword to Text",
    "reverseKey": "isMentioning",
    "isOriented": true
  }
],

Create this relation between your object types. In our example, recipes can mention ingredients.

"objectTypeRelations": [
   {
       "fromObjectTypeKey": "recipe",
       "toObjectTypeKey": "ingredient",
       "relationTypeKey": "isMentioning"
   }
],

The reverse relation is already defined using the reverse key, which is why the opposite relation is created automatically.