{"id":10132,"date":"2021-12-02T10:24:09","date_gmt":"2021-12-02T08:24:09","guid":{"rendered":"https:\/\/atostek.com\/?p=10132"},"modified":"2021-12-02T14:03:48","modified_gmt":"2021-12-02T12:03:48","slug":"computationally-clarifying-user-intent-for-improved-question-answering","status":"publish","type":"post","link":"https:\/\/atostek.com\/en\/computationally-clarifying-user-intent-for-improved-question-answering\/","title":{"rendered":"Computationally clarifying user intent for improved question answering"},"content":{"rendered":"

A common drawback of chatbots is the impractical handling of unrecognized user inputs. These situations force the system to return a response along the lines of \u201csorry, could you rephrase that?\u201d and do not move the conversation forward in any meaningful way. When in conversation with humans we do not end up in these situations nearly as often as we can use our domain knowledge to infer the meaning even if we wouldn\u2019t have a perfect understanding of the context.<\/p>\n

In my master\u2019s thesis, I set out to explore if domain knowledge could be added to modern chatbots, allowing more humanlike interactions and improved performance in terms of giving more relevant answers.<\/p>\n

This thesis is a part of an ongoing project at Atostek, titled Oxilate<\/strong>. As a part of the project, we are developing a digital assistant for industrial product and maintenance needs. The goal is to produce an assistant for various industrial settings capable of referencing the user with the right parts of the source material and answering questions about the data.<\/p>\n

This bot is built using the Rasa framework, an open-source framework for developing dialogue systems. Rasa allows easy integrations to other software and high configurability and thus was the technology of choice for this project.<\/p>\n

The current version of the project is trained to answer users\u2019 questions about Ubuntu operating system, and the source materials for the chatbot include Ubuntu\u2019s official documentation, Ubuntu launchpad forum conversations, and Ubuntu community help wiki pages. Now the task is to add domain knowledge to this system and allow the system to query this data when it can\u2019t link the user\u2019s input to anything it knows.<\/p>\n

Ontologies as a source of domain knowledge<\/h2>\n

Before adding domain knowledge, we need to find a way to represent a domain appropriately. This is where ontologies come to play. They are presentations of different domains or languages, where entities and their relationships are presented as interconnected nodes.<\/p>\n

The figure below shows a simplified example of what an ontology can look like. The circles represent different entities in the context, and the lines represent different relationships. In a more complex domain, there can and should be multiple different types of relationships between entities to fully capture the domain.\"\"<\/a>In the context of this work, an ontology detailing the Ubuntu domain was needed. A perfect match was not readily available, and thus Computer Science Ontology (CSO) was chosen. It is an automatically generated ontology detailing the whole field of computer science. Generated using an algorithm that combines machine learning and semantic technologies, some relationships are also reviewed by experts. CSO contains multiple different kinds of relationships, but for this use case we are mostly interested in relationships that define synonyms or other closely related entities.<\/p>\n

Fitting the ontology to the chatbot<\/h2>\n

Adding the domain knowledge offered by the CSO to our chatbot was relatively straightforward thanks to Rasa\u2019s extendability. A custom module for handling low-confidence inputs was built, which is activated when the input falls below a fallback threshold set in Rasa. This module then takes the user\u2019s input, extracts the relevant entities from it, and queries the ontology with these entities. The result terms are fed back into the answer generating model running in the background, and if it is able to retrieve meaningful results, those are shown to the user along with a fitting message (e.g. did you mean to ask about \u201cBluetooth\u201d?).\"\"<\/a><\/p>\n

Evaluating the model & results<\/h2>\n

Because of the nature of the implementation – it only activates when the baseline model fails – and the already poor universal metrics for chatbot evaluation, the implementation was evaluated by organizing a user study of 20 people. These testers first evaluated the baseline model by interacting with it and answering a set of questions. Then this group asked the same questions from the model using the custom implementation and answered the same questionnaire.<\/p>\n

The results of these tests can be summarized as follows: the custom module is able to produce recommendations, but they are not relevant enough to be useful, although they are slightly more useful than the baseline\u2019s responses. The cause for this lies within the ontology: CSO, although a large ontology detailing the field of computer science, does not contain enough information about Ubuntu or other operating systems to produce relevant recommendations in this domain.<\/p>\n

Although the results show that the domain of the chatbot is not captured by the ontology used, the implementation can provide the users with recommendations even when the ontology does not match the domain but the quality is questionable. Further research is required to find out the true potential of the implementation, such as finding a domain specific ontology for a use case, more complex utilization of different relationships in the ontology, and developing a heuristic for choosing the best matching answer out of the found ontology results.<\/p>\n

For now the problem with poor domain understanding in chatbots still persists, but combining more sophisticated machine learning models with better fitting ontologies might be able to conquer this in the future.<\/p>\n

\n
\n

Ari-Pekka H\u00e4rk\u00f6nen<\/strong>
\nSoftware designer<\/em><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

A common drawback of chatbots is the impractical handling of unrecognized user inputs. These situations force the system to return a response along the lines of \u201csorry, could you rephrase that?\u201d and do not move the conversation forward in any meaningful way. When in conversation with humans we do not end up in these situations…<\/p>\n","protected":false},"author":24,"featured_media":10129,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[114],"tags":[385,387,388,243,206,386],"class_list":["post-10132","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","tag-chatbot-en","tag-computer-science-ontology-en","tag-masters-thesis-en-2","tag-opinnaytetyo-en","tag-oxilate-en","tag-rasa-en","entry","has-media"],"_links":{"self":[{"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/posts\/10132"}],"collection":[{"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/comments?post=10132"}],"version-history":[{"count":0,"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/posts\/10132\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/media\/10129"}],"wp:attachment":[{"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/media?parent=10132"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/categories?post=10132"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/atostek.com\/en\/wp-json\/wp\/v2\/tags?post=10132"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}