Graphical User Interface (GUI) Interface

An application that provides a more user friendly interface than the traditional command line interface.


DEPLOYMENT REQUIREMENTS

      - Java version 1.7 or higher

      - 1248 x 1024 screen resolution


WINDOWS: decompress the aseryla_w.zip in "C:\Users\Public\Aseryla" (you can decompress in any other folder, but you have to adapt the route of each .cmd file).


LINUX: The GUI is distributed with the latest Aseryla server version.

Requisites:

      - QT runtime libraries v5. Newest Ubuntu distros has already included, otherwise [sudo apt-get install libqt5*]

      - Give executation permissions to any ".exe" and ".sh" file in the deployment folder.

      - Optionally, but very recommended (or you couldn't provide new knowledge to the system), the socket version of Stanford core NLP.

           Required at least the version 1.7 of Java [java -version] + [sudo apt-get install default-jre]

      - For running the optional features:

      Python version 2.7 (already included in the Ubuntu 14.0.4 LTS and higher distros) and the package Python installer pip [sudo apt-get install python-pip]

           * Spell Checker (Feed & Verify tabs): PyEnchant [sudo pip install pyenchant] and TkInter [sudo apt-get install python-tk] libraries

           * Ontology Graph Visualizer (Explore tab): requires the libraries networkx [sudo pip install network] and matplotlib [sudo pip install matlibplot].






THE GUI (USER MANUAL)

This is interface pretending to provide the user a natural and intuitive way of interact with the features the Aseryla project provides.

Using the graphical interface you can profit all the characteristics the aseryla server provides (except process directly Internal Language [orders Tc and Ti ]),

such as asking questions, check the existent knowledge and to teach to the system providing sentences.


The design is centered in tabs, which splits and simplifies the main tasks:

On the upper left position of each tab, appears a help icon that open an specific help for the correspondent task; it's a link to this web page, that will opens the default web browser and requires an internet connection


      It provides a common mechanism shared by each tab for start the Aseryla server and even the NLP server (if they aren't running yet).


      If the Aseryla server is not running or not listening a the same defined port, the application will show an OFF state and remains disabled.


Pressing the "Start" button, appears the following window:

where you can indicate the location of a executable file that start the Aseryla server, and their parameters, in the indicated port.

It also offers the option of start/stop the service when the application is raised/shutdown.

After pressing the "Launch" button, if everything goes fine, the parameters are saved (for the next time you want to run again), the server is started, and the application goes to an enable state, where you can interact.

The distributable provides some .cmd/.sh with the default configuration, that allow to run the services also in background mode (this window won't be visible).


      The case of the feed and verify tabs, also requires having running the Stanford NLP core

The process is the same, you can run it pressing the correspondent "Start" button:

And pressing "launch" after indicate where the .cmd (startNLP.cmd) is. Which is provided in the installation, where the server was deployed

Note: this process can take several seconds, even some minutes depending of the features of the machine is running

Watch out: The application checks if the aseryla server is communicating in the same port (@n) that the port launched in the NLP server.

So, in case this port doesn't match with the defined in the Aseryla Server configuration a warning indication will be shown.

Then you have to close the aseryla server [pressing the Stop button] and [starting again the server but setting the correct port in the indicated .cmd/.sh] (or applying the same process to the nlpServer).


As option, there is an spell checker available, that could be enable or disable with the correspondent button:

In the case of Linux distributions, it requires python 2.7 with PyEnchant library (pip install pyenchant)



IMPORTANT NOTES ABOUT THE COMMUNICATION BETWEEN SYSTEMS:




EXPLORE TAB

The fisrt tab allows you can check the relations and associations that a concrete concept has.


You have to type a concept into the search box that you find in the left side.

As you go typing, you will receive a feedback with the existent concepts.

Have you noticed the stats, showed at the top of the window, about the number of concepts, relations and sentences the system contains?

And if you click in the "..." button, it extends the information [@a]:


There is also a button for purging the memory content.

Take in mind that depending of the size of the memory this process could spend several minutes (about 10 secs for each MB of file size).

And other for saving the content of the memory into Internal Language Code format.


When you write one existent concept and press the enter key, or choose one from the autocomplete list, all the knowledge about the concept that system has learnt will be displayed.

The application will show the type of word (adjective, noun, verb) the concept is (it could be more than one, e.g.: run is a verb, but also a noun)

And each positve and negative relation (frames) and association (sets). In the case of attributes also shows their related number or - if it is not defined.

When the concept name appears as highlighted means the relation was manually confirmed by any user.

Each skill may have related interactions, in the "iac" column it will show the number of interactions the skill has.

Clicking on the correspondent row (if the number is higher than zero) it will show the list of interactions with their associated tendency.



Optionally, you can show or hide the tendency and the number of sources of the relations.


And apply an ascending or descending sort to each column (concept, number, tendency, interaction) of every box:

If you click in any concept, this tab will be refreshed with the data about the selected concept.

If the enter key is pressed then the current selected concept is refreshed; this is usefull, for example, when you are checking a concept, then you provide some new senteces about this concept from feed tab and then came back to the explore tab and you want to check the new scenario (how the relation has been modified, after new knowledge has been added).


Highlighted in green is shown the weight of the concept. If you press on it then it will display a list with the concepts with the highest weight.


Apart from the ask tab, this panel offers a easily way of confirming relations.

If you perform a right click inside any concept box (also into the interaction lists [see above] of any skill relation), then appears the correspondent contextual menu:

Where you can easily confirm a relation (the tendency will be increased and the source set as confirmed), mark as erroneous for correcting relations, e.g. a positive relation it's really a negative one (the tendency change of sign but not the ...), obviously you can also remove any relation.

In addition to ilc tab, this menu also provides an alternative way of creating direct relations. And even new concepts (ensure of providing only concepts that are lemmas).

    

if the relation is not valid (e.g. selecting a concept that not exists in the dictionary or choosing a concept that is not a noun), the application will shows a popup indicating the cause of the error.


In addition, if you press the correspondent button, that appears when the selected concept is a noun (that is not an attribute), It will draw a graph with the ontology of that concept.

As the ontology of a concept could be very large, the nodes are limited to about 50 nodes, if limits are reached a "+ + + +" label is shown.


The following arrow appears when this tab is reached from any other tab (direct access when you click a concept). If you press it, you will came back to the origin tab.


As additional option the word type of any concept could be added or removed:


This section manages [@e] orders




FEED TAB

This tab is used for adding new knowledge to the system or to check previous procesed sentences.


You can type any text, into the input text box:


Note that you also have alternative ways of providing text, such as the microphone (only Windows) or loading a text file :

In case of the file loading, you have to choose if split the content of the file, assuming the sentences are line by line or split them a natural way where the dot is the indicator of the end of sentence.


And when the "PROCESS" button is clicked, then the provided text is cleaned, splitted into valid sentences, analyzed and stored into the memory their identified relations:


Unless there is any problem with the input:


The valid sentences will be listed in the right box, and it has associated an icon, that could be an green icon indicating the sentence has been correctly processed or an red icon that means that something wrong was happened (E.g.: the NLP server is not listening at that concrete moment)

In case of any error, it appears a button for reprocessing those failed sentences.


When you select any of the processed sentences, the GUI will show a description of the processing results.

  • That are their syntax

        parsed = the words of the sentences after being cleaned

        lemmas = basic form of the word

        types = POS (Part of the Speech) = the type of word

        Entities = NER (Named Entity Recognized) = if the word has been identified as Person, Location, Organization, Number ...


  • The constituent tree, or what it's the same the grammar, with highlighted those words that has been identified as subject, verb or object.


  • And the extracted relations in ILC format.


In case you select one that has failed (shows the red icon), it will show the error ocurred during its processing.


Finally, you can query the already processed sentences:

Choosing between latest provided or searching for a certain word(s) or combinations of words, using the symbol | that means OR, or the symbol & that means AND. The number of maximum sentences returned is limited by the indicated number. [@d]


Take in mind, the system stores the sentences after being correctly processed, so each word of the original sentences is replaced by their lemma.

Then you should made the searchs in lowercase, avoid the plurals, the numbers are replaced by integers, each word that does not contains letters or numbers is removed, and capital letters (only the first letter) are permitted for propers.E.g.: The cats of Anne MARIE are 10,9 % mammals → the cat of Anne be 11 mammal.


For maintenance reasons, in case any processed sentence suffered a syntax or grammar error, this is stored in the database for a posterior analysis.

Note: if you choose option "clear", all the sentences will be removed from this list, and can't be recovered.


This section mainly manages [Tp], [Tl] and [Tf] orders




ASK TAB

This tab is focused in questioning the system, it provides a structured guides to facilitate the user what kind of question can ask, and if the input fit with a valid scenario.


The first form correspond with the affirmative question type.


Where you can write the concepts and select the valid options (verb keys).

The question form changes based in the key value; For example, if the selected key is HAVE then appears the option to input numbers.

Each time a concept input box is modified, it appears a list with the matching concepts the system has (autocomplete function).


When the form is populated the question button ("?") is shown, and can be clicked (also pressing the return key) to perform the question.

Note the indicators that appears below each concept input box when the focus is lost:

    green tick = the word(s) are valid and exists into the lexicon

    yellow warning = the input doesn't match with any existing concept


When the question is asked, the response is shown.That it could be an error

Or the answer (Yes, No, Unknown).

Do not forget the "Deep Search" options (see at the top-right of the tab), since affects the answer.


Moreover, in this kind of question, You can confirm if the answer is right or not.


The "True/False"confirmation buttons only can be pressed once.

If you want to check if the system learned this, you can ask again the same question and see, in case of correcting, if the answer changed (take in mind, the question button remains disabled until the question form is modified). Or you can validate the relation, exploring the content of the concept.


As the relation affects directly the concept frame (nor the attribute or parent that matched the conditions). In case any filter is activated (see at the top-right of the tab), a warning icon is shown indicating that the response of the question could be determined by those limitations.


If the search returns a non "Unknown" response, and the asked characteristic is not found directly in the concept frame. Unless any filter is active, it appears the button "Show Path" ; if you press it then you can visualize the route traveled until reaching the concept that matched the question.


This section manages [?a] and [?c] orders



The second form allows group questioning.


The interaction is the same as above, you can select the question key, input the concepts (being helped by the autocomplete and the concept flags) and perform the question. Note that you can change from one input box to another using the "tab" key, and the grey button is for clean and reset the question form.

Remember that the subject (first input concept box) in this type of question is optional.

Unless there is an input issue or no concepts has fitted the asked conditions, a list of concepts that satisfied the question will be shown.

You can access directly to their content by clicking on them.


For security reasons the results are limited up to 20 concepts.


This section manages [?s] orders



The next form allows numbered attribute questioning.

In this case you have to indicate both concepts and receive the answer,

that can be a list of the associated numbers to the indicated attribute relation; "Any" if the relation exists but no numbers are indicated;

"None" if doesn't exist that relation or an issue description in case of validation error


This section manages [?n] orders



This group box allows to question about interaction groups.

In this case you have to indicate the kind of question type:

What DO(es) {not} CONCEPT ACTION ?


or What CAN {not} ACTION CONCEPT ?

that will answer the question with the list of interactions that fits the question (limited up to 20 concepts).


This section manages [?i] orders



The next group box allows to question about affirmative interactions.

In this case you have to indicate the concepts, the action verb and the receiver concept,

that will answer the question with a "Yes" or "No" and gives the opportunity of confirm the fact in the same way as affirmative kind question does.


If the question is satisfied but the relation is not found directly in the concept frame. Unless any filter is active, it appears the button "Show Path" ; if you press it then you can visualize the route traveled until reaching the concept that matched the question.


This section manages [?t] orders



And the last one it's a free input box where you can type any kind of question.

Where you can write any question, or even dictating via voice:

This feature is only available for Windows OS, it requires the Windows Speech API and a correctly installed microphone.


And after click on the "?" button (or pressing the enter key), and it will show the results.


That it could be an error description if there is a problem, such as, the question has not a valid and accepted format.

or showing the answer (and it populates the question values) into the correspondent box, based into the kind of asked question:

In case of being a correct multiple condition question, it moves to the correspondent guess tab

Note that the search parameters of the guess tab will be modified. Setting "strict" as ticked and "deep search" with the same values as this tab has.


This section manages [?f] orders




GUESS TAB

This tab manages multiple condition questions, as the same way as the previous tab, it provides a structured guides to facilitate the input.


You can select the valid options, write the concepts (obviously being helped by the autocomplete and icons), adding new conditions (pressing the "+" button), removing conditions ("-" button), clicking on the "reset" button, if you want to clean the form . . .


You also have the option of typing the question in raw text:


When there was inputted a valid multiple condition question (in the form or in the free box question text), the "?" button to perform the question is shown.

And if you click on them or press the enter key then the answer of the question will be displayed. But take in mind the the free box has preference, so everything you had typed in the form will be replaced by the typed question in the free box. Since the form values are populated with the typed question.


Don't worry about leaving empty condition lines (considered when their concept is empty) or not consecutive, as the application it removes automatically when the question is performed. Anyway, only for the GUI available size, there is a limitation of 10 condition as maximum.


Since it affects a lot the results and answers, when the question is asked, check the search conditions, both the strict and deep search checkboxes.

And do not forget the tendency and source filters.




When a question is asked you can receive the following responses:

  • And error description if the question is not correctly typed, there has a non existent concept, is not an accepted type, or even if it's a valid question but not a multiple condition group question:



  • In case the question is well formulated but no concepts had fit into the conditions:



  • Or the list of concepts that has a hit ratio higher than zero over the requirements of the question:



    There are also two result filters: Threshold and Limit (though the results are always limited to a maximum of 500 concepts)



    In the case of strict search, as each concept has 100% of hit ratio, the "threshold" and the " % Hit "it doesn't have sense, so are disabled:


    Remind that you can access directly to their content by clicking on them.


This section manages [?g] and [?f] orders




VERIFY TAB

This tab allows to analyze if a sentence is true or false based in the previously learned by the system.

Also for verify the extracted relations and characteristics and the sense of the identified concepts.


When you type a sentence it will draw the sentence highlighting in bold the semantics words,

placing in boxes the identified subjects, verbs and objects, and creating arrows between the extracted relations and characteristics.

It will show the correspondent error in case an invalid sentence or a sentence without relations has been inserted.


On valid sentences, it will list the identified internal language codes of the relations (it matches with the arrows drew above the sentence)

and the rest of identified characteristics (the arrows printed below the sentence).


The items of the lists will have associated a "green tick" indicating the relation exists, a "yellow cross" means the relation exists but with different tendency as the sentence indicates; and a "red cross" means the relation does not exists.

If you put the mouse over the icon, a little explanation is shown.

If the item is a "red or yellow cross", you can click in the icon to append that relation into the system ("force to true" option).

If it's not a "red cross", then clicking the icon it will graph the relation path ("see relation path" option).

Finally it provides the sense of the identified subjects.

This means that if a concept has more than one parent, depending of the characteristics extracted, it decides what is the correct meaning for the context of the inputted sentence.

E.g.: "that mouse is broken" → the sense of mouse in this case is a PC device. On the other hand, "the mouse ate all the cheese" → the sense is mammal or animal in this context.

A "yellow cross" means the sense couldn't be identified, when is "green tick" will show the associated concepts with the confidence the system has about it.


This section mainly manages [Ta], [Tw], [Ty] and [Ti] orders




ILC TAB

The last tab is used for adding knowledge directly to the memory.


You can append new concepts from the WORD section:

The content of the input boxes is validated, when both boxes are right (green check appears) the button of "save" goes to enabled.

A word/lemma is valid when only contains English letters (a-z), it could be splitted by an dash symbol (e.g.: warm-blooded) or by an white space (multiple word concept such as: basketball team). The word is allowed to be propercase (the first letter in uppercase; e.g. Bird). Never in the case of the lemma.

When the "save" button is pressed the results of the processing is shown; "OK" if everything is fine; "KO" otherwise.



Also, there is an specific guided section for the relations between the concepts:

The "save" button appears when both concepts are valid, in other words, it exists in the dictionary/lexicon.

An optional extra could be inputted in case of:

- attribute(HAVE): comma separated list of numbers or ranges. E.g.: 1,4,5-15,22

- skill(CAN): a comma separated list of interactions with the format [existent concept*tendency]. E.g.: fly*1,swim*-2

A green check is shown under the extra box when the valid value is inputted.


Obviously, you have the option of typing the entire internal language code:

In case the check box is active the system also will create any word and relation needed to append correctly the indicated knowledge.


Or processing a file:


This section mainly manages [Ti] and [Tc] orders



This interface has been developed using QT which it has some license limitations.

You can download this GUI and their source code (also of the included "Windows API microphone utility" from the download section.