Following up on my previous post of January 2008, “Corpus linguistics & Concgramming in Verbots and Pandorabots”, you can now see the demo of this VagaBot at http://www.mendicott.com . The results of this trial were not satisfying due to the limitation of the VKB engine at verbotsonline.com not being able to process consecutive, or random, responses from identical input or triggers, basically tags. In other words, the responses with identical input hang on the first response, and not cycle through the series of alternatives. Apparently a commercial implementation of the Verbots platform does allow for the consecutive firing of related replies. Thanks to Matt Palmerlee of Conversive, Inc. for increasing the online knowledgebase storage to accommodate this trial and demo.
Dr. Rich Wallace has recently blogged a very helpful post, “Saying a list of AIML responses in order”, on his Alicebot blog at http://alicebot.blogspot.com . After considerable fiddling, I have successfully installed Program E on my Windows desktop under Wampserver (Apache, MySQL, PHP). I have also found a very easy commercial product for importing RSS feeds into MySQL. Next I will try to bridge the RSS database and the Program E AIML database with Extensible Stylesheet Language Transformations (XSLT) using the previously mentioned xsl-easy.com database adapters… as well as implement Dr. Wallace’s "successor" function on the Program E AIML platform. Once I get the prototype working on my desktop, I will then endeavor to replicate it on a remote server for public access.
The long term goals of Project VagaBot are to create a conversational agent that can not only “read” books, but also web feeds, and “learn” to reply intelligently to questions, in this case on “green travel”, in effect an anthropomorphic frontend utilizing not only my book, "Vagabond Globetrotting 3", but also my entire http://meta-guide.com feed resources as backend. I am not aware of another project that currently makes the contents of a book available using a conversational agent, nor one that “learns” from web feeds. I hope to eventually be able to send the VagaBot avatar into smartphones using both voice output and input. I would be very interested in hearing from anyone interested in investing or otherwise supporting this development.
Showing posts with label XSLT. Show all posts
Showing posts with label XSLT. Show all posts
17 August 2008
20 January 2008
Corpus linguistics & Concgramming in Verbots and Pandorabots
One of the definitions of semantic, as in Semantic Web or Web 3.0, is the property of language pertaining to meaning, meaning being significance arising from relation, for instance the relation of words. I don’t recall hearing about corpus linguistics before deciding to animate and make my book interactive. Apparently there has been a long history of corpus linguistics trying to derive rules from natural language, such as the work of George Kingsley Zipf. As someone with a degree in psychology, I do know something of cognitive linguistics and its reaction to the machine mind paradigm.
The man who coined the term, called artificial intelligence "the science and engineering of making intelligent machines,” which today is referred to as "the study and design of intelligent agents." Wikipedia defines intelligence as “a property of the mind that encompasses… the capacities to reason, to plan, to solve problems, to think abstractly, to comprehend ideas, to use language, and to learn.” Computational linguistics has emerged as an interdisciplinary field involved with “the statistical and/or rule-based modeling of natural language.”
In publishing, a concordance is an alphabetical list of the main words used in a text, along with their immediate contexts or relations. Concordances are frequently used in linguistics to study the body of a text. A concordancer is the program that constructs a concordance. In corpus linguistics, concordancers are used to retrieve sorted lists from a corpus or text. Concordancers that I looked at included AntConc, ConcordanceSoftware, WordSmith Tools and ConcApp. I found ConcApp and in particular the additional program ConcGram to be most interesting. (Examples of web based concordancers include KWICFinder.com and the WebAsCorpus.org Web Concordancer.)
Concgramming is a new computer-based method for categorising word relations and deriving the phraseological profile or ‘aboutness’ of a text or corpus. A concgram constitutes all of the permutations generated by the association of two or more words, revealing all of the word association patterns that exist in a corpus. Concgrams are used by language learners and teachers to study the importance of the phraseological tendency in language.
I was in fact successful in stripping out all the sentences from my latest book, VAGABOND GLOBETROTTING 3, by simply reformatting them as paragraphs with MSWord. I then saved them as a CSV file, actually just a text file with one sentence per line. I was able to make a little utility which ran all those sentences through the Yahoo! Term Extraction API, extracting key terms and associating those terms with their sentences in the form of XML output, as terms equal title and sentences equal description. Using the great XSLT editor xsl-easy.com, I could convert that XML output quickly and easily into AIML with a simple template.
The problem I encountered was that all those key terms extracted from my book sentences when tested formed something like second level knowledge that you couldn’t get out of the chatbot unless you already knew the subject matter…. So I then decided to try adding the concgrams to see if that could bridge the gap. I had to get someone to create a special program to marry the 2 word concgrams from the entire book (minus the 100 most common words in English) to their sentences in a form I could use.
It was only then that I began to discover some underlying differences between the verbotsonline.com and pandorabots.com chatbot engine platforms. I've been using verbotsonline because it seemed easier and cheaper, than adding a mediasemantics.com character to the pandorabot. However, there is a 2.5 Meg limit with verbotsonline knowledgebases, which I've reached three times already. Also, verbotsonline.com does not seem to accept multiple SAME patterns with different templates, at least the AIML-Verbot Converter apparently removes the “duplicate” categories.
In verbots, spaces automatically match to zero or more words, so wildcards are only necessary to match partial words. This means in verbots words are automatically wildcarded, which makes it much easier to achieve matches with verbots. So far, I have been unable to replicate this simple system with AIML, which makes AIML more precise or controllable, but perhaps less versatile, at least in this case. Even with the AIML knowledgebase replicated eight times with the following patterns, I could not duplicate the same results in pandorabots as the verbots do with one file, wildcarding on all words in a phrase or term.
dog cat
dog cat *
_ dog cat
_ dog cat *
dog * cat
dog * cat *
_ dog * cat
_ dog * cat *
The problem I encountered with AIML trying to “star” all words was that when starred at the beginning of a pattern only one word was accepted and not more words, and when replaced with the underscore apparently affects pattern prioritization. So there I am at the moment stuck between verbots and pandorabots, not being able to do what I want with either, verbotsonline for lack of capacity and inability to convert “duplicate” categories into VKB, and pandorabots for inability to conform to my fully wildcarded spectral word association strategy….
The man who coined the term, called artificial intelligence "the science and engineering of making intelligent machines,” which today is referred to as "the study and design of intelligent agents." Wikipedia defines intelligence as “a property of the mind that encompasses… the capacities to reason, to plan, to solve problems, to think abstractly, to comprehend ideas, to use language, and to learn.” Computational linguistics has emerged as an interdisciplinary field involved with “the statistical and/or rule-based modeling of natural language.”
In publishing, a concordance is an alphabetical list of the main words used in a text, along with their immediate contexts or relations. Concordances are frequently used in linguistics to study the body of a text. A concordancer is the program that constructs a concordance. In corpus linguistics, concordancers are used to retrieve sorted lists from a corpus or text. Concordancers that I looked at included AntConc, ConcordanceSoftware, WordSmith Tools and ConcApp. I found ConcApp and in particular the additional program ConcGram to be most interesting. (Examples of web based concordancers include KWICFinder.com and the WebAsCorpus.org Web Concordancer.)
Concgramming is a new computer-based method for categorising word relations and deriving the phraseological profile or ‘aboutness’ of a text or corpus. A concgram constitutes all of the permutations generated by the association of two or more words, revealing all of the word association patterns that exist in a corpus. Concgrams are used by language learners and teachers to study the importance of the phraseological tendency in language.
I was in fact successful in stripping out all the sentences from my latest book, VAGABOND GLOBETROTTING 3, by simply reformatting them as paragraphs with MSWord. I then saved them as a CSV file, actually just a text file with one sentence per line. I was able to make a little utility which ran all those sentences through the Yahoo! Term Extraction API, extracting key terms and associating those terms with their sentences in the form of XML output, as terms equal title and sentences equal description. Using the great XSLT editor xsl-easy.com, I could convert that XML output quickly and easily into AIML with a simple template.
The problem I encountered was that all those key terms extracted from my book sentences when tested formed something like second level knowledge that you couldn’t get out of the chatbot unless you already knew the subject matter…. So I then decided to try adding the concgrams to see if that could bridge the gap. I had to get someone to create a special program to marry the 2 word concgrams from the entire book (minus the 100 most common words in English) to their sentences in a form I could use.
It was only then that I began to discover some underlying differences between the verbotsonline.com and pandorabots.com chatbot engine platforms. I've been using verbotsonline because it seemed easier and cheaper, than adding a mediasemantics.com character to the pandorabot. However, there is a 2.5 Meg limit with verbotsonline knowledgebases, which I've reached three times already. Also, verbotsonline.com does not seem to accept multiple SAME patterns with different templates, at least the AIML-Verbot Converter apparently removes the “duplicate” categories.
In verbots, spaces automatically match to zero or more words, so wildcards are only necessary to match partial words. This means in verbots words are automatically wildcarded, which makes it much easier to achieve matches with verbots. So far, I have been unable to replicate this simple system with AIML, which makes AIML more precise or controllable, but perhaps less versatile, at least in this case. Even with the AIML knowledgebase replicated eight times with the following patterns, I could not duplicate the same results in pandorabots as the verbots do with one file, wildcarding on all words in a phrase or term.
dog cat
dog cat *
_ dog cat
_ dog cat *
dog * cat
dog * cat *
_ dog * cat
_ dog * cat *
The problem I encountered with AIML trying to “star” all words was that when starred at the beginning of a pattern only one word was accepted and not more words, and when replaced with the underscore apparently affects pattern prioritization. So there I am at the moment stuck between verbots and pandorabots, not being able to do what I want with either, verbotsonline for lack of capacity and inability to convert “duplicate” categories into VKB, and pandorabots for inability to conform to my fully wildcarded spectral word association strategy….
30 December 2007
AIML <-> OWL ??
Since I posted my original query to the pandorabots-general list in July, I'm beginning to understand the concepts involved a little better, thanks also to replies from this group and others, such as the protege-owl list.
In a comment to my recent blog entry ("I'm dreaming of RSS in => AIML out"), Jean-Claude Morand has mentioned that RSS 1.0 would probably be more conducive to conversion into RDF or AIML than RSS 2.0. He also mentioned that the Dublin Core metadata standard may eventually overtake RSS in primacy....
So, can XSL transforms really be used to translate between RSS and RDF, and between RDF and AIML?? My understanding at this point is that talking about AIML and OWL is a bit like apples and oranges.... Apparently the output from an OWL Reasoner would be in RDF? I have by now discovered the Robitron group and am finding that archive to be a rich resource....
What does this have to do with Pandorabots? I would like to address a brief question, in particular to Dr. Wallace... what do you see as the impediments to upgrading the Pandorabots service to include an OWL Reasoner (or in chaining it to another service that would provide the same function)? Surely you've considered this.... Where are the bottlenecks (other than time and money of course)? Is it an unreasonable expectation to be able to upload OWL ontologies much the same as we can upload AIML knowledgebases today?
As I have mentioned previously, one of my interests is creating knowledgebases on the fly using taxonomies. My belief is that quick and dirty knowledgebases are a more productive focus than pouring time and energy into trying to meet the requirements of the Turing test (another rant for another day....) Certainly with chatbots there is a substantial element of smoke and mirrors involved in any case.... One can always go back and refine as needed based on actual chat logs.
The next step for me will be to try and convert my most recent book, VAGABOND GLOBETROTTING 3, into a VagaBot.... I would like to hear from anyone with experience in converting books into AIML knowledgebases! My supposition is that a *good* book index is in effect a "taxonomy" of that book.... My guess is that I can use the index entries as patterns, and their referring sections as templates... to create at least the core of a knowledgebase. If more detail is needed then a concordance can always be applied to the book.
After that I hope to tackle creating quick and dirty AIML knowledgebases on the fly from RSS feed title and description fields... not in pursuit of the chimera of the Turing test, but simply to build a better bot. (Now, I wonder if anyone has ever created RSS from a book?!? ;^))
In a comment to my recent blog entry ("I'm dreaming of RSS in => AIML out"), Jean-Claude Morand has mentioned that RSS 1.0 would probably be more conducive to conversion into RDF or AIML than RSS 2.0. He also mentioned that the Dublin Core metadata standard may eventually overtake RSS in primacy....
So, can XSL transforms really be used to translate between RSS and RDF, and between RDF and AIML?? My understanding at this point is that talking about AIML and OWL is a bit like apples and oranges.... Apparently the output from an OWL Reasoner would be in RDF? I have by now discovered the Robitron group and am finding that archive to be a rich resource....
What does this have to do with Pandorabots? I would like to address a brief question, in particular to Dr. Wallace... what do you see as the impediments to upgrading the Pandorabots service to include an OWL Reasoner (or in chaining it to another service that would provide the same function)? Surely you've considered this.... Where are the bottlenecks (other than time and money of course)? Is it an unreasonable expectation to be able to upload OWL ontologies much the same as we can upload AIML knowledgebases today?
As I have mentioned previously, one of my interests is creating knowledgebases on the fly using taxonomies. My belief is that quick and dirty knowledgebases are a more productive focus than pouring time and energy into trying to meet the requirements of the Turing test (another rant for another day....) Certainly with chatbots there is a substantial element of smoke and mirrors involved in any case.... One can always go back and refine as needed based on actual chat logs.
The next step for me will be to try and convert my most recent book, VAGABOND GLOBETROTTING 3, into a VagaBot.... I would like to hear from anyone with experience in converting books into AIML knowledgebases! My supposition is that a *good* book index is in effect a "taxonomy" of that book.... My guess is that I can use the index entries as patterns, and their referring sections as templates... to create at least the core of a knowledgebase. If more detail is needed then a concordance can always be applied to the book.
After that I hope to tackle creating quick and dirty AIML knowledgebases on the fly from RSS feed title and description fields... not in pursuit of the chimera of the Turing test, but simply to build a better bot. (Now, I wonder if anyone has ever created RSS from a book?!? ;^))
Subscribe to:
Posts (Atom)