Replies: 27 comments 103 replies
-
How can we maintain it? Also, I would like that it would be based on the master branch, so all library future improvements will get to it, and also so we will be able to merge it eventually with the master branch without conflicts. It means every PR to the master branch should also be applied to the "generate" branch (even if it is totally unrelated, like applied to |
Beta Was this translation helpful? Give feedback.
-
Originally, I intended to trigger the "generating" mode using "-test=generate". However, I then noted that in the PR for the improved Suggestion for a less-hacky API for that:
I propose to use option (1) for now. |
Beta Was this translation helpful? Give feedback.
-
Suggestion for a better API:
I propose to use option (1). It is also somewhat more flexible in case you would like to generate sentences with certain words, like only sentences ending with a period. |
Beta Was this translation helpful? Give feedback.
-
Exactly. This would work. In the old demo (on which I wrote in the discussion group), something like |
Beta Was this translation helpful? Give feedback.
-
|
Yes, something like that would be nice. Possibly something like If you want to work on this, we can talk more. The right way to start would be a top-down design: figure out how to take MTT-style input and convert it into what LG can work with. |
Beta Was this translation helpful? Give feedback.
-
II don't see a problem to restore this previous behavior.
There is a need to have a feature-annotated English word dictionary. |
Beta Was this translation helpful? Give feedback.
-
|
I have just modified Possibilities:
(1) is simpler, but maybe hackish. |
Beta Was this translation helpful? Give feedback.
-
Or maybe -t (--template) |
Beta Was this translation helpful? Give feedback.
-
|
Continuing discussion from pull req #1143
Oh, OK. Interesting. One is enough, but it should not be a "random sampling", it should really output a category label. This allows some later stage to pick the right word from that category: e..g. by uniform random sampling, Zipfian random sampling, or by using some external info to select a specific "synonym" in that category. The tricky question is "how to print that category label". I can avoid this question in the short-term, by using dictionaries that have only one unique word per category.
I'll move the "generate" branch over to this repo. Then I think you can do pull reqs on it (I guess, I've never done that before). Another possibility is you can pull my |
Beta Was this translation helpful? Give feedback.
-
There is no problem to provide that. This way the disjuncts (including their multiplexed costs) can be easily included too in case there is a need to use them.
Hence I would like to pedantically keep it identical as possible to As I said, I would like to force-push a |
Beta Was this translation helpful? Give feedback.
-
As a test I made a PR for branch
Updating by rebasing is ideal for me, but you will need to So the question is which option to use. |
Beta Was this translation helpful? Give feedback.
-
|
Since I'm going to do a force-push of my initial fixes, rebased on |
Beta Was this translation helpful? Give feedback.
-
I applied it this way. As expected, it got applied only to the |
Beta Was this translation helpful? Give feedback.
-
I mildly prefer 1. but only because it preserves history (mistakes, bad ideas and all). Merging preserves all authorship, all date-stamps, everything. It seems easy and uncomplicated to me. But if you really want to/need to do it some other way, I guess that's OK. |
Beta Was this translation helpful? Give feedback.
-
First thing, it is needed to rebase it on |
Beta Was this translation helpful? Give feedback.
-
I fetched the new Here is the workflow I propose: To start "cleanly", I propose that I force-push a one-time rebase on From that time on, I will take care to apply the There are some changes in the I will try to not make conflicting changes in Does all of that seem fine to you? |
Beta Was this translation helpful? Give feedback.
-
|
I merged I don't see any particular reason to restart |
Beta Was this translation helpful? Give feedback.
-
|
OK. |
Beta Was this translation helpful? Give feedback.
-
|
I said:
However, it turned out it is not that simple, since the disjunct eliminating code (in the |
Beta Was this translation helpful? Give feedback.
-
I encountered a problem when trying to include these files in |
Beta Was this translation helpful? Give feedback.
-
Right now, I don't care about these inconsistencies. They can stay that way longer. What was needed was a way for the opencog NLP subsystem to get access to the LG Exp structures. There are no other users, which is why I don't care about them. The opencog NLP subsystem uses the LG Exp structures to try to generate sentences ("sureal" and "microplanner") |
Beta Was this translation helpful? Give feedback.
-
Yes.
What if the includes were changed to use angle-brackets instead of quotes? e.g. |
Beta Was this translation helpful? Give feedback.
-
My current implementation has a big problem in providing useful expressions: These are expressions as they appear in the dict, i.e. with unresolved symbolic dialect components. (It also means that old projects that use dict expressions would misbehave and even may crash if they would use a dict with dialects.) I see two possibilities to solve that:
It should actually be: Exp *exp_with_dialect_resolved = lg_exp_set_dialect(const Exp *e, const Parse_Options opts);
void free_Exp(Exp *e);
I propose to use possibility (1), as it is much easier to implement and is more useful and flexible. It is also much more efficient (no need to pre-resolve all the expressions, no need to provide a copy). |
Beta Was this translation helpful? Give feedback.
-
This doesn't help, as the needed include files contain references to other include files. This is working fine if all are in the same directory, like in the system include directory under |
Beta Was this translation helpful? Give feedback.
-
|
Dictionaries stored in sqlite3 don't seem to work. The |
Beta Was this translation helpful? Give feedback.
-
|
The category data structure is being built in
I guess that option (2) is desired. But I don't know anything about the structure of the sqlite3 dict.
|
Beta Was this translation helpful? Give feedback.
-
|
BTW: This was the bug fix: --- a/link-grammar/dict-file/read-dict.c
+++ b/link-grammar/dict-file/read-dict.c
@@ -1884,11 +1884,11 @@ static bool read_entry(Dictionary dict)
dnx->exp = n;
i++;
}
- dict->insert_entry(dict, dn, i);
-
if (IS_GENERATION(dict))
add_category(dict, n, dn, i);
+ dict->insert_entry(dict, dn, i);
+
if (dict->suppress_warning)
{
free((void *)dict->suppress_warning);But the word selection still doesn't look like a random one at all. So there is at least one more bug in this regard. I will update the GitHub branch after I solve that (hopefully tomorrow). |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
I will push to my repository a new one in a few minutes.
BTW, |
Beta Was this translation helpful? Give feedback.
-
I have no plans for using json for anything ...
If you look at
Open a pull request whenever you are ready. I don't want to guess when that might be. |
Beta Was this translation helpful? Give feedback.
-
So I guess I can forget it for now. |
Beta Was this translation helpful? Give feedback.
-
It is possible, but then we expose the API user to internal implementation details. |
Beta Was this translation helpful? Give feedback.
-
So I should remove it before I send the next pull request. Please tell me if this is desired.
I will send a PR after you decide whether I should remove this code (for that I just need to discard the last commit). |
Beta Was this translation helpful? Give feedback.
-
|
(Starting a new thread for that topic.)
A word can indeed appear in several categories, especially with different subscripts. |
Beta Was this translation helpful? Give feedback.
-
|
Good question. You might not like the answer, which is that probablilities are proportional to exp(-cost) (or 2^{-cost} if cost is meaasured in bits instead of nats.) I don't know what the frequency of a word is in a language, for several reasons:
However, what I do know, to fairly good precision, is the ratio of different disjuncts for a given word. For any given word, I do count how many times each different disjunct is observed. This is central to the learning process. This raw count is not useful by itself; it has to be processed to obtain the mutual information (MI) between a word and a disjunct. The MI is interpreted as being minus the cost, and so the "most correct" sentences are those with the highest total MI (least cost). FYI, for word where
Equivalently where the P's are probabilities: i.e. Thus, given a sentence and a parse (a sequence of (w,d) pairs) the MI of the sentence is the sum of the MI of each pair. The likelihood or "probability" of that sentence is proportional to 2^{total MI} FWIW, individual MI's range from +30 to -10, form a bell curve with about +10 in the middle. ... they conform to the bell curve quite nicely, especially at the tails. Pairs with an MI below about +4 are "junk" or "nonsense" or "meaningless" (low-quality), while those with high MI are "very precise". Examples of high MI come from geographic place names, e.g. "Emek Ayalon Bridge" -- you will almost never see the word "Emek" by itself, it will almost always be followed by "Aylon Bridge" so the disjunct to match that will have a very high MI (very low cost) By contrast, nonsense sentences might be parseable, but will usually be filled with "junk disjuncts" and have a low or negative MI (very high cost) Oh, and FYI, all this has been measured and works well for English (I've measured for disjuncts, other people have measured other variations over the last 2-3 decades). The goal of the current language learning project is to improve the overall accuracy by a factor of 10x. That is, the above "works well", it just doesn't "work really really well". |
Beta Was this translation helpful? Give feedback.
-
|
A disjunct is typically derived from several categories, and first, you have to choose a category from them. According to your explanation, you know the chance of each disjunct to appear. What I particularly wondered is:
|
Beta Was this translation helpful? Give feedback.
-
|
During learning, categories are learned. Initially, there are only pseudo-disjuncts; connector types are learned by clustering. For example, the corpus: "the dog saw the cat. The dog chased the cat." results in two psuedo-disjuncts: From this, one can deduce that "saw chased" belong in the same word category. Correctly clustering different word meanings is harder. Consider a corpus that contains "He will saw the wood". The psuedo connector-sets are now: It is no longer obvious that "saw chased" belong in the same category. It's now a judgement call. The current code base notices that there is an overlap, and if the overlap is large enough, it will subdivide the list of pseudo-disjuncts, clustering like so: If the corpus contains "he cut the wood", one can now deduce Determining when to merge, when to split is both algorithm specific, and are controlled by tunable parameters. The goal is to find the best algorithm the best parameter tuning. Anyway, the above is for learning, not for generation. There are two very different generation modes. One is to generate a random corpus. In this case, I start by generating a random dict. Start with N words and C connector types (N and C are a user-settable parameter) Create random disjuncts of length 1,2,3... with some probability distribution (a user-settable parameter). Assign an average of A words to each disjunct (A is a parameter) etc. a few more parameters. Right now, this random-dict generator is not assigning any costs. Then, given such a random dict, create a random corpus of M sentences, (with some tuneable probability of short vs. long sentences.) The initial goal is to generate a corpus that is "good enough" for initial testing and learning-pipeline bringup. What we have now is more-or-less good enough for that (and I have fully automated most of the pipeline). I would like to add costs, and make other tweaks, but first priority is to get learning working well with the current set of tuneable parameters. The long term goal is to generate "random grammars that are just like English" but I don't yet know how to measure if a grammar is "just like English" (or "just like Russian", with morphology, or "just like Hebrew", with the things that it does.). If I can prove that (a) the learning system can accurately learn random grammars and that (b) the random languages are just like natural language X then I can claim that the learning system can accurately learn natural-language X. So that's random generation. The other, completely different kind of generation, is to create sentences from a bag of words and features: e.g. create a valid sentence that contains the words "pizza, Ben, eat", make the verb be past-tense, and be sure to say that there were two pizzas (if possible) and that they were large (if possible). And, if you can keep the sentence short, then also say that this happened on a Sunday. Otherwise create two sentences. So this is the other kind of generation: we know approximately the kinds of topics to be discussed. An approximate priority for those topics. Some limitations about time and length (write a two page short story about Ben eating pizza; I can supply you with a network of interconnected facts about where, when, how all of this happened.) -- the trick is to find grammatical word-sequences. i.e. sentences that express this network of interconnected facts. This is the other kind of generation. ... (There is some existing code that attempts to do this second kind of generation, but it does not work well. (for many reasons) and just right now, no one is actively working to fix it.) |
Beta Was this translation helpful? Give feedback.
-
|
To sum up:
|
Beta Was this translation helpful? Give feedback.
-
As I said in another post here, I will need for that en English word features dictionary. Do you know about a free one?
I intend to look at that after we merge the I also have an idea of a very fast generator for sentences with some fixed words and intervening variable wild-card words. Anyway, the idea is to first fully parse a long-enough sentence, wholly consisting of wild-card words. This may take much time and memory (and may be considered as some kind of initial learning or training). Then the connector table would be copied to a new one, without the zero count entries, and without the count fields, so its size would be very much smaller. The result connector table, along with the disjuncts of the wildcard words would then be dumped to a file, to be reread on program start. When a sentence template is then given, the program would prune these disjuncts according to the template disjuncts, and |
Beta Was this translation helpful? Give feedback.
-
|
Bug: it seems that |
Beta Was this translation helpful? Give feedback.
-
This is clearly a bug. Hence it happened on the word "a". I guess that "10" category was then due to bogus values seen by the incorrect |
Beta Was this translation helpful? Give feedback.
-
|
I just sent PR #1161. |
Beta Was this translation helpful? Give feedback.
-
Note that currently if For the generation mode to be always enabled, the categories should be loaded unconditionally, and |
Beta Was this translation helpful? Give feedback.
-
|
Always building the categories was scaffolding, while I wrote the code. I changed it to test in the same way that |
Beta Was this translation helpful? Give feedback.
-
|
I added the one-liner |
Beta Was this translation helpful? Give feedback.
-
|
For each category, it seems you store the word, and not the subscripted word. Why? Does it matter? Is something easier, if done one way and not another? |
Beta Was this translation helpful? Give feedback.
-
|
I actually store the subscripted word. You see unsubscripted words in sentence output because by default I remove the subscript when I plug a word into a category. You can use Should I remove the word plugging - since it will be done by |
Beta Was this translation helpful? Give feedback.
-
|
The word-plugging should be done in the generator. |
Beta Was this translation helpful? Give feedback.
-
OK, I will write the API and will move the word-plugging as-is to BTW, the "problem" is that almost surely some of the API (at least) will be a false-start and this may add many dummy commits until it gets stable. The word plugging will also need a total overhaul, resulting in more commits. Usually, you don't see such commits in the PRs I send, as I reorder and squash commits before I send a PR, else it would have included zillions of obfuscated commits. Especially, usually, I don't leave commits that have known buggy code - I just squash them with the fix. Else it may be harder to bisect. |
Beta Was this translation helpful? Give feedback.
-
|
I mostly don't care about buggy commits and false starts. When I code for myself, I don't bother to reorder and squash; it takes too much energy. For large projects, such as the Linux kernel, yes, having very clean and logical commits that appear in the right order is very important. But given the nature of this project, its just us two, I'm not concerned. Besides, maybe some future superintelligence will analyze the commit history to create a scientific theory of how human beings design code. |
Beta Was this translation helpful? Give feedback.
-
|
@linas, |
Beta Was this translation helpful? Give feedback.
-
|
Done |
Beta Was this translation helpful? Give feedback.
-
|
p.s. I'm putting a thumbs up every time I think a conversation is probably finished. That means I will sometimes thumbs-up my own posts. But I'm not doing this consistently; mostly its so I can keep track of what I've already read vs. what I haven't read yet. Feel free to continue any "finished" conversations. Feel free to use reactions in the same way. I wish there were more reaction types, like a check-mark or something. |
Beta Was this translation helpful? Give feedback.
-
|
Bug: With the SQL-backed dict, I am seeing sentences with two or three left-walls in them, even though left-wall has no left-facing connectors in them ... |
Beta Was this translation helpful? Give feedback.
-
|
But I cannot reproduce this bug with a flat-file equivalent dictionary... so it's something else. |
Beta Was this translation helpful? Give feedback.
-
|
You should not put them in the categories data structure.
(For the dict file, they also don't include macros/regex labels and
limit-directives).
בתאריך יום ד׳, 17 במרץ 2021, 06:03, מאת Linas Vepštas <
***@***.***>:
… Bug: With the SQL-backed dict, I am seeing sentences with two or three
left-walls in them, even though left-wall has no left-facing connectors in
them ...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1146 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAGWXGRZ66GDZAZUKNJCJTTTEAS2VANCNFSM4YKEE45Q>
.
|
Beta Was this translation helpful? Give feedback.
-
|
oh, OK. ... How did you guess? |
Beta Was this translation helpful? Give feedback.
-
|
.. I believe you, and I'll fix this tomorrow ... but how did that make a difference? Why would this mistake cause LEFT-WALL to appear two or more times? |
Beta Was this translation helpful? Give feedback.
-
This happened to me too before I excluded them... If we would like to include anything (i.e. walls, length-limit directives, macros) in the category data structures, we will need a flag field to signify that such words should not be used for a wild-card. Note that I still include idioms, but |
Beta Was this translation helpful? Give feedback.
-
If you look at the disjuncts you will see these sentences still have a full linkage. |
Beta Was this translation helpful? Give feedback.
-
|
Indeed they do. I learn something new every day. |
Beta Was this translation helpful? Give feedback.
-
|
For your (and my own) entertainment: (this is from a tiny test corpus so its "acceptable".) |
Beta Was this translation helpful? Give feedback.
-
|
Amir, you wrote:
but I can't find these remarks in the threads above. So;
My knee-jerk reaction is to say "hurrah!" because this sounds like it could be very useful. But then again, is it actually useful? Is it really needed? It sounds like something one might want to be able to do, but ... why? Life is short, this set of problems are complex enough that I don't want to spend energy designing something that is not really needed. |
Beta Was this translation helpful? Give feedback.
-
|
There s weird browser bug or github bug. The link you gave: goes to "Yes, something like that would be nice..." You used the word "deep", but its not in any comments (except in the one I just added) Closing and refreshing the window doesn't fix it. Very unusual. |
Beta Was this translation helpful? Give feedback.
-
|
You need to click on "(reply in thread)" |
Beta Was this translation helpful? Give feedback.
-
Right. But it doesn't go to the right place, it goes to some other comment (high-lighiting it and everything.) |
Beta Was this translation helpful? Give feedback.
-
|
Indeed there is inconsistent behavior (a bug).
|
Beta Was this translation helpful? Give feedback.
-
|
I needed to click on "view more". Long threads have a "view more" button, and the text is not downloaded until I click on it. Thus, search fails. And, apparently, the URL-to-comment also fails, until I hit "view more". |
Beta Was this translation helpful? Give feedback.
-
No problem. |
Beta Was this translation helpful? Give feedback.
-
|
I finished implementing the API we talked about and updated
Since it is an array of I still need to implement diagram printing in generation mode (printing disjunct-strings for category disjuncts). I noted that the category words of It seems we forgot the need to set the chosen disjunct My next step is to move |
Beta Was this translation helpful? Give feedback.
-
That's a bug. (ad I don't see where it is)
why is this needed? |
Beta Was this translation helpful? Give feedback.
-
I'll try to find out.
Just for debug and inspiration. |
Beta Was this translation helpful? Give feedback.
-
But it is not clear it is needed, as the API user can use my proposed |
Beta Was this translation helpful? Give feedback.
-
|
I proposed:
The name is not good, as it said to set the word and cost in the linkage. struct Linkage_s
{
WordIdx num_words; /* Number of (tokenized) words */
bool is_sent_long; /* num_words >= twopass_length */
const char * * word; /* Array of word spellings */
float * cost; /* Array of disjunct costs */
...
};And the API will then be: int linkage_set_word(Linkage linkage, WordIdx w, char *word);
int linkage_set_cost(Linkage linkage, WordIdx w, double cost); |
Beta Was this translation helpful? Give feedback.
-
|
Right now, I see no reason to print diagrams in generation mode, or to set costs. I'm using the generator in two ways: to generate n initial corpus from a random dict, and to generate a final corpus from the learned dict, to verify that what was learned matches the input. The actual corpus is boring, and neither the disjuncts, nor the diagram is needed, except maybe for debugging (except I have not needed to debug, and don't expect to.) I'd recommend just skipping diagram printing. It's just adding pointless complexity. |
Beta Was this translation helpful? Give feedback.
-
|
I notice you were reading commits in 'learn'; have you tried running the code? |
Beta Was this translation helpful? Give feedback.
-
|
I haven't tried the code yet but I tried to read some comments related to generation. |
Beta Was this translation helpful? Give feedback.
-
|
During generation, the dictionary may contain disjuncts there are "impossible", in that they are never used (and can't be used, e.g. because they contain connectors that don't have a matching connector going in the other direction, or because of other reasons.) It would be useful to obtain a list of these disjuncts. |
Beta Was this translation helpful? Give feedback.
-
|
I just proposed the possibility:
but it is not good since it doesn't provide access to the So only this seems fine:
|
Beta Was this translation helpful? Give feedback.
-
|
I implemented the following: Disjunct ** sentence_unused_disjuncts(Sentence); // NULL terminated.
char * disjunct_expression(Disjunct *); // To be free()'ed by the caller.
Category_cost *disjunct_categories(Disjunct *); // NULL terminatedIn I also added to |
Beta Was this translation helpful? Give feedback.
-
|
Result example. Note that the shown |
Beta Was this translation helpful? Give feedback.
-
|
I finished the API of unused disjuncts. link_public_api(Disjunct **)
sentence_unused_disjuncts(Sentence);
link_public_api(char *)
disjunct_expression(Disjunct *);
link_public_api(const Category_cost *)
disjunct_categories(Disjunct *);
BTW, I still haven't implemented unused disjuncts consideration for fixed words. If this API looks fine, I will send a PR. |
Beta Was this translation helpful? Give feedback.
-
|
The API seems reasonable. Mark it as "experimental, subject to change". Using
Some aspects of this are confusing, so let me think out loud a bit. The request arises because I'm generating random dictionaries, and I suspect that those dictionaries include disjuncts that can never be used, no matter what the length of a sentence. But I don't know, and thought it would be interesting to find out. I don't even know if this is a tiny fraction of all, or whether most of them are unusable. To figure that out, what you propose seems adequate. To obtain a list of unused disjuncts for any length, the lists for fixed sentence lengths would need to be intersected; but this is easy enough with bash, perl, scheme. For sentences with fixed words, its confusing. There are multiple pruning steps; should pruned disjuncts on fixed words count as "unused" or not? What about "unused" disjuncts on *-words? I don't know. .. Right now I have no need for a good answer for this situation. |
Beta Was this translation helpful? Give feedback.
-
|
I modified However, with some effort, it is possible to restore (with library code) the full set of different possible costs for each sentence. |
Beta Was this translation helpful? Give feedback.
-
My algo for finding unused disjuncts compares the disjuncts of *-words in the parse results to those of a *-word as obtained from the dict (i.e. before pruning). The missing ones are reported as unused. The extension for fixed words is to consider them as used if they are used on some fixed word (for long sentences maybe these are only the walls). It seems possible to use the list of unused disjuncts as feedback to prune the dictionary, to obtain a dictionary with less redundant disjuncts (or maybe even no redundant disjuncts). |
Beta Was this translation helpful? Give feedback.
-
|
Regarding this comment
A few years ago, I realized that I should think of dictionaries as having the shape of a spider or centipede, having a complex body, with legs on one side, and legs on another. Each leg on one side is a lists of words sharing a common expression. Each leg on the other side is a macro such as The act of creating a dictionary, either by hand, or automatically, is to define these legs, and the body. Formally, it resembles matrix factorization. Consider the matrix I claim tells me there are about 2300 different word-lists or "spider-legs" in the dict. Meanwhile: tells me there are about 500 macro definitions in the dict. The "grammar" of the dict, the complexity, is A key part of the language-learning project is to obtain this matrix factorization. Your code already has the concept of "wordclasses" hard-coded into it. Wordclasses are exactly the One reason that I want to think of this as a matrix factorization problem is that there is a lot of interesting theory to draw on. There are some very traditional matrix factorization algos: these are used by google and amazon to factorize (customer, product) matrices. There are some more sophisticated algos, used for factoring (antigen,immunoglobulin) pairs (I kid you not - the paper I read did it for zebrafish, which have a simple immune system, and they contrasted several matrix factorization methods, including Ising models and Markov matrices and more complex combinatorial techniques). Last but not least, the matrix The above is easy to say (well, easy for me to say). It takes a little more work to understand (I have an 80 page paper that goes through the details, including a reference to the google, amazon and zebrafish papers). The very hardest part, however, is writing the code to make it happen, to actually do it. Anyway: You now understand what the |
Beta Was this translation helpful? Give feedback.
-
Did you mean 20M? Some comments:
|
Beta Was this translation helpful? Give feedback.
-
I guess I did! Wow!
That means there are at most 3M disjunct classes (right?), and maybe a whole lot less.
OK, so here's the deal. 10 years ago, when the For the rules that are still active, it might be possible to deactivate them by altering the dictionary to be more tightly constrained. This seemed to be possible in the few cases I looked at. Doing this is a chore - its time-consuming work. Testing to see if it made things better or worse is hard to do. So I'm happy to leave post-processing alone, for now. Then, the meta-issues. The learning code doesn't have any way of learning post-processing rules. It could, I suppose, but should it? It's complicated. The other meta-issue is testing. Just because LG produces a (low-cost) parse, that doesn't mean its the right parse. I don't know how to automate testing for this, except to manually review each and every parse, and decide if it is good or not. It's hard to bake this into a unit test, because changes to the dict can convert good parses into even-better parses, while the unit test would flag the even-better parse as an error. There is no obviously-good solution to this, and I am not convinced that it is even worth the effort to find a solution. If the ultimate goal is to learn dictionaries automatically, "goodness" has to be measured some other way.
Here's a deeply flawed, non-sense estimate: This suggests perhaps 7K x 5K = 35M disjuncts. So the right ballpark. |
Beta Was this translation helpful? Give feedback.
-
I'm not sure about what you mean in "X disjunct classes". Here is the result of The 3598714 duplicates are with the same
Because I once started to rewrite the post-processing (and found it can be done several times faster), I looked at your check of which post-processing rules are not triggered, and it has a problem: it doesn't take into account that
I think that in general, it may be impossible since post-processing deals will link names, which you cannot generally restrict in the dict, and also they deal with link formats, like "must form a circle". Since link names are known in In the "old times", while constructing a more advanced Hebrew dict (incomplete and unpublished work), I encountered a case in which it seemed that only post-processing is the reasonable solution. It was a case involving a general rule of combining several words, that had 2 exceptions of certain links that are forbidden. Because such exceptions cannot be described in the dict, trying to directly do it in the dict involved abandoning the general rules and inventing a lot of connectors just to implement a long list of private cases. I will try to simplify this problem to a simple rule plus a few words in order to demonstrate that (it may take a while).
I think it can bypass the need for post-processing by just generating a lot of private cases.
The test suite may test mainly simple sentences in which the best parse is "known" and less likely to change, and this only in order to check if the development process didn't introduce obvious bugs to the library or the dictionary. I guess this is more what it does now.
A partial way may be to just do an accept/reject test on a corpus in which it is already known which sentences are grammatical (or that we can suppose that absolutely most of the sentences int are all grammatical or all ungrammatical). This of course doesn't tell if the detailed parsing is correct. (From here I continued to think about the possibility of automatic learning of punctuation or the roll of uppercase letters. This maybe belongs to the previous discussion ("next steps")). |
Beta Was this translation helpful? Give feedback.
-
I do not currently have any good formal definition for these. It's more of an informal observation that they are there, but that the boundary between the
Ooops! Well, that's a mistake.
The use of lower case letters in link names can enforce long-distance constraints. How to do this is not obvious; it was a long time before I realized this could be done. Anyway, the learning code does not know how to use lower-case letters in link names, or how to impose constraints. That's a different project for some other day. |
Beta Was this translation helpful? Give feedback.
-
|
In https://groups.google.com/g/link-grammar/c/GWLVSVWaUDU/m/6Klgg3oyAQAJ @linas said:
Indeed. After some thought, I still don't have an idea how to apply the "entropy collapse algorithm" using LG. One problem is that the LG constraints are not local at all. Another problem is that a local minimum doesn't imply a minimum sentence cost. However:
I have some ideas regarding that, using Python libraries like inflect combined with LG.
LG currently does this when generating a sentence: The words at the edges are severely constrained by the walls, which limits their disjunct length, and then the power pruning does a good job there. In any case, we may try to extend the LG library with an infrastructure that enables better generation. Just now a very limited form of this idea is already supported: A wildcard prefix. For example, try this: I.e. each slot can use the current wildcard expansion infrastructure. This can trivially be expanded to using a regex (and would even speed up the generation much due to the added word constraints): By a proper API, this can be extended to a set of sentence fragments for each slot (this is what I referred to above by ordered list of words, but they can be marked as unordered if desired): On top of such (and more) experimental infrastructure, an experimental generator may be written using Python bindings (and several other external libraries). The idea is that the "classic" parser is extremely fast and as long as you are not going to assign an extremely big list of words in each slot (like now, using I already started to expand the library infrastructure by adding the ability to parse text fragments and words which may expand to a list of other words (unfinished). |
Beta Was this translation helpful? Give feedback.
-
|
See also discussion in #1290 |
Beta Was this translation helpful? Give feedback.
-
Previous discussion on that at these places:
Beta Was this translation helpful? Give feedback.
All reactions