Jeremy Neiman
1 min readJan 5, 2019

--

You’re right about the discrepancy between Japanese and English.

As for a representation of English that would be easier to learn, one option is the one I mentioned about using phonemes. Each one uniquely carries a weight of either one or zero syllables. So learning to count syllables is only a function of counting a subset of symbols. I’m already using phonemes in the data pre-processing step to determine the number of syllables in the training data, so it would be pretty straightforward to create a model on top of that. But it introduces the added complexity of determining words from phonemes, since they aren’t one-to-one.

Your idea about using a hyphenated representation is interesting because that further simplifies the task to counting hyphens.

Using the hyphenated representation, it would also be possible to learn a sequence of syllables, in which case the task could become very easy, returning to the original Japanese of a fixed number of characters on each line. Although I suspect that the number of unique syllables is too great to make that work effectively.(http://web.archive.org/web/20160822211027/http://semarch.linguistics.fas.nyu.edu/barker/Syllables/index.txt)

--

--

Jeremy Neiman
Jeremy Neiman

Written by Jeremy Neiman

Coder of random things. Software engineer and data scientist. All views are my own. http://jeremyneiman.com

No responses yet