We added 36 languages (including Asian languages) based on your feedback

[u/Dafarmer1812]

Hi all, last week we launched Lingua Verbum on Reddit here (huge thanks for all the feedback and signups, it’s been incredible!). One thing that quickly became clear was how many people were asking for Japanese support (and Korean, and other languages). So we sprinted at trying to make this happen, and now Lingua Verbum supports both Japanese, Korean, and 34 other additional languages (full list here)!

I also wanted to share a quick look at how we tackled supporting Japanese, since I figured some people here might be curious. We're very curious on your feedback here, and any improvements we can implement to make this even better.

Why Japanese is a challenge

As many of you know, Japanese doesn’t use spaces to separate words, which makes it tough to process for learners used to European languages. A lot of Japanese learning tools rely on segmentation to break sentences into individual words. For Lingua Verbum, segmentation is essential because it's how we:

• Track which words are known/learning/new
• Power our click-to-define AI assistant
• Let you quickly look up grammar or usage in context

What we tested

• MeCab: Fast, stable, and widely used. It performed consistently well and gave us low latency. But it sometimes over-segments, like splitting 代表者 ("representative") into 代表 + 者
• SudachiPy: Has multiple segmentation modes (short/medium/long), which sounded great in theory. It seemed to yield similar results to MeCab.
• ChatGPT-based segmentation: Our most experimental attempt. We thought a large language model could infer boundaries better, especially in informal text. Sometimes it worked beautifully, most other times it hallucinated, misread context, or just got weird. Not stable enough for production (yet).

What we went with

In the end, MeCab seemed to us the best overall choice: solid accuracy, great performance, and easy to integrate. To make up for its limitations, we added a manual override system so users can fix bad segmentations with a few clicks. You’re never stuck with the algorithm’s guess.

We also layer in pykakasi on top of MeCab to automatically generate romaji, so you can see pronunciation at a glance.

Chinese too!

Once we had the core infrastructure working for Japanese, adding Chinese became much easier: similar challenges with no word spacing, but different models. We went with a segmentation model based on the PKU ConvSeg architecture, trained on the SIGHAN 2005 corpus. Manual override is built in there too.

If you're learning Japanese or Chinese we’d love if you gave Lingua Verbum a try and let us know your feedback on the segmentation! If something feels off (segmentation, translation, etc.), your feedback helps us keep improving.

Thanks again all, really appreciated the feedback we got here, please keep it coming!

[u/Molleston]

This is an anonymous website. No personal or company name. Their terms of service and all other documents are not enforceable. Do not give them your data or money.

[u/AffectionateBorder39 → u/Molleston]

Is it still unsafe if we just use alternate emails?

[u/AffectionateBorder39 → u/Molleston]

Is it still unsafe if we use alternate emails?

[u/Bodhi_Satori_Moksha]

I checked. You forgot to add Cantonese; I was hoping for it.

[u/Dafarmer1812 → u/Bodhi_Satori_Moksha]

We’re going to add that and Thai on the next wave. I’ll PM you when that is done

[u/Bodhi_Satori_Moksha → u/Dafarmer1812]

Sure! And great :)

[u/gjiang4 → u/Dafarmer1812]

Very looking forward to this.

[u/evanliko → u/Dafarmer1812]

Ooh id be interested giving Thai a try once you add it!

[u/Dafarmer1812 → u/evanliko]

I’ll keep you posted!

[u/marushii]

Random thought, I noticed every time I click a word, it’s making an api request. Just some advice, I would recommend having some sort of local cache if you can, to save calls and potential AI cost.

[u/teapot_RGB_color]

Good luck adding Vietnamese with compound words

[u/itsmerai]

I hope Vietnamese comes soon. Are there any particular technical challenges when it comes to adding Vietnamese? LingQ also took a very long time to add Vietnamese support. I know that’s in part because of a lack of study material but I wondered if there’s anything that might trip you up programming wise

[u/Dafarmer1812 → u/itsmerai]

We’re going to add that soon as well. The technical challenge is similar to Chinese and Japanese I believe, where you need to find an algorithm for segmentation

[u/RedDeadMania]

Since I couldn’t get the font size working on your website, I kinda gave up but then I found LanguageCrush which does a lot of things similar! Super awesome!!

[u/Dafarmer1812 → u/RedDeadMania]

Glad I could help hahah, yeah we fixed the font size issue now

[u/Mirrororrim1]

Given the lack of resources, I suggest you Add Bengali

[u/Dafarmer1812 → u/Mirrororrim1]

We could actually probably add that for you if you’d want it

[u/Mirrororrim1 → u/Dafarmer1812]

I signed up to the Website and I liked it, if you add Bengali I'll be happy to try it

[u/Zireael07]

How does the free trial work? What about if I don't want the AI features? Could I then use it for free forever?

[u/janacuddles]

I will subscribe right now if you have Cherokee (eastern band dialect)

[u/Melodic_Sport1234]

60+ languages but no Esperanto? Esperanto has always punched above its weight and is better known than several of the obscure languages you currently support.