Various tasks where LLM might help

[nicolas-raoul]

• help find correct category name, showing suggestions from "Cuisine of Japan" when "Japan food" is typed
• fix obvious typos in depiction search
• detect unwanted nudity pictures from caption
• detect meaningless captions such as PXL1234 etc
• show translation of captions/etc not available in the user's languages. Either show next to original, or replace original when "AI translation" button shown. Better than OS-level because we know source language.

Feel free to comment if you want to add more, thanks! :-)

[Thejas775]

Addition ? Maybe check the caption with image

[nicolas-raoul]

Maybe check the caption with image

Indeed great idea! All Android device-embedded models so far seem to be text-to-text, but that could change in the future.

[Thejas775]

Maybe check the caption with image

Indeed great idea! All Android device-embedded models so far seem to be text-to-text, but that could change in the future.

yeah in future maybe. Now too very less devices support on device model running.
I also have a doubt why are we not considering api calls?

[whym]

I think API calls to a Wikimedia-controlled Toolforge server would be fine from a privacy perspective. (We already call Wikimedia servers, including a Toolforge server, so more calls to Wikimedia servers would be fine.) However, whether Toolforge will host a large vision-language model would depend on their policy and is still unclear (https://phabricator.wikimedia.org/T336905).

[nicolas-raoul]

I implemented a proof-of-concept for LLM use in the app: #6553
It works for me, please everyone test it if you are interested in this topic. :-)
In particular, if you have an out-of-the-box recent phone (Pixel 9/10) and you have not done anything special related to AICore (not installed it voluntarily, not joined a AICore-specific Google Group, nor done anything special to that a normal user would not have done), please let me know whether it works or not. Thanks a lot!

By the way, now the LLM embedded in recent phones supports image-to-text too.