Skip to content

Conversation

@nicolas-raoul
Copy link
Member

@nicolas-raoul nicolas-raoul commented Nov 5, 2025

screen-20251105-195935-1762352939473.mp4

The proof of concept is working, as seen in the screencast above, but more work is needed:

  • Test on various devices, including non-AICore ones.
  • See whether adding -Xskip-metadata-version-check was really a good idea, Migrate to Kotlin 2.1.0 and Compose Compiler Plugin #6552 would probably be better.
  • Fix CI.
  • Reduce model output length, we just need 2 or 3 characters.
  • Make the prompt faster.
  • Possibly have a second more reliable prompt with a few shots, used only if the fast prompt detects a mismatch.

I am OK with supporting only French at first, the model does not know all Wiki languages. We can extend to more languages in a next phase.

When run on an old phone (Pixel 5), there is nothing special visible to the user, just a silent error in logcat but the app behaves like if the feature was not there. Which is good. Here is the logcat for reference:

11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: Error detecting language
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: com.google.mlkit.genai.common.GenAiException: [ErrorCode -101] AICore is either not installed or the installed version is too low.
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.google.android.gms.internal.mlkit_genai_prompt.zzajd.zze(com.google.mlkit:genai-prompt@@1.0.0-alpha1:3)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.google.android.gms.internal.mlkit_genai_prompt.zzajd.zzw(com.google.mlkit:genai-prompt@@1.0.0-alpha1:2)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.google.android.gms.internal.mlkit_genai_prompt.zzajd.zzA(com.google.mlkit:genai-prompt@@1.0.0-alpha1:4)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.google.android.gms.internal.mlkit_genai_prompt.zzajh.zzP(com.google.mlkit:genai-prompt@@1.0.0-alpha1:1)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.google.android.gms.internal.mlkit_genai_prompt.zzot.zzq(com.google.mlkit:genai-prompt@@1.0.0-alpha1:13)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.google.android.gms.internal.mlkit_genai_prompt.zzot.generateContent(com.google.mlkit:genai-prompt@@1.0.0-alpha1:2)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.UploadActivity$isCaptionFrench$1.invokeSuspend(UploadActivity.kt:902)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:363)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:21)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:88)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:123)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:52)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.BuildersKt.launch(Unknown Source:1)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:43)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source:1)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.UploadActivity.isCaptionFrench(UploadActivity.kt:893)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.UploadActivity.onNextButtonClicked(UploadActivity.kt:832)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.UploadActivity$receiveSharedItems$uploadMediaDetailFragmentCallback$1.onNextButtonClicked(UploadActivity.kt:561)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.mediaDetails.UploadMediaDetailFragment.onImageValidationSuccess(UploadMediaDetailFragment.kt:446)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.mediaDetails.UploadMediaPresenter.handleCaptionResult(UploadMediaPresenter.kt:216)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.mediaDetails.UploadMediaPresenter$verifyCaptionQuality$1.invoke(UploadMediaPresenter.kt:181)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.mediaDetails.UploadMediaPresenter$verifyCaptionQuality$1.invoke(UploadMediaPresenter.kt:179)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.mediaDetails.UploadMediaPresenter.verifyCaptionQuality$lambda$7(UploadMediaPresenter.kt:179)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.mediaDetails.UploadMediaPresenter.$r8$lambda$A76sHcUjuWre4t-wRXfD9J9Sy4g(Unknown Source:0)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at fr.free.nrw.commons.upload.mediaDetails.UploadMediaPresenter$$ExternalSyntheticLambda2.accept(D8$$SyntheticClass:0)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at io.reactivex.internal.observers.ConsumerSingleObserver.onSuccess(ConsumerSingleObserver.java:62)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at io.reactivex.internal.operators.single.SingleObserveOn$ObserveOnSingleObserver.run(SingleObserveOn.java:81)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at io.reactivex.android.schedulers.HandlerScheduler$ScheduledRunnable.run(HandlerScheduler.java:124)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at android.os.Handler.handleCallback(Handler.java:942)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at android.os.Handler.dispatchMessage(Handler.java:99)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at android.os.Looper.loopOnce(Looper.java:201)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at android.os.Looper.loop(Looper.java:288)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at android.app.ActivityThread.main(ActivityThread.java:7898)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at java.lang.reflect.Method.invoke(Native Method)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
11-06 17:38:16.413  9698  9698 E UploadActivity$isCaptionFrench: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936)

google-labs-jules bot and others added 15 commits November 5, 2025 08:36
Adds a dialog to confirm if the caption is in French when the user enters 'bonjour' as the only caption and the language is the default, but not French.
Adds a dialog to confirm if the caption is in French when the user enters 'bonjour' as the only caption and the language is the default, but not French. This check is performed when the user proceeds to the next step in the upload wizard.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. If the user enters a caption in French, has only one caption, and the language is the default (but not French), a dialog is shown to confirm if the language should be changed to "fr". This check is performed when the user proceeds to the next step in the upload wizard.
…bility

Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. If the user enters a caption in French, has only one caption, and the language is the default (but not French), a dialog is shown to confirm if the language should be changed to "fr". This check is performed when the user proceeds to the next step in the upload wizard.

This feature is only enabled for Android API level 26 and above to maintain compatibility with older devices. This is achieved by adding a version check and overriding the library's minSdk in the manifest.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. If the user enters a caption in French, has only one caption, and the language is the default (but not French), a dialog is shown to confirm if the language should be changed to "fr". This check is performed when the user proceeds to the next step in the upload wizard.

This feature is only enabled for Android API level 26 and above to maintain compatibility with older devices. This is achieved by adding a version check and overriding the library's minSdk in the manifest. This commit also fixes a build error caused by a transitive dependency by adding it to the tools:overrideLibrary attribute.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. If the user enters a caption in French, has only one caption, and the language is the default (but not French), a dialog is shown to confirm if the language should be changed to "fr". This check is performed when the user proceeds to the next step in the upload wizard.

This feature is only enabled for Android API level 26 and above to maintain compatibility with older devices. This is achieved by adding a version check and overriding the library's minSdk in the manifest. This commit also fixes build errors caused by transitive dependencies by adding them to the tools:overrideLibrary attribute.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. This check is performed when the user proceeds to the next step in the upload wizard, but only on Android API 26+ to ensure backward compatibility.

This commit resolves multiple manifest merger build failures by overriding the `minSdkVersion` for the ML Kit library and its transitive dependencies (`genai-common`, `play-services-basement`). All ML Kit-related code is wrapped in SDK version checks to prevent runtime crashes on older devices.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. This check is performed when the user proceeds to the next step in the upload wizard, but only on Android API 26+ to ensure backward compatibility.

This commit resolves multiple manifest merger build failures by overriding the `minSdkVersion` for the ML Kit library and its transitive dependencies (`genai-common`, `play-services-basement`). All ML Kit-related code is wrapped in SDK version checks to prevent runtime crashes on older devices.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. This check is performed when the user proceeds to the next step in the upload wizard, but only on Android API 26+ to ensure backward compatibility.

This commit resolves multiple manifest merger build failures by overriding the minSdkVersion for the ML Kit library and its transitive dependencies (genai-common, play-services-basement). All ML Kit-related code is wrapped in SDK version checks to prevent runtime crashes on older devices.
Adds logging to the French caption detection feature to show the prompt and the model's response. This will help with debugging and monitoring the feature.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. This check is performed when the user proceeds to the next step in the upload wizard, but only on Android API 26+ to ensure backward compatibility.

This commit resolves build failures by:
1.  Adding the `-Xskip-metadata-version-check` Kotlin compiler flag to handle the metadata version incompatibility between the ML Kit library and the project.
2.  Updating the `AndroidManifest.xml` to override the `minSdkVersion` for all conflicting transitive dependencies.
3.  Making the `uploadMediaDetailAdapter` in `UploadMediaDetailFragment` public to allow access from `UploadActivity`.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. This check is performed when the user proceeds to the next step in the upload wizard, but only on Android API 26+ to ensure backward compatibility.

This commit resolves multiple build failures by:
1.  Adding the `-Xskip-metadata-version-check` Kotlin compiler flag to handle the metadata version incompatibility between the ML Kit library and the project.
2.  Updating the `AndroidManifest.xml` to override the `minSdkVersion` for all conflicting transitive dependencies.
3.  Adding the `lifecycle-runtime-ktx` dependency to enable `lifecycleScope` for coroutines.
4.  Making the `uploadMediaDetailAdapter` in `UploadMediaDetailFragment` public to allow access from `UploadActivity`.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. This check is performed when the user proceeds to the next step in the upload wizard, but only on Android API 26+ to ensure backward compatibility.

This commit resolves multiple build failures by:
1.  Adding the `-Xskip-metadata-version-check` Kotlin compiler flag to handle the metadata version incompatibility between the ML Kit library and the project.
2.  Updating the `AndroidManifest.xml` to override the `minSdkVersion` for all conflicting transitive dependencies.
3.  Adding the `lifecycle-runtime-ktx` dependency to enable `lifecycleScope` for coroutines.
4.  Making the `uploadMediaDetailAdapter` in `UploadMediaDetailFragment` public to allow access from `UploadActivity`.
5.  Fixing coroutine implementation by adding the correct import and accessing the ML model response correctly.
Adds a feature to detect if a caption is in French using the offline ML Kit Prompt API. This check is performed when the user proceeds to the next step in the upload wizard, but only on Android API 26+ to ensure backward compatibility.

This commit resolves multiple build failures by:
1.  Adding the `-Xskip-metadata-version-check` Kotlin compiler flag to handle the metadata version incompatibility between the ML Kit library and the project.
2.  Updating the `AndroidManifest.xml` to override the `minSdkVersion` for all conflicting transitive dependencies.
3.  Adding the `lifecycle-runtime-ktx` dependency to enable `lifecycleScope` for coroutines.
4.  Making the `uploadMediaDetailAdapter` in `UploadMediaDetailFragment` public to allow access from `UploadActivity`.
5.  Fixing coroutine implementation by adding the correct import and accessing the ML model response correctly.
6.  Adding a test-specific `AndroidManifest.xml` to resolve manifest merger failures during unit test builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants