[Bug]: Unable to restart failed uploads

[mnalis]

Summary

I'm unable to upload pictures that I've taken (while in limited connection mode).

Related to #5231 ?

Steps to reproduce

1. enable "Limited connection mode" while on expensive mobile data plan
2. take few dozen pictures
3. few days later, when in range of free WiFi, disable limited connection mode.
4. Notice that nothing seems to happen (i.e. pictures don't start uploading)
5. try to pause and restart picture
6. sometimes it works (i.e. picture starts uploading) but most often error is displayed and it is impossible to upload picture, no many how many times you retry

Expected behaviour

I would expect in step 6. that I am able to upload pictures that I've painstakingly taken, described and categorized!

also (less importantly) one would expect that in step 4. queued uploads would start automatically

Actual behaviour

Unable to upload pictures in any way I can think of. Fear complete data and effort loss, very demotivating.

Device name

Samsung Galaxy S23+

Android version

Android 14 (OneUI 6.1)

Commons app version

5.0.1~af028cbdd (latest F-droid)

Device logs

CommonsAppLogs.zip

Screen-shots

small_Screen_Recording_20240624_185252_Commons.mp4

Would you like to work on the issue?

None

[sivaraam]

Sorry about this experience @mnalis. Thank you for taking the time to communicate the feedback, though. It is very much appreciated.

You seem to have a huge list of pending uploads. I'm wondering if the failure is possible due to that. 🤔

While the changes done during previous GSoC should've improved the status quo in some respect. I think there are still some areas which still need improvements.

Fortunately, our GSoC project this year also focuses on upload queue management and should hopefully improve your situation! If the changes in that PR are ready to some extent, it would be helpful if you could give early feedback given that you are willing to try it.

cc @kanahia1 @nicolas-raoul @RitikaPahwa4444

[kanahia1]

Thanks @mnalis for reporting this issue. We are currently working on making upload more reliable. So we have removed the limited connection mode. And There will be separate screen for the uploads where you can resume/pause uploads anytime you want. Here is pull request for the current progress #5752

I will try to resume/pause after many days with the PR I am currently working on to simulate the same effect.

[nicolas-raoul]

@mnalis Would you able to install an APK (to test Kanahia's development branch)?

[mnalis]

You seem to have a huge list of pending uploads. I'm wondering if the failure is possible due to that. 🤔

@sivaraam Perhaps, but I do not think so - after all even when I later I try to start only single transfer (with nothing else running), it also fails to upload. The fact it is huge list does increase my aggravation at data loss significantly, though 😭 . But thank you for your condolences!

I will try to resume/pause after many days with the PR I am currently working on to simulate the same effect.

@kanahia1 I think that it may be on the way to the root cause. Because my earlier tests (done on same day) after previous upload-reliability GSoC didn't seem to readily reproduce the problem. Yet this rears its ugly head again, just like it did last time, and IIRC the pictures were also waiting for several days until I could get WiFi back then in #5231

I.e. this error message in particular:

2024-06-24 18:50:54,353 [ERROR] [file-logging-thread-1] [UploadWorker] : java.io.FileNotFoundException: /data/user/0/fr.free.nrw.commons/cache/CommonsContributions/559b0a6d-76a3-48ba-9e04-bdb4c2eb8aea.jpg: open failed: ENOENT (No such file or directory)

Seems to indicate that picture is no longer present there. And I suppose it was present there at some point in the past, or why would CommonsApp have that path otherwise? And given that directory name contains /cache/ it might be highly likely that app or Android OS or whoever decided to "clean it up" (as it is just cache which by definition can be easily automatically re-created, and not real data!) thus leading up to the problem.

If that is indeed the case, the solution would be to keep all important data (pictures, metadata... i.e. anything that cannot be automatically re-generated without user interaction) out of /cache/ directory and store it in some permanent-data directories instead (like /files/xxxx).
(note that there might also be other cases when Android decides to randomly delete files even from non-cache directories depending on APIs used. e.g. I've had issues with DownloadManager for example see e.g. MarcusWolschon/osmeditor4android#2359)

@mnalis Would you able to install an APK (to test Kanahia's development branch)?

@nicolas-raoul if somebody builds .apk for me, yes, I can download and test it. Note however that:

• it should have different app name (as I do not want to uninstall original Commons app, and that one has F-droid signature, so I cannot just upgrade over it)
• I'm on low information diet (about ~50MB/day) until 19 July, so I'm somewhat limited in what I can try until then (but I can try taking pictures and letting them "sit" to see if they will expire)

[nicolas-raoul]

I just got this problem, even though this area has changed a lot.

1. Launch a multi- upload.
2. Press the pause button.
3. Launch several other multi- uploads.
4. Wait for 10 hours overnight.
5. Open the app again.

Expected: Pressing "play" starts the uploads
Actually:

/data/user/0/fr.free.nrw.commons/ cache/CommonsContributions/ 75563893-bdc3-40da-affb-8293e3ab4 d88.jpg: open failed: ENOENT (No such file or directory)

Pressing the "Refresh" buttons has no effect.

[nicolas-raoul]

Do we temporarily store files in a path which gets "cleaned" by the Android OS once in a while?

If yes, any better place we can use? Maybe the app's own storage or even database?