All files uploaded via Auto Upload are modifyed - SHA256 is different from the original

[GAS85]

⚠️ Before posting ⚠️

• This is a bug, not a question or an enhancement.
• I've searched for similar issues and didn't find a duplicate.
• I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
• I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

1. Setup auto upload, ensure full access
   
2. Upload photos, Videos
3. Go to the Original Files folder and execute find -type f -exec sha256sum '{}' \; > ~/2024.hash.phone to calculate original hash sums.
4. Upload this file to the Server
5. Go to the server folder with uploaded files via ssh and run sha256sum -c 2023.hash.phone to check SHA 256 sums. Add | grep OK | wc -l to the command to see amount of OK files.
6. Be shocked.

sha256sum: WARNING: 303 computed checksums did NOT match
10 # This is amount of files that checksums are matched

11. Upload file via WebUI and see that SHA256 is different from what was uploaded via Android App, but it is matching original file SHA256. On the Left File from Cloud, on the Right fresh uploaded via WebUI:
12. Goto Memories and check EXIF. In original File I have a Location saved, in uploaded via UI it was erased. On the Right File from Cloud, on the Left fresh uploaded via WebUI:

😢 I lost tons of GPS locations in Photos that I do not have in Original files anymore.

Expected behaviour

Files are never modified as it was before December 2023. I do check Maps app and have no locations saved since December!
I have auto update on in google Play, so seems around 3.27.0 version was broken.

Actual behaviour

Never expected that file will be modified in any matter, but seems related to #8622 and #8646, especially #12973

Android version

14

Device brand and model

Samsung Galaxy S21 FE

Stock or custom OS?

Stock

Nextcloud android app version

3.29.0

Nextcloud server version

27.1.9

Using a reverse proxy?

No

Android logs

No response

Server error logs

No related error

Additional information

JPEG and MP4 files are affected

[TippyTurtle]

To "fix" (er work around) this on a Pixel 8 Pro running Android 14 do the following:
Settings-->Apps-->Special App access (at the bottom)-->All Files Access-->Nextcloud-->Turn on "Allow access to manage all files"
...in addition to the standard "App Permissions" being set correctly.

I have verified on my phone the sha256sum are the same:
Pixel 8-->Nextcloud App-->InstantUpload-->NextCloud 28.0.5-->Nextcloud Ubuntu File Sync-->sha256sum on local file system
...vs...
Pixel 8-->MTP USB 3 connection-->sha256sum over the MTP mount

I believe Android is "cleaning" your private data before it gives it to any app...unless the app has "Special App access" which is not in the normal "App Permissions"

One example:
ZZZZZ@i5BRIX:~$ sha256sum ~/Nextcloud/InstantUpload/Camera/PXL_20240516_152713497.jpg

3b25ea8365bbb327ed7ba3c25dac78af3ea6fc6deb305b0bac10f616c02cd8c4 /home/ZZZZZ/Nextcloud/InstantUpload/Camera/PXL_20240516_152713497.jpg

ZZZZZ@i5BRIX:~$ sha256sum "/run/user/1000/gvfs/mtp:host=Google_Pixel_8_Pro_39080DLJG000NX/Internal shared storage/DCIM/Camera/PXL_20240516_152713497.jpg"

3b25ea8365bbb327ed7ba3c25dac78af3ea6fc6deb305b0bac10f616c02cd8c4 /run/user/1000/gvfs/mtp:host=Google_Pixel_8_Pro_39080DLJG000NX/Internal shared storage/DCIM/Camera/PXL_20240516_152713497.jpg
ZZZZZ@i5BRIX:~$

[joshtrichards]

What do the ten files that matched have in common?

Were they created after the permission change?

Put another way, now that access is granted, if you take a new photo does it also differ or is it a match?

[TippyTurtle]

What do the ten files that matched have in common?

Were they created after the permission change?

Put another way, now that access is granted, if you take a new photo does it also differ or is it a match?

I'm not the original poster, but my guess is that the 10 matching files had no metadata to start...like a TXT file, or Android doesn't know how to strip the metadata for those type files.

In my experiments just now....

All new photo's (after make the settings change "Special App access-->All Files Access") all new files come through the WHOLE pipeline with matching sha256sum.

Old Files:

(After deleting the copy of all the files on my NextCloud Server) On my phone, if I reconfigure the "Auto upload" to "Also upload existing files" and it goes to town, the newly uploaded, old photo's, have their exif data (it wasn't stripped this time). In other words, the files on the phone were not touched by anything, including the Android Nextcloud App.

My Guess: the normal (many versions) old Android API call to get files will now strip all the files of meta data before the file is given to the calling app...probably even MS Word Doc's, etc...unless that Android app has "Special App access-->All Files Access"

[joshtrichards]

My Guess: the normal (many versions) old Android API call to get files will now stripping all the files of meta data before the file is given to the calling app...probably even MS Word Doc's...unless that Android app has "Special App access (at the bottom)-->All Files Access"

No need for guessing. That's expected behavior. Location data for media files is not provided to the Nextcloud Android client unless that permission is granted interactively by the user. That's how Android OS works (these days):

https://developer.android.com/training/data-storage/shared/media#media-location-permission

To "fix" (er work around) this on a Pixel 8 Pro running Android 14 do the following:
Settings-->Apps-->Special App access (at the bottom)-->All Files Access-->Nextcloud-->Turn on "Allow access to manage all files"

The original reporter has already done this from what I can tell. :-)

Also, this isn't a workaround. This is precisely why the client asks for those permissions from the user at installation time. They also have the option of choosing the lesser one (which doesn't provide location data to Nextcloud).

In any case, I'm interested in the reporter's response rather than guessing what the situation is with those ten files. :-)

[TippyTurtle]

My Guess: the normal (many versions) old Android API call to get files will now stripping all the files of meta data before the file is given to the calling app...probably even MS Word Doc's...unless that Android app has "Special App access (at the bottom)-->All Files Access"

No need for guessing. That's expected behavior. Location data for media files is not provided to the Nextcloud Android client unless that permission is granted interactively by the user. That's how Android OS works (these days):

https://developer.android.com/training/data-storage/shared/media#media-location-permission

To "fix" (er work around) this on a Pixel 8 Pro running Android 14 do the following:
Settings-->Apps-->Special App access (at the bottom)-->All Files Access-->Nextcloud-->Turn on "Allow access to manage all files"

This isn't a workaround. This is precisely why the client asks for those permissions from the user at installation time. They also have the option of choosing the lesser one (which doesn't provide location data to Nextcloud).

In any case, I'm interested in the reporter's response rather than guessing what the situation is with those ten files. :-)

Agreed. It's expected behavior, but what's unexpected is that the app doesn't request these permissions during install. It does ask for the normal permissions.

It feels like a heavy lift for a normal user because the permissions required are not under the normal app permissions.

They're under special permissions would you have to go digging for in your system settings after you install the app.

[GAS85]

As you see, I grant full access and even before to open a ticket I cancel it, granted again and upload few files via app. Somehow it is not working for me. May be Samsung update caused this, because before December everything was fine.

[TippyTurtle]

As you see, I grant full access and even before to open a ticket I cancel it, granted again and upload few files via app. Somehow it is not working for me. May be Samsung update caused this, because before December everything was fine.

Yes those are the "normal" permissions that that app requests when you install it.

You need to additionally turn on:

Settings-->Apps-->Special App access (at the bottom)-->All Files Access-->Nextcloud-->Turn on "Allow access to manage all files"

Although on Samsung the navigation is probably different. When you get to the right setting it will look similar to:

[joshtrichards]

Agreed. It's expected behavior, but what's unexpected is that the app doesn't request these permissions during install. It does ask for the normal permissions.

It feels like a heavy lift for a normal user because the permissions required are not under the normal app permissions.

I would agree it would be heavy lifting, but we do request it at install time. :-)

I just uninstalled and reinstalled:

Selecting the right most option brings up the expected screen:

Which can either be turned on or not:

[joshtrichards]

@GAS85 Don't want to lose your thread:

What is the nature of the ten files that passed the SHA checks? Were they newer media than all the others? Did the originals already lack location data? Etc. I'm trying to figure out why some of your media were okay. It might help determine why your other media are not okay.

[TippyTurtle]

Agreed. It's expected behavior, but what's unexpected is that the app doesn't request these permissions during install. It does ask for the normal permissions.
It feels like a heavy lift for a normal user because the permissions required are not under the normal app permissions.

I would agree it would be heavy lifting, but we do request it at install time. :-)

I just uninstalled and reinstalled:

Selecting the right most option brings up the expected screen:

Which can either be turned on or not:

You're exactly right. I hadn't reinstalled in a long time....I guess there is an odd corner case when this happens. Probably not something the app can fix.

For whatever reason I ended up in this state...and had to hunt down this odd additional setting, when an uninstall/reinstall would have solved it.

Thanks for the clarification.

[GAS85]

@GAS85 Don't want to lose your thread:

What is the nature of the ten files that passed the SHA checks? Were they newer media than all the others? Did the originals already lack location data? Etc. I'm trying to figure out why some of your media were okay. It might help determine why your other media are not okay.

I double check and they ether do not have location tag, or were uploaded via web UI.

[GAS85]

You need to additionally turn on:

Settings-->Apps-->Special App access (at the bottom)-->All Files Access-->Nextcloud-->Turn on "Allow access to manage all files"

Wow, that was not easy to find. Seem now files uploaded with the same sha, so no modifications to exif.

Totally agree that this must be highlighted to user. I didn't get this prompt after install the app.

[QuentinLemCode]

Hello
I encountered the same issue on S23 Ultra, with the app downloaded from Play store.
When I launched the app, it asked for the permission to access media but not full access.
The special permission to allow access to all files was unavailable (even in the "special access" settings : nextcloud was not in the list).
I uninstalled the app and installed the one from the github release page and I had the proper permission pop-up asking for all files access and then it worked.