this post was submitted on 10 Nov 2024
32 points (100.0% liked)

Linux

48315 readers
702 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
 

I'm running Jellyfin on a Debian-server in my home, and I have the associated media folders set up as samba shares so that I can transfer any new media from my laptop to the server through Dolphin (KDE file manager).

This has for the most part worked very well (except slow speeds), but I've had an issue recently where the files are not copied over properly. This resulted in glitches in for example music files that would stop playback. I checked the checksums of some of these files, and they were different from source. Seems like the glitchy files are missing some data, but at no point were I notified about this. It works fine after I removed the files and transferred again, and now the checksums match.

Is this a common issue with samba, or could it be a sign that my HDD is acting up?

top 27 comments
sorted by: hot top controversial new old
[–] [email protected] 9 points 1 week ago* (last edited 1 week ago) (1 children)

If you're transferring Linux to Linux then I really wouldn't recommend samba. Why not SFTP/Rsync? Compression, and error checking built in.

[–] [email protected] 1 points 1 week ago (1 children)

I think I will go with rsync for future transfers, but I would like for it to be browsable through the file browser still. Is there a better way than samba I should consider? I guess it is not an issue just keeping them as samba shares for that purpose?

[–] [email protected] 6 points 1 week ago (1 children)

The only thing Samba is really great for is interop with Windows. If that's not an issue, Dolphin can browse SFTP directly by adding it as a network share (you may need to setup a password-less key pair to avoid having to login). SSHFS is a similar option and works even if the client is totally naive (it just looks like any other mounted FS).

[–] [email protected] 1 points 1 week ago

Cheers, I'll look into setting up SFTP in Dolphin.

[–] [email protected] 5 points 1 week ago (1 children)

I don't know if samba has any error checking built in, so it may be even down to wonky network.

I would try doing a copy using rsync or ssh (sftp - you can use dolphin for that too), and see if that helps.

[–] [email protected] 1 points 1 week ago

rsync was written by one of the original Samba developers. I wonder if Tridge and co have any idea about how to shuffle data from A to B safely?

CIFS/SMB will only indicate received and not received and written. This is unlikely to be an issue.

I would start by proving that my network works properly, especially that dodgy cable with only wires 1,2,3,7 connected - because that's all 100Mb/s needs, or the solid core cable that runs for 150m with plugs at each end instead of sockets and drop leads.

[–] [email protected] 5 points 1 week ago (1 children)

Apart from copying the files again after comparing the checksums what else have you done to troubleshoot the issue? Have you looked at logs, run fsck or anything?

[–] [email protected] 1 points 1 week ago (1 children)

I am unsure what logs to look at for this and have not done any filesystem check on the harddrive as I am unfamiliar with these tools. It's an external Seagate HDD with an ext4 filesystem.

[–] [email protected] 2 points 1 week ago

If you have time to not use the drive, you could use badblocks to test it. I usually use it with -svn as the options. s=Show progress, v=Verbose Mode, and n=Non destructive read/write mode,

[–] [email protected] 4 points 1 week ago (1 children)

"Is this a common issue with samba" - no.

Samba shuffles rather a lot of data, quite happily. You have not given us an exhaustive description of the shoddy wiring, dodgy switches and wonky configuration that makes up your network. If it was perfect, you would not be posting here.

There is one snag with CIFS (Samba follows MS's standards and ironically, I think that CIFS is now renamed back to SMB) that I am aware of, so SMB ... snag: SMB will indicate that a chunk of data has been received successfully but not that it has been written to disc successfully. NFS will notify that a chunk of data has been written to disc.

The difference is subtle but if there is not a battery backed RAID involved then SMB/CIFS can lose data if the system restarts part way through a write.

Your issue is probably hardware related. Test your network with say iperf3. Have a look at network stats. Don't rely on cargo cult bollocks - do some investigations. Nowadays we have nearly all the tools as open source to do the entire job - we did not have that 30 years ago. Grab wireshark, nmap, mtr and the rest and get nerdy (or hire me to do it - don't do that please!)

[–] [email protected] 1 points 1 week ago

Samba shuffles rather a lot of data, quite happily. You have not given us an exhaustive description of the shoddy wiring, dodgy switches and wonky configuration that makes up your network. If it was perfect, you would not be posting here.

The network is by no means ideal. I am transferring from a laptop on WiFi to a server on WiFi located some distance from the WAP. If I owned the place I would do a rewire, but for now it's the best I can do. I think I assumed that there would be error-checking involved when copying. Since following the advice here of using rsync i stead, I have found that files tend to fail in bunches and I need to rerun several times for it to actually complete. Am I right to assume that comes down to packet loss due to poor signal?

Your issue is probably hardware related. Test your network with say iperf3. Have a look at network stats. Don’t rely on cargo cult bollocks - do some investigations. Nowadays we have nearly all the tools as open source to do the entire job - we did not have that 30 years ago. Grab wireshark, nmap, mtr and the rest and get nerdy (or hire me to do it - don’t do that please!)

This is above my skill level for now, but I'm adding it to my notes to go back to. I have some ambition of upping my network knowledge in the coming year, and being able to do use such tools to troubleshoot would be great.

[–] [email protected] 4 points 1 week ago (1 children)

It is not common at all no. Is your network between the 2 machines wired ethernet or wifi?

[–] [email protected] 2 points 1 week ago

Both machines are WiFi-connected.

[–] [email protected] 3 points 1 week ago (3 children)

rysnc might be a faster and more reliable option. It can compress the files for transfer and does checksums after the transfer is complete

I used something like this to transfer 12 TB from offsite to onsite with zero failures


rsync -arvzip --progress /path/to/host /path/to/destination

You can set up a screen and let this run in the background all the time

[–] [email protected] 4 points 1 week ago (2 children)

and since op is using kde, they can use a konsole with dolphin to drag and drop /path/to/host and /path/to/destination as well.

[–] [email protected] 3 points 1 week ago (1 children)

Thanks for sharing! I’m a pure headless Linux user, so I don’t know much about desktop environments

[–] [email protected] 2 points 1 week ago

i'm headless for servers too; but i'm foolish enough to use the x server as my daily driver. lol

[–] [email protected] 2 points 1 week ago

Oh, I didn't know that. Neat!

[–] [email protected] 2 points 1 week ago (1 children)

Thanks! Glad to know rsync includes check after transfer, as I've just recently used it to backup everything on these drives to another hard drive that will not always be spinning. But I did not consider using it to transfer new media onto these hard drives.

I'll try to use it to resync the files that were acting up.

[–] [email protected] 1 points 1 week ago (1 children)

Depending on your file structure, you could probably keep this running all the time so you don’t have to manually intervene in the future

[–] [email protected] 1 points 1 week ago (1 children)

How would I achieve that? With cron?

[–] [email protected] 2 points 1 week ago
  1. Open a screen
  2. Start an rsync job to maintain parity between source and destination
  3. Exit the screen, but keep it running
  4. Now rsync will be running in the background until you kill it

You can reattach the screen whenever you want to check on status, change parameters or kill it

[–] [email protected] 2 points 1 week ago

I tried to resync now, and had to pass the -c flag to make sure it checked the cheksums to see if they should be updated. Then it worked. Looks like that does not affect the after-transfer checksum check though, so that's good (from documentation):

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer lqDoes this file need to be updated?rq check. 
[–] [email protected] 2 points 1 week ago

Check if you get big memory spikes when transferring, or OOM entries in dmesg.

Had some trouble with a new samba install on Debian causing OOM issues recently (this was admittedly in LXC). Resulted in files apparently transfered but were not. This was due to optimistic locks, which are apparently poorly implemented (according to the 2-year old open bug) but on by default. I haven't done due diligence to say what's the best solution, I just turned oplocks off.

https://bugzilla.samba.org/show_bug.cgi?id=15261

[–] [email protected] 2 points 1 week ago (1 children)

Is the "Dolphin" that you're using the same dolphin that KDE uses as a gui file manager?

[–] [email protected] 1 points 1 week ago (1 children)

Yes, I should have specified that.

[–] [email protected] 1 points 1 week ago

do you get the same problem with something like rsync?