this post was submitted on 29 Oct 2023
251 points (96.0% liked)

Linux

48165 readers
617 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by AlpΓ‘r-Etele MΓ©der, licensed under CC BY 3.0

founded 5 years ago
MODERATORS
 

Dust is a rewrite of du (in rust obviously) that visualizes your directory tree and what percentage each file takes up. But it only prints as many files fit in your terminal height, so you see only the largest files. It's been a better experience that du, which isn't always easy to navigate to find big files (or atleast I'm not good at it.)

Anyway, found a log file at .local/state/nvim/log that was 70gb. I deleted it. Hope it doesn't bite me. Been pushing around 95% of disk space for a while so this was a huge win πŸ‘

top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 108 points 1 year ago (4 children)

I think something might be wrong with your Neovim if it aggregated 70 gigs of log files.

[–] [email protected] 55 points 1 year ago (1 children)

don't worry, they've just been using neovim for 700 years, it'll be alright

[–] [email protected] 8 points 1 year ago (1 children)

Sure, that's also a possibility. I'd be interested in their time machine though.

[–] [email protected] 29 points 1 year ago (2 children)

So I found out that qbittorrent generates errors in a log whenever it tries to write to a disk that is full...

Everytime my disk was full I would clear out some old torrents, then all the pending log entries would write and the disk would be full again. The log was well over 50gb by the time I figured out that i'm an idiot. Hooray for having dedicated machines.

[–] [email protected] 29 points 1 year ago

That's not entirely your fault; that's pathological on the part of the program.

[–] [email protected] 13 points 1 year ago

I once did something even dumber. When I was new to Linux and the CLI, I added a recursive line to my shell config that would add it self to the shell config. So I pretty much had exponential growth of my shell config and my shell would take ~20 seconds to start up before I found the broken code snippet.

load more comments (2 replies)
[–] [email protected] 87 points 1 year ago (13 children)

ncdu is the best utility for this type of thing. I use it all the time.

[–] [email protected] 18 points 1 year ago

I install ncdu on any machine I set up, because installing it when it's needed may be tricky

[–] [email protected] 16 points 1 year ago* (last edited 1 year ago)

Try dua. It's like ncdu but uses multiple threads so it's a lot faster., especially on SSDs.

load more comments (11 replies)
[–] [email protected] 54 points 1 year ago (1 children)

Came in expecting a story of tragedy, congrats. πŸŽ‰

[–] [email protected] 2 points 1 year ago (1 children)

But did he even look at the log file? They don't get that big when things are running properly, so it was probably warning him about something. Like "Warning: Whatever you do, don't delete this file. It contains the protocol conversion you will need to interface with the alien computers to prevent their takeover."

[–] [email protected] 8 points 1 year ago

PTSD from the days long ago when X11 error log would fill up the disk when certain applications were used.

[–] [email protected] 54 points 1 year ago (2 children)

Try ncdu as well. No instructions needed, just run ncdu /path/to/your/directory.

[–] [email protected] 11 points 1 year ago

If you want to scan without crossing partitions, run with -x

[–] [email protected] 38 points 1 year ago (6 children)

I usually use something like du -sh * | sort -hr | less, so you don't need to install anything on your machine.

[–] [email protected] 8 points 1 year ago

Same, but when it's real bad sort fails πŸ˜… for some reason my root is always hitting 100%

I usually go for du -hx | sort -h and rely on my terminal scroll back.

[–] [email protected] 5 points 1 year ago (1 children)

dust does more than what this script does, its a whole new tool. I find dust more human readable by default.

[–] [email protected] 2 points 1 year ago (1 children)

Maybe, but I need it one time per year or so. It is not a task for which I want to install a separate tool.

load more comments (1 replies)
[–] [email protected] 4 points 1 year ago* (last edited 1 year ago) (2 children)

Almost the same here. Well, du -shc *|sort -hr

I admin around three hundred linux servers and this is one of my most common tasks - although I use -shc as I like the total too, and don't bother with less as it's only the biggest files and dirs that I'm interested in and they show up last, so no need to scrollback.

When managing a lot of servers, the storage requirements when installing extra software is never trivial. (Although our storage does do very clever compression and it might recognise the duplication of the file even across many vm filesystems, I'm never quite sure that works as advertised on small files)

[–] [email protected] 3 points 1 year ago (1 children)

I admin around three hundred linux servers

What do you use for management? Ansible? Puppet? Chef? Something else entirely?

[–] [email protected] 3 points 1 year ago (1 children)

Main tool is Uyuni, but we use Ansible and AWX for building new vms, and adhoc ansible for some changes.

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago) (3 children)

Interesting; I hadn't heard of Uyuni before. Thanks for the info!

load more comments (3 replies)
[–] [email protected] 2 points 1 year ago (4 children)

We'd use du -xh --max-depth=1|sort -hr

load more comments (4 replies)
[–] [email protected] 2 points 1 year ago

I'd say head -n25 instead of less since the offending files are probably near the top anyway

load more comments (2 replies)
[–] [email protected] 24 points 1 year ago

So like filelight?

[–] [email protected] 21 points 1 year ago (1 children)
[–] [email protected] 3 points 1 year ago

Yeah I got turned onto ncdu recently and I’ve been installing it on every vm I work on now

[–] [email protected] 15 points 1 year ago (1 children)

check out dua. I usually use it in interactive most which lets you navigate through the file system with visual representations of total dir/file size.

Here is a screenshot randomly found from the github issues

screenshot

I also recently found this gui program called k4dirstat buried in the repos. There are a few more modern options but this one blows them all out of the park.

Screenshot from the github repo:

screenshot

Too bad they used such an ugly configuration for the screenshot.. It allows you to modify the visualization to look better and display information differently. Anyway just thought I'd share as the project is old and little known.

[–] [email protected] 4 points 1 year ago

thanks for sharing a screenshot of ncdu, should help others discover it

for the visualization itself IMHO Disk Usage Analyzer gives aesthetically pleasing results, not a fan of the UX but it works well enough to identify efficiently large files or directories

[–] [email protected] 9 points 1 year ago (1 children)

A 70gb log file?? Am I misunderstanding something or wouldn't that be hundreds of millions of lines

[–] [email protected] 9 points 1 year ago

I've definitely had to handle 30gb plain text files before so I am inclined to believe twice as much should be just as possible

[–] [email protected] 8 points 1 year ago

You guys aren't using du -sh ./{dir1,dir2} | sort -nh | head?

[–] [email protected] 7 points 1 year ago* (last edited 1 year ago)

Maybe other tools support this too but one thing I like about xdiskusage is that you can pipe regular du output into it. That means that I can run du on some remote host that doesn't have anything fancy installed, scp it back to my desktop and analyze it there. I can also pre-process the du output before feeding it into xdiskusage.

I also often work with textual du output directly, just sorting it by size is very often all I need to see.

[–] [email protected] 6 points 1 year ago (3 children)

I miss WinDirStat for seeing where all my hard drive space went. You can spot enormous files and folders full of ISOs at a glance.

For bit-for-bit duplicates (thanks, modern DownThemAll), use fdupes.

[–] [email protected] 5 points 1 year ago (1 children)
[–] [email protected] 2 points 1 year ago

Qdirstat will not size its damn rectangles properly in Mint. Massive empty voids for no discernible reason.

Filelight is just objectively worse than a grid-based overview.

[–] [email protected] 5 points 1 year ago

If WizTree is available on Linux then I highly recommend it over all other alternatives.

It reads straight from the table and is done within a couple of seconds.

[–] [email protected] 4 points 1 year ago

Filelight on linux

Squirreldisk on windows

Both libre

[–] [email protected] 4 points 1 year ago

I use gdu and never had any issues like that with it

[–] [email protected] 2 points 1 year ago

Yeah, it helped me unblock my server where I ran out of space

load more comments
view more: next β€Ί