Ask Slashdot: Is It Time To Replace File Systems?

DidgetMaster writes: Hard drive costs now hover around $20 per terabyte (TB). Drives bigger than 20TB are now available. Fast SSDs are more expensive, but the average user can now afford these in TB capacities as well. Yet, we are still using antiquated file systems that were designed decades ago when the biggest drives were much less than a single gigabyte (GB). Their oversized file records and slow directory traversal search algorithms make finding files on volumes that can hold more than 100 million files a nightmare. Rather than flexible tagging systems that could make searches quick and easy, they have things like “extended attributes” that are painfully slow to search on. Indexing services can be built on top of them, but these are not an integral part of the file system so they can be bypassed and become out of sync with the file system itself.

It is time to replace file systems with something better. A local object store that can effectively manage hundreds of millions of files and find things in seconds based on file type and/or tags attached is possible. File systems are usually free and come with your operating system, so there seems to be little incentive for someone to build a new system from scratch, but just like we needed the internet to come along and change everything we need a better data storage manager.
See Didgets for an example of what is possible.
In a Substack article, Didgets developer Andy Lawrence argues his system solves many of the problems associated with the antiquated file systems still in use today. “With Didgets, each record is only 64 bytes which means a table with 200 million records is less than 13GB total, which is much more manageable,” writes Lawrence. Didgets also has “a small field in its metadata record that tells whether the file is a photo or a document or a video or some other type,” helping to dramatically speed up searches.

Do you think it’s time to replace file systems with an alternative system, such as Didgets? Why or why not?

Read more of this story at Slashdot.

Ask Slashdot: How Many Files Are on Your Computer?

With some time on their hands, long-time Slashdot reader shanen began exploring the question: How many files does my Windows 10 computer have?

But then they realized “It would also be interesting to compare the weirdness on other OSes…”

Here are the two data points in front of me:

(1) Using the right click on properties for all of the top-level folders on the drive (including the so-called hidden folders), it quickly determined that there are a few hundred thousand files in those folders (and a few hundred thousand subfolders). That’s already ridiculous, but the expected par these days. The largest project I have on the machine only has about 3,000 files, and that one goes back many years… (My largest database only has about 5,000 records, but it’s just a few files.)

(2) However, I also decided to take a look with Microsoft’s malicious software removal tool and got a completely different answer. For twisted grins, I had invoked the full scan. It’s still running a day later and has already passed 10 million files. Really? The progress bar indicates about 80% finished? WTF?

Obviously there is some kind of disagreement about the nature of “file” here. I could only think of one crazy explanation, but my router answered “No, the computer is not checking all of the files on the Internet.” So I’ve already asked the specific question in three-letter form, but the broader question is about the explosive, perhaps even cancerous, “population growth” of files these days.

Maybe we can all solve this mystery together. So use the comments to share your own answers and insights.

How many files are on your computer?

Read more of this story at Slashdot.