datahoarder
- DVDs, Blu-rays, and VHS tapes: How long does physical media last?qz.com DVDs, Blu-rays, and VHS tapes: How long does physical media last?
The exact lifespan of physical media is hard to pin down
- My name is lars and I’m a hoarder. Too.www.pcmag.com Firefox Power User Keeps 7,400+ Browser Tabs Open for 2 Years
Hazel doesn't close tabs because it's 'like a trip down memory lane.' Mozilla says it has some profile and tab group features coming later this year that might help her organize those memories.
cross-posted from: https://programming.dev/post/13631943
> Firefox Power User Keeps 7,400+ Browser Tabs Open for 2 Years
- [Request] Any Guides to FFMPEG, Transcoding, Codecs, and Metadata?
cross-posted from: https://leminal.space/post/6179210
> I have a collection of about ~110 4K Blu-Ray movies that I've ripped and I want to take the time to compress and store them for use on a future Jellyfin server. > > I know some very basics about
ffmpeg
and general codec information, but I have a very specific set of goals in mind I'm hoping someone could point me in the right direction with: > > 1. Smaller file size (obviously) > 2. Image quality good enough that I cannot spot the difference, even on a high-end TV or projector > 3. Preserved audio > 4. Preserved HDR metadata > > In a perfect world, I would love to be able to convert the proprietary HDR into an open standard, and the Dolby Atmos audio into an open standard, but a good compromise is this. > > Assuming that I have the hardware necessary to do the initial encoding, and my server will be powerful enough for transcoding in that format, any tips or pointers? - Question about price per TB
So it's been a a few years since I've bought hard drives for my little home server and wanted to get a bead on what's the target on dollar to TB in the post Covid world. Thanks!
- Samsung boosts 1Tb V-NAND bit density by 50 percentwww.eenewseurope.com Samsung boosts 1Tb V-NAND bit density by 50 percent
Groundbreaking double-stack V-NAND cell structure maximizes fabrication productivity through advanced ‘channel hole etching’ technology. %
- Looking for all the three books in the Moving Caste series
I'm looking for EPUB formats of all the three books.
- Effectively manage and store books/comics/manga
Hello, I'm wondering what do you guys use and recommend for efficient book, comic, manga and lightnovel file management, tagging, directory structures and automated tools for all that.
My collection is mostly made from humble bundle book bundles, for getting tags into comics I use comictagger and as for file structure, it was mostly just me just putting something to separate the books.
I wan't to hear you guys input because most of you are a lot more efficient or have a lot more experience in saving big ammounts of data, and I wan't to make my process as painless and future proof as possible as my collection starts to grow.
Edit: I use linux so software like comicrack which I heard a lot about isn't really accessible to me. The files also need to be accessible to my kavita server.
- How to store digital files for posterity? (hundreds of years)
How to store digital files for posterity? (hundreds of years)
I have some family videos and audios and I want to physically save them for posterity so that it lasts for periods like 200 years and more. This allows great-grandchildren and great-great-grandchildren to have access.
From the research I did, I found that the longest-lasting way to physically store digital content is through CD-R gold discs, but it may only last 100 years. From what I researched, the average lifespan of HDs and SSDs is no more than 10 years.
I came to the conclusion that the only way to ensure that the files really pass from generation to generation is to record them on CDs and distribute them to the family, asking them to make copies from time to time.
It's crazy to think that if there were suddenly a mass extinction of the human species, intelligent beings arriving on Earth in 1000 years would probably not be able to access our digital content. While cave paintings would probably remain in the same place.
What is your opinion?
- Whats do you use in your yt-dlp config for the best downloads?
I formatted my PC recently and I'm reinstalling some stuff I forgot to make a backup of settings like yt-dlp so I started searching for what is a good config to download the best mp4 quality and found some interesting setups so I figured I'd make a thread for people to share what they use.
Here's the most best setup I found so far which downloads a 1080p mp4 with the filename and includes metadata, english subtitles, and chapters if available:
yt-dlp -f 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' -S vcodec:h264 --windows-filenames --restrict-filenames --write-auto-subs --sub-lang "en.*" --embed-subs --add-metadata --add-chapters --no-playlist -N 4 -ci --verbose --remux-video "mp4/mkv" URL
Ideally it would also mark the sponserblock section and download to a specified folder
- The Internet Archive Just Backed Up an Entire Caribbean Island
@datahoarder The Internet Archive Just Backed Up an Entire Caribbean Island https://www.wired.com/story/internet-archive-backed-up-aruba-caribbean-island/
- SAS hardware questions
Hi, anyone could point me at an ELI5 about SAS hardware? I'd like to assemble a NAS using an old HP Z200, I want SAS because I'd get also a tape drive for backups and I cannot find SATA tape drives. For example, is a Dell Perc H310 pci-e card good for me? Can I avoid hardware RAID?
- Archiving Lemmy instances
While clicking through some random Lemmy instances, I found one that's due to be shut down in about a week — https://dmv.social. I'm trying to archive what I can onto the Wayback Machine, but I'm not sure what the most efficient way to go about it is.
At the moment, what I've been doing is going through each community and archiving each sort type (except the ones under a month, since the instance was locked a month ago) with capture outlinks enabled. But is there a more efficient way to do it? I know of the Internet Archives save from spreadsheet tool, which would probably work well, but I don't know how I'd go about crawling all the links into a sitemap or csv or something similar. I don't have the know-how to setup a web crawler/spider.
Any suggestions?
- SSD hides contents after a few days
Seems the SSD sometimes heats up and the content disappears from the device, mostly from my router, sometimes from my laptop. Do you know what I should configure to put the drive to sleep or something similar to reduce the heat?
I'm starting up my datahoarder journey now that I replaced my internal nvme SSD.
It's just a 500GB one which I attached to my d-link router running openwrt. I configured it with samba and everything worked fine when I finished the setup. I just have some media files in there, so I read the data from jellyfin.
After a few days the content disappears, it's not a connection problem from the shared drive, since I ssh into the router and the files aren't shown. I need to physically remove the drive and connect it again. When I do this I notice the somewhat hot. Not scalding, just hot.
I also tried this connecting it directly to my laptop running ubuntu. In there the drive sometimes remains cool and the data shows up without issue after days. But sometimes it also heats up and the data disappears (this was even when the data was not being used, i.e. I didn't configure jellyfin to read from the drive)
I'm not sure how I can be sure to let the ssd sleep for periods of time or to throttle it so it can cool off. Any suggestion?
- What's your most treasured data?
What in your hoard do you treasure the most? I imagine to a lot of us it is photos and videos of our families, which I'd love to hear about, but also interested in rare bits of media or information that makes your collection unique.
- Does anyone have an archive of the @ErmnMusk Twitter account?
Last year Elon Musk accidentally revealed that he has a burner account on Twitter called @ErmnMusk.
Now that account is gone.
I'm looking for an archive: its tweets, its likes, anything and everything. Does anyone know where to find?
- A Data Hoarder, I Am Not
So I’ve been consolidating all of my storage and removing all the duplicates and junk files.
In actual physical storage, this was spread across 12TB worth of hard drives, all partially full.
After everything was said and done, I’m using 1.3TB of space if you don’t include games. ¯\\\(ツ)/¯
This is stuff dating back to 2015. Sometimes it’s actually worth it to just clean up your junk files.
- DVD’s New Cousin Can Store More Than a Petabitspectrum.ieee.org DVD’s New Cousin Can Store More Than a Petabit
Containing more data than the entire Internet transmits in a second
- Get Intel's legendary Optane 905P 1.5TB SSD at a discount in the USwww.rockpapershotgun.com Get Intel's legendary Optane 905P 1.5TB SSD at a discount in the US
Intel's Optane drives are legendary in PC tech circles, offering random performance and low latency that remains unmatc…
- [ExplainingComputers] Explaining File Compression Formats
I imagine a lot of people who are into data hoarding already know a lot of this but I thought the video was pretty neat. It briefly talks about the history of different compression formats and provides a brief blurb about why you may want to use one or the other.
I'd recommend checking it out if you want 15 minutes of background noise.
---
For anyone new to data compression TechQuickie and CrashCourse have videos on it. If you really want to go down the rabbit hole you could check out media compression and see how things like JPEGs and PNGs work.
- 3d Printz's Case Turns a Raspberry Pi Into a Neat, 10TB-Capacity Network-Attached Storage Appliancewww.hackster.io 3d Printz's Case Turns a Raspberry Pi Into a Neat, 10TB-Capacity Network-Attached Storage Appliance
With friction-fit mounts for USB hard drives, a cooling fan, and a dedicated stats display, this compact NAS packs in the features.
- Get URLs for all tweets within date range?
cross-posted from: https://sh.itjust.works/post/14280067
> What is the best tool to get URLs for all tweets within a given date range? > > The ideal behaviour I'm looking for would be something like this: > > Input:
https://twitter.com/SpaceX 2023-09-01 2024-02-08
> > Output: > > -https://twitter.com/SpaceX/status/1755763378449183003#m
> -https://twitter.com/SpaceX/status/1755759459765567825#m
> -https://twitter.com/SpaceX/status/1755752291578302545#m
> -...
> > What would be the best tool to achieve this? Thanks in advance! - Best solution for a distributed filesystem?
Not sure if this is better fit for datahoarder or some selfhost community, but putting my money on this one.
The problem
I currently have a cute little server with two drives connected to it running a few different services (mostly media serving and torrents). The key facts here is that 1) it's cute and little, 2) it's handling pretty bulky data. Cute and little doesn't go very well with big raid setups and such, and apart from upgrading one of the drives I'm probably at my limit in terms of how much storage I can physically fit in the machine. Also if I want to reinstall it or something that's very difficult to do without downtime since I'd have to move the drive and services of to a different machine (not a huge problem since I'm the only one using it, but I don't like it).
Solution
A distributed FS would definitely solve the issue of physically fitting more drives into the chassi, since I could basically just connect drives to a raspberry pi and have this raspi join the distributed fs. Great.
I think it could also solve the issue of potential downtime if I reinstall or do maintenance, since I can have multiple services read of the same distributed FS and reroute my reverse proxy to use the new services while the old ones are taken offline. There will potentially be a disruption, but no downtime.
Candidates
I know there are many different solutions for distributed filesystems, such as ceph, moosefs, glusterfs and miniio. I'm kinda leaning towards ceph because of it's integration in proxmox, but it also seems like the most complicated solution in the bunch. Is it worth it? What are your experiences with these, and given the above description of my use-case which do you think would be the best fit?
Since I already have a lot of data it's a bonus if it's easy to migrate from my current filesystem somehow.
My current setup uses a lot of hard links as well, so it's a big bonus if the solution has something similar (i.e. some easy way of storing the same data in multiple places without duplicating it)
- [Help] [Seeding] DDoS secrets, responsible for hosting several leaks, will stop its activities.
cross-posted from: https://lemmy.dbzer0.com/post/13532369
DDoS secrets responsible for hosting leaks such as EpikFail and BlueLeaks will stop its activities, I would like help from anyone who has space left so we can download everything and keep seeding.
Torrent download links: https://data.ddosecrets.com/
- Changing Video Encoding Properly
I'm going to archive some youtube videos, what's the proper way to change it from mp4 to webm to etc etc or vice versa?
In the past when I couldn't run a video file for whatever reason I would just rename the file, but I'm assuming there's better ways to do it. And is there a specific order I have to go in? (e.g. with audio going from .mp3 to .flac doesn't make sense.)
Thanks in advance.
- The 3-2-1 Backup Strategy
The 3-2-1 Backup Strategy https://www.backblaze.com/blog/the-3-2-1-backup-strategy/ @datahoarder
- scraped media links from instagram and threads
I have scraped a lot of links from instagram and threads using selenium python. It was a good learning experience. I will be running that script for few days more and will see how many more media links I can scrape from instagram and threads.
However, the problem is that the media isn't tagged so we don't know what type of media it is. I wonder if there is an AI or something that can categorize this random media links to an organized list.
if you want to download all the media from the links you can run the following command: ```bash
This command will download file with all the links
wget -O links.txt https://gist.githubusercontent.com/Ghodawalaaman/f331d95550f64afac67a6b2a68903bf7/raw/7cc4cc57cdf5ab8aef6471c9407585315ca9d628/gistfile1.txt
This command will actually download the media from the links file we got from the above command
wget -i links1.txt ``` I was thinking about storing all of these. there is two ways of storing these. the first one is to just store the links.txt file and download the content when needed or we can download the content from the links save it to a hard drive. the second method will consume more space, so the first method is good imo.
I hope it was something you like :)
- Hobbes OS/2 Archive to shut down in three months – OSnews - Lemmy.Worldlemmy.world Hobbes OS/2 Archive to shut down in three months – OSnews - Lemmy.World
The announcement on https://hobbes.nmsu.edu/ [https://hobbes.nmsu.edu/] ATTENTION After many years of service, hobbes.nmsu.edu [http://hobbes.nmsu.edu] will be decommissioned and will no longer be available. You the user are responsible for downloading any of the files found in this archive if you w...
- Seagate Expansion external HDD power supply making a very faint alarm sound
I've got a fairly new 14tb Seagate Expansion. It works fine, and I've been using it for a month and a bit.
I don't know how long it's been doing this, but the power supply is making a very faint alarm sound. The power supply is plugged into a Belkin surge protector powered on and with the "protected" status light lit, and it is plugged into an outlet. The HDD is currently not plugged in to a computer.
It's not a beep or electricity. It's a distinct weewooweewoo. I couldn't even determine the source until I pressed my ear against it.
Googling just points me towards typical "my HDD is making a sound, how long do I have until it dies", but nothing pointed me to the alarm sound from the power supply.
I'll check again if it makes the alarm in other conditions, but in the meanwhile, I was hoping someone here might know something.
Thanks in advance!
EDIT: The sound only happens when...
- Power adapter is plugged into the HDD, AND the outlet
- HDD is NOT plugged into the computer.
Plugging it into the computer stops the noise from the power adapter.
- Whatever happened to DNA-based storage research?
It seems like 6 or 7 years ago there was research into new forms of storage, using crystals or DNA that promised ultra high density storage. I know the read/write speed was not very fast, but I thought by now there would be more progress in the area. Apparently in 2021 there was a team that got a 16GB file stored in DNA. In the last month there's some company (Biomemory) that lets you store 1KB of data into DNA for $1,000, but if you want to read it, you have to send it to them. I don't understand why you would use that today.
I wonder if it will ever be viable for us to have DNA readers/writers... but I also wonder if there are other new types of data storage coming up that might be just as good.
If you know anything about the DNA research or other new storage forms, what do you think is the most promising one?
- How to datahoard Lemmy
Sorry for not doing much research beforehand and asking a newbee question. I am looking for some entrypoint info to the question:
How would one go about datahoarding lemmy?
It seems to be a grade above what I've been doing so far (downloading video/audio from streaming platforms and backing up web articles and blogposts as pdfs) due to the distributed nature and the activitypub protocol. ___
Relevant stuff that I've found so far but havent studied extensively:
- This does not seem to store most of the data https://github.com/tgxn/lemmy-explorer
- What's an elegant way of automatically backing up the contents of a large drive to multiple smaller drives that add up to the capacity of the large drive? (on Linux)
So I have a nearly full 4 TB hard drive in my server that I want to make an offline backup of. However, the only spare hard drives I have are a few 500 GB and 1 TB ones, so the entire contents will not fit all at once, but I do have enough total space for it. I also only have one USB hard drive dock so I can only plug in one hard drive at a time, and in any case I don't want to do any sort of RAID 0 or striping because the hard drives are old and I don't want a single one of them failing to make the entire backup unrecoverable.
I could just play digital Tetris and just manually copy over individual directories to each smaller drive until they fill up while mentally keeping track of which directories still need to be copied when I change drives, but I'm hoping for a more automatic and less error prone way. Ideally, I'd want something that can automatically begin copying the entire contents of a given drive or directory to a drive that isn't big enough to fit everything, automatically round down to the last file that will fit in its entirety (I don't want to split files between drives), and then wait for me to unplug the first drive and plug in another drive and specify a new mount point before continuing to copy the remaining files, using as many drives as necessary to copy everything.
Does anyone know of something that can accomplish all of this on a Linux system?
- How do I archive a podcast I've already downloaded in Playapod (iOS) that has since gone offline?
Hello c/datahoarder! I need your help. Not sure whether this has been asked before—I've tried searching the web, but the only advice I can find is how to download episodes for podcasts whose feeds are still active.
The problem I'm trying to solve is that one of my favorite podcasts, Endless Boundaries Jam Radio, went offline during the pandemic. All the usual feed aggregators still show up in internet searches, but as they are not file hosts, just feed aggregators, all the episodes are now dead links (e.g. Podbay, Tunein, etc).
Thing is, I had already downloaded several episodes using the Playapod app on my iPhone. It's usable for now, but I'm very concerned about when I need to upgrade to a new phone.
Is there a trick for access the individual files on my iPhone that were downloaded through a third party app such as Playapod? TIA
EDIT: I figured out how to do what I wanted. Once I had installed ifuse and related dependencies (e.g. libimobiledevice) on my Linux PC, I could connect my iPhone to my PC via USB and browse the files on my iPhone in my distro's default file browser. Many folders are named as GUIDs, making it harder to tell what's what by just looking at their names, but I narrowed down the right folder by opening up the Disk Usage Analyzer app in Linux. In my case, the Playapod app is one of very few apps with more than a gigabyte of data. I still have to go through and figure out which episode each mp3 file is, but that's still better than having nothing at all.
Thanks to everyone who responded. I hope this info helps anyone else in a similar predicament!
- What to do with extra HDDs
Hey guys, I'm setting up my NAS (openmediavault) and very much enjoying it! It now runs my Nextcloud and a couple of services. I got a mirror ZFS setup of two 8TB drives.
I got another two 8TB drives and am doubting whether I should add them as an extra mirror vdev, or create a new pool for extra backup. I'm not sure if that extra backup is necessary though, since I got a cloud backup already every day. My drives are only used 14% so I'm not even sure if I should already put them in the pool. What do you guys think?
- Super micro sas826a backplane
Just wondering if anyone knows which SAS connectors on the SAS826A backplane control which ports?
On my current setup only ports 8-11 are working so got some troubleshooting ahead of me.
The online manuals show the connectors but unhelpfullyndont indicate which ports are being used for each.
Also, anyone know what the ribbon cable beside the SAS wires is used for on supermicro cables? I don’t recall seeing it on other SAS cables.
- What guides, wikis, or megathreads are available for those new to archiving and storing data?
I've recently aquired the hardware to build a home server/NAS. I'd love to know some community-guided advice on tools I should consider, and what best practices are?
For instance, how does redundancy work? Whay about automated backups? What OS should be running on a NAS? What utilities can I use to monitor the safety of my data? Perhaps even a guide about how to safely share that data outside my home network for personal use, or even open for the internet, without compromising my network?
Thanks for the discussion
- A dozen or two TB of storage for media on the cheap?
Well I'm just about fed up with streaming bullshit. I currently have a home server that's just a raspberry pi4 with a bunch of docker containers and it served my light usage well.
But with transcoding on Jellyfin I'll be needing some more power. And a bunch of storage. So wanting to perhaps build a new little server.
CPU requirements aren't high at all. Need to transcode maybe 2 concurrent 4K streams, A cheap discrete GPU or a CPU with a decent enough iGPU could handle this. Other applications are basically negligible, like Vaultwarden and PiHole, torrent, using as a general file storage server.
I also recently acquired a mini PC which is plenty powerful, but doesn't have any way of adding a bunch of drives. So another option is setting up a pure NAS and just using the mini PC as the server. It's got an i7 10700T and iris 630 iGPU.
I've been using Linux and self hosting basic things for years, but I'm pretty new to this level of hardware and little experience with RAID.
Budget: ~$500ish - storage goal: 12+ TB
- Mechanical or SSD for offsite storage that’s updated at least yearly?
I have about 100gb and growing that is critical for my business. File size growth is slow, so it will be years and years before it even gets to 200gb.
I have multiple local copies and a copy in google drive, but I want to leave a hard drive at my mother-in-law’s house.
I only want 2.5 form factor or smaller as my mother-in-law will be carrying it here when she comes to visit us on the city.
I’m not sure what the recommendation is. I’m not a millionaire, I’m just freelance. So, I’d like to minimize cost.