Ideas for a federated anime tracker
Ideas for a federated anime tracker
Hello Lemmings.
I will be attempting to make a federated anime tracker this summer, but I am not quite sure what features people would want and how I would get the details for animes, mangas, etc.
For the latter: What I thought was to either scrape other anime websites continuosly in the background, but this most likely is against the ToS of every anime tracking website, such as AniList or MAL. (I actually asked anidb.net for special access to their DB because apparently you can request access to it, but I've been left on read by the two staff members) My second idea was to make it an anime tracker website where animes are only user-submitted. And the user submissions would be approved by assigned moderators. However, I think this would be quite inconvenient. I'd like to get your opinions and/or ideas for this.
For the former: So if you have any requests or suggestions, please drop it down in the comments section.
Thanks in advance.
Never go for user-submitted / approved by moderator system. I've been in one project like that years ago and it's not gonna work.
I'll make another comment once I get up.
Is there a good reason why? Even MAL takes user submissions.
It's a slow system. MAL takes user submissions because they already have a big database and those submissions help filling the cracks. If you don't have a database to begin with it slows things down, especially if these submissions will go by trust-based. Doing this by randoms is also very risky.
However, I'm not completely oppose to that idea because it can be helpful for some areas. My suggestion is start with a basic dog tag system, where the anime name, alternative names, status (airing, completed), season and start date, studios (also licensors and producers), age rating etc. These information needs to be scraped for the fastest way to form a quick database, they are publicly available (even on Wikipedia) so it should be fine to scrape. You can even go full Wikipedia after got only the names. User submissions could be useful for the introduction / summary parts of the titles at this stage. For only names (and basic tags), you can scrape AniDB from this list. It's just a search query so shouldn't be against their ToS.
You can also check Kitsu for ideas, I like their DB request system. Pretty basic but can be done differently with the power of ActivityPub.
You might want to discuss this in one of the communities at https://ani.social.
I'll crosspost there later.
but this most likely is against the ToS of every anime tracking website
AFAIK scraping publicly accessible websites is fine in most countries (IANAL, look into it)
If you did this for all audiovisual content and not just anime I'd be all in.
I'm pretty sure that I/someone could just fork this project once it's done and make a few changes to it to make it a audiovisual content tracker.
I think the biggest challenge will be to standardize the scraping process since databases for audiovisual content are very differently structured. Will probably require database-by-database API access/negotiations so that you know how their databases are structured - completely FOSS audiovisual content databases are rare.
I had a try ad Bookwyrm, which seems similar to what you intend to build. I was disappointed because the database itself is federated, meaning it's full of duplicated entries.
Correct. I thought there was some way to prevent those duplicates because of the federation, but apparently there's no such check being done so it's all duplicates. For that reason I won't put any effort into making the database federated. Only threads, reviews and comments will be federated. This might change as I make it though. There are also clubs in MAL which I might copy and implement in this project and they could be federated as well.
Decentralizing the database in a federated structure for anime tracking is a very good idea. Right now I'm using Anitrend that is open source but is only a interface to Anilist.
Like you pointed I think the major challenge will be to establish a solid new shows database entries police(s). Not sure how we could manage that effectively. Governance will be key but you also don't want to be a hostage of this project for your life.
One aspect to be taken in consideration is privacy. I think a lot of people would appreciate to have access to the new tracker without having to share anything about them.
Oh well, no. The database won't be decentralized. That just invites chaos. Bookwyrm did that and now there are lots of duplicates in their library, which I definitely want to avoid. The things that will be federated are animes' forums (as communities in Lemmy for example), and those communities will have threads that will be either a forum thread or a review which others can comment on from any federated platform. Some other things might get federated in the future.
I decided to just scrape either MAL alone or multiple sources.
If you have any ideas on how the database can be decentralized while efficiently avoiding duplicates and spam, please do say it.
https://www.kenmei.co/ may be close to what you want for manga.