It is currently 28 Jul 2017, 06:56
   
Text Size

IPFS for picture storage?

Discuss Card Scans and Other Artwork Here

Moderator: CCGHQ Admins

IPFS for picture storage?

Postby MidnightLightning » 07 Nov 2016, 18:24

Hey all! For distributing/managing/maintaining the growing collection of card scans, has IPFS (InterPlanetary File System) been looked into as a solution? I know the main FAQ thread talks about the modern sets have been moved to Mega as a host since it allows easier browsing and download of individual files.

I think IPFS might be a good option for this community as it offers the same sort of "browse for individual files" flexibility, but doesn't have "single point of failure" problem (if Mega were ever forced to shut down, those files are gone). IPFS is more like a torrenting setup, where community members can locally seed the files, but allows for picking and choosing individual files from the collection.

For example, I seeded the 4th Edition XLHQ folder, and it now can be accessed from:

https://gateway.ipfs.io/ipfs/QmRPFDMW86 ... q6WcCVFZTp

Anyone who wants to run a local IPFS node can issue the command:

Code: Select all
ipfs pin QmRPFDMW86yhqwVK7DUPYaa43eN59XrraCXdq6WcCVFZTp
And their node will cache the whole folder locally, and start serving as another peer.

The downside is adding files to a folder changes the hash/identifier of the folder. So, while right now the Mega link of https://mega.nz/#F!p8RBBT6Y!ksgSGJbMsKU0HX_ho-QS5g will always go to the most up-to-date folder with all the newest sub-folders added. Directing users to an IPFS folder would mean updating the link's hash when new folders were added to the root level. Or, IPFS has a naming system (which seems to not quite be finalized yet), where CCGHQ could keep a node up-to-date with the most recent hash's name, though that part of the IPFS protocol seems to not be finalized yet.

Anyone have thoughts on this or tried working with this in the past?
MidnightLightning
 
Posts: 2
Joined: 07 Nov 2016, 18:09
Has thanked: 1 time
Been thanked: 0 time

Re: IPFS for picture storage?

Postby JohnnyTH » 10 Nov 2016, 17:17

I would never consider a alpha software non-widely-used for production purposes.

Plus for this kind of repository we need something more similar to Git, because people tend to sync to this archive so if random files are patched or updated we also need a mean to view the changes history and fetch only the updates from a certain point onward.
JohnnyTH
 
Posts: 33
Joined: 23 Mar 2015, 13:03
Has thanked: 11 times
Been thanked: 13 times

Re: IPFS for picture storage?

Postby MidnightLightning » 16 Nov 2016, 06:51

Thanks for your thoughts! I agree on not cutting over to alpha software directly; abandoning the existing setup, but I'll try implementing it for a smaller project of mine and see how it goes.
MidnightLightning
 
Posts: 2
Joined: 07 Nov 2016, 18:09
Has thanked: 1 time
Been thanked: 0 time

Re: IPFS for picture storage?

Postby JohnnyTH » 19 Dec 2016, 12:44

I still think it would be great to have a distributed repository for all the material produced in this project.

Do you have any other solution in mind? I started a git repository locally checking in all the sets that i'm interested about, but rebuilding the whole tree in git could take a toll, I'm not sure how git behaves with so much data.

Git also has the advantage of being a distributed versioning system, so each one of us could keep his own repo and we just need to be able to sync regularly somehow
JohnnyTH
 
Posts: 33
Joined: 23 Mar 2015, 13:03
Has thanked: 11 times
Been thanked: 13 times

Re: IPFS for picture storage?

Postby skibulk » 19 Dec 2016, 14:09

@JohnnyTH I think it would be cool to host a card database via git, including printed text and stats. There are a few scrapers for Gatherer, but there are so many errors and left out cards. Community management would be the only way to cover all the languages and stay on top of bug fixes. I have two problems with it though:

1) It would take a ton of work to setup and I just don't have the time to do it. It might be manageable with a tried and true database like Magic Album's, but GH has requested not to use his data.
2) I fear that somebody would complain about copyright infringement and our git would just be taken down. I posted a question to Stack Exchange Law, without any input on this topic.
User avatar
skibulk
HQ Team Member
 
Posts: 985
Joined: 19 Jul 2010, 20:34
Location: Northeast USA
Has thanked: 375 times
Been thanked: 301 times

Re: IPFS for picture storage?

Postby charlequin » 19 Dec 2016, 16:52

mtgjson is pretty close to what you're describing: https://mtgjson.com The version "with extras" includes original printed text along with a bunch of other useful metadata. (They actually namecheck us on the front page as a source of truth for card texts, heh.) At one point in the past, the creator of mtgjson also maintained a second site called mtgimage which basically provided an API to get card pictures in different sizes; that site got a C&D while mtgjson did not, so I think they're probably in the clear of WotC legal.

I haven't compared closely but at least on my spot checks in the past their data has been pretty spot on.
charlequin
HQ Team Member
 
Posts: 429
Joined: 15 Oct 2008, 18:34
Has thanked: 56 times
Been thanked: 654 times


Return to Pictures

Who is online

Users browsing this forum: No registered users and 4 guests


Who is online

In total there are 4 users online :: 0 registered, 0 hidden and 4 guests (based on users active over the past 10 minutes)
Most users ever online was 279 on 11 Jul 2013, 22:03

Users browsing this forum: No registered users and 4 guests

Login Form