Page 1 of 1

IPFS for picture storage?

PostPosted: 07 Nov 2016, 18:24
by MidnightLightning
Hey all! For distributing/managing/maintaining the growing collection of card scans, has IPFS (InterPlanetary File System) been looked into as a solution? I know the main FAQ thread talks about the modern sets have been moved to Mega as a host since it allows easier browsing and download of individual files.

I think IPFS might be a good option for this community as it offers the same sort of "browse for individual files" flexibility, but doesn't have "single point of failure" problem (if Mega were ever forced to shut down, those files are gone). IPFS is more like a torrenting setup, where community members can locally seed the files, but allows for picking and choosing individual files from the collection.

For example, I seeded the 4th Edition XLHQ folder, and it now can be accessed from: ... q6WcCVFZTp

Anyone who wants to run a local IPFS node can issue the command:

Code: Select all
ipfs pin QmRPFDMW86yhqwVK7DUPYaa43eN59XrraCXdq6WcCVFZTp
And their node will cache the whole folder locally, and start serving as another peer.

The downside is adding files to a folder changes the hash/identifier of the folder. So, while right now the Mega link of!p8RBBT6Y!ksgSGJbMsKU0HX_ho-QS5g will always go to the most up-to-date folder with all the newest sub-folders added. Directing users to an IPFS folder would mean updating the link's hash when new folders were added to the root level. Or, IPFS has a naming system (which seems to not quite be finalized yet), where CCGHQ could keep a node up-to-date with the most recent hash's name, though that part of the IPFS protocol seems to not be finalized yet.

Anyone have thoughts on this or tried working with this in the past?

Re: IPFS for picture storage?

PostPosted: 10 Nov 2016, 17:17
by JohnnyTH
I would never consider a alpha software non-widely-used for production purposes.

Plus for this kind of repository we need something more similar to Git, because people tend to sync to this archive so if random files are patched or updated we also need a mean to view the changes history and fetch only the updates from a certain point onward.

Re: IPFS for picture storage?

PostPosted: 16 Nov 2016, 06:51
by MidnightLightning
Thanks for your thoughts! I agree on not cutting over to alpha software directly; abandoning the existing setup, but I'll try implementing it for a smaller project of mine and see how it goes.

Re: IPFS for picture storage?

PostPosted: 19 Dec 2016, 12:44
by JohnnyTH
I still think it would be great to have a distributed repository for all the material produced in this project.

Do you have any other solution in mind? I started a git repository locally checking in all the sets that i'm interested about, but rebuilding the whole tree in git could take a toll, I'm not sure how git behaves with so much data.

Git also has the advantage of being a distributed versioning system, so each one of us could keep his own repo and we just need to be able to sync regularly somehow

Re: IPFS for picture storage?

PostPosted: 19 Dec 2016, 14:09
by skibulk
@JohnnyTH I think it would be cool to host a card database via git, including printed text and stats. There are a few scrapers for Gatherer, but there are so many errors and left out cards. Community management would be the only way to cover all the languages and stay on top of bug fixes. I have two problems with it though:

1) It would take a ton of work to setup and I just don't have the time to do it. It might be manageable with a tried and true database like Magic Album's, but GH has requested not to use his data.
2) I fear that somebody would complain about copyright infringement and our git would just be taken down. I posted a question to Stack Exchange Law, without any input on this topic.

Re: IPFS for picture storage?

PostPosted: 19 Dec 2016, 16:52
by charlequin
mtgjson is pretty close to what you're describing: The version "with extras" includes original printed text along with a bunch of other useful metadata. (They actually namecheck us on the front page as a source of truth for card texts, heh.) At one point in the past, the creator of mtgjson also maintained a second site called mtgimage which basically provided an API to get card pictures in different sizes; that site got a C&D while mtgjson did not, so I think they're probably in the clear of WotC legal.

I haven't compared closely but at least on my spot checks in the past their data has been pretty spot on.