Flirting with parallel programming

[myk]

That's going to serve as an introduction to concurrent programming for me :)

Concurrent processing can certainly speed up some tasks, especially I/O. Just remember that anything coded with threads must be tested ten times as many times as single-threaded code to check for race conditions, deadlock, livelock, and starvation. Keep all concurrency within a single class or file, if you can. Use asserts liberally to check your assumptions. Most importantly, this kind of development belongs in a branch until it is well tested across many different types of systems. Trunk is not the place for experiments. I say this not because I am afraid of threads. I say this because concurrent programming is my professional expertise, and I have already made the mistakes of using them where they should not be used.

That said, it does sound like doing threaded loads is probably a good idea, at least for SSD hard drives. I'm not sure how spindles would react. For the people that are seeing slowdowns, what kind of hard drives are you using?

[friarsol]

That said, it does sound like doing threaded loads is probably a good idea, at least for SSD hard drives. I'm not sure how spindles would react. For the people that are seeing slowdowns, what kind of hard drives are you using?

It's just a simple laptop so I'm fairly positive its a 5400 rpm HDD.

[Max mtg]

Sol, I have made a code path for single-core systems, try the loading time now.

[Max mtg]

Myk, thanks for warnings.
I'll move to a branch with Input synchronization task.

[Max mtg]

myk, I understand now why the loading time increases in some cases. Due to sequental read being faster than random access one. And it looks like there is no way to detect if an SSD is used or cardfolder is already cached by OS.

So, what do you think, if cardfolder is processed in a different way, will it be faster?
Like read all files' contents in a single thread and then distubute List<List<String>> (that will hold read lines) between threads?

[myk]

With performance optimization-related questions, the only real answer is to try and test, and then get some volunteers with different types of systems, and have them test it too. Splitting the I/O from the processing might speed things up, but the processing we do is really minimal -- just parsing -- and the gains from separating I/O from processing might be erased by the overhead of just having the thread at all. It's impossible to know before you try and measure, though.

For what it's worth, the sharded loading algorithm in place right now makes forge load on my systems (Macbook Pro, with flash-based HDD and a Linux workstation with an SSD cache drive) in the blink of an eye, and I'd miss it if it were changed back : ) Perhaps a user-selectable preference for "parallel loading" with explanatory text in the tooltip, and let the user decide which algorithm to use?

[Max mtg]

decks loader needs a similiar optimization, single threaded loader is not fast enough:
Read decks (1517 ms): 7658 constructed, 0 sealed, 0 draft, 23 cubes, 0 scheme, 0 planar.