Mage load testing

[noxx]

Hi,

This message is about load tests I've finished recently that finally reproduced all the issues we have with public Mage server.
Not sure it is interesting for anyone but I'd like the team to be aware of these tests: load test plan, the test implementation and the results.

Let's start step by step.

The reason for implementation:
Public server is unstable and has to be restarted very often.
Description:
While Mage works fine for one-two games, it crashes whenever it is used as public server and players start to use it. I would say that it was more or less ok when we had very powerful server from Hans. So it required reboot but only every 2-3 days eating 2-3 Gb of memory. And we could live with that.
Nowadays we use very weak server starting Mage with only 512 Mb of memory. As a result I restarted the server about 30 times during last week. Thanks to russian community that informs me whenever Mage server is not responsible.

From the logs it could be found that there are several issues: socket can't be created, OutOfMemoryException and "WARNING! GameController.sendMessage - couldn't find session for action execution."
The last one is very strange, but now seems I know where it comes from, as well as the issue with sockets - from memory leak we have in server.

Goals:
* Server should be able to run for 48 hours without reboot on 512 Mb configuration when one game is played by two players over and over
* Server should be able to handle 16 players playing two-player games simultaneously on 512 Mb configuration

Load test plan:
The idea of load testing was creating automated tests in java that would emulate players and test server with games played.
As a result it would be great if they could reproduce the issue described in the description.

There were several requirements I wanted to be taken into account while implementing them:
* The tests should be integration tests.
That means that mage server should be started as dedicated server (in compare with card module tests we have where we start server within the test). The main reason is being more realistic as well as the possibility to profile it as starting it separately means it still runs when test finished and can be analyzed, profiled, etc.
* There should be several load patterns
Tests should be written from simple till more complicated and support different user actions emulation.
The patterns are described below.
* Tests should be created as a part of Mage.Tests module
* Tests should have @Ignore annotation to be executed on demand only and not be included into main build process

Load patterns
Patterns for one game played at the same time:
1) Emulate two players connecting to the server and leaving without disconnect call.
So server is unaware of players leaving the game and server.
2) Emulate simple game for two players
Two players join, start the game and just skip phases doing nothing but discarding cards at the end of turn.
3) Emulate basic game for two players
Two players join, start the game, add lands, creatures, enchantments at game startup, then play a land per turn.
One of the players plays "deal 1 damage to target player" spell every turn targeting opponent so the game finishes in 40 turns.

Concurrent games:
4) Emulate 16 basic two-players games running on the server at the same time.
Runs the 3) patterns in 20 separate threads simultaneously.

Implementation notes
The main idea for the implementation is simple. To emulate the actions in Mage.Client but without UI.
That's possible if we create our own version for callback that wouldn't display any information in user interface waiting for player response but would response immediately based on predefined rules instead.
So a basic test creates two players with separate sessions and callbacks and starts the game handling various game situations in the callback function.

What was found during the implementation that almost every game "freezes" because of message sent from client side and no reaction from server side. And what's important such case happens in 9 of 10 games. The reason is unknown at the moment but I could manage it by adding control thread that checks the test being idle for 5 seconds or more and repeatd last client action once again. It may relate to the fact that test runs "too fast" spamming server, don't know. Anyway seems not critical issue as in real life user would just press button once again.

Results
At the moment only 1) and 2) patterns were implemented but seems that it is enough for now as both of them fortunately reproduced what they intended to reproduce.

The first one results in session not being closed for ages on server side. The most reason is session state that was implemented. The main idea was that it gets DISCONNECTED state on exception happening in error handling callback based on JBoss Remoting API. And it works locally, but adding additional logs on server side have revealed that it doesn't happen always. So I had to implement it other way by adding ping() method and removing state using for expired sessions. Seems now that part is ok.

The second one reproduced the memory leak. As an example, Mage server starts with about ~40 Mb of heap memory usage.
5 simple games played in a row (second load pattern) resulted in 514 Mb of head memory used on server. And calling Perform GC reduces it to 495 Mb.
Thus all those games were locked on server side.
As server was started with 512Mb memory configuration, java had no enough memory anymore. And this resulted in the message that I'm very glad for being reproduced:
WARNING! GameController.sendMessage - couldn't find session for action execution.

That's it.

Since now I would start investigating this issue and this is the main priority issue we have at the moment.
Possibly we should postpone the release till the fix.

Best regards,
Noxx