›

Programming issue

TomtomClancyTomtomClancy Member
in Questions & Troubleshooting
Hi everybody,

I'm a programmer myseft and work on AAA games.

I follow the game since the very beggining, and I have notice that your AI loop are on the main thread, that cause some lag very often.

My question is simple : Why !? I know that Unity API is not thread safe that's why I am pretty sure you use coroutines. But coroutine are all executed on the main Thread !

When I play My core 0 is in 100% loaded and my 7 others sleep, result is 16 % average load on 8 cores but the main is completly overloaded and the game is not playable / onjoyable....

Please Hire a Render guy, move you code in thread for path calculation and AI stuff.

I'm playing City Syline that run on Unity as well, and its fully playable and multithreaded.

I hope you are working on it.
100 ms per trame result on 10 FPS on single core, but 60 FPS if threaded on 8 cores (16ms per core for target 60 fps).

- A Friendly pro that care about the game.

Comments

  • faviannfaviann Administrator
    Sorry for the late response, I wanted to take the time to answer considering I've heard this question a lot and its always fun writing with a fellow developer.

    I'm one of the devs at Sauropod Studio.

    You are right by saying that the AI runs on the main thread. Actually, "everything" in Unity runs on the main thread. Though rendering is already threaded in Unity 5.4. (Still in beta, good news though)

    You might not know, but what makes up most of the lag spikes, by far, is garbage collection (which we're working hard to make it less of an issue: pooling, abuse of value types, Generics (to prevent boxing), etc). Heap thrashing can sometimes be caused by code which one wouldn't even suspect a little.

    To be honest threading is just part of the solution of performance. While some parts of the game would be less painful to thread, they also usually better benefit from reworking their data structures/algorithm for the same amount of work. I'm definitely not saying threading wouldn't help though.

    What I'm saying is more that if the various allocation/garbage issues are not fixed the threading point is moot since the GC could ​*still*​ stall the main thread for hundreds of milliseconds. It might even worsen the lag spike issue because more allocations could be done since the execution would no longer be not time bound (excluding context switching).

    Even with the allocation/garbage issues "fixed", multi-threading is unfortunately far from just being a simple consumer-producer issue (which is much more threading-friendly). One of the problems with Unity, which you might not know, is that APIs are NOT usable outside of the main thread. And by not usable I mean, Unity throws an exception if you use its APIs outside of the main thread. So if you design/couple your game with the engine (using Monobehaviours and GameObjects) well you need major refactor everywhere for pretty much all the data that would normally be accessed via the Engine.

    For example, something as stupid/simple as getting the position of an object would throw an exception if you try getting it outside of the main thread. You're left NOT programming your game using MonoBehaviours (base class of your components) and instead use somekind of POCO (https://goo.gl/6DLbZt) to reimplement some needed features already usually offered by Unity. You're stuck using the monobehaviours as somekind of wrapper that use the POCO as delegates, which is definitely not the initial design behind the Unity Engine.

    Another example. Getting a component on a GameObject also requires an Unity API call (as in, not accessible outside the main thread). To fix this, your components need proper dependency injection for all you external dependencies (external to the component). This means "global" objects (like sound, voxel engine, etc) but also local ones (other components on the GameObject).

    If we wanted to then make the AI work properly in threads, you'd need to make sure the whole execution pipeline would not depend on any of the APIs of Unity anywhere. And its a lot of work considering the amount of object and data the AI "touches" to take decisions; which implies a form of risk unfortunately. It's a lot of fencing for writing and reading considering the amount of data analyzed (be it pathing, object querying, other character querying, object being destroyed mid analysis, etc).

    Now it's only a broad overview of the issues. I've skipped over a lot of them, destroyed object detection, thread safety for you components, rise in debugging costs (time and complexity), race conditions, and probably more I've forgotten.

    Now it seems like I painted a real dark picture of the whole thing but a lot of the required work has been underway. The pretty big one being memory handling. Modules have also been separated using the Signals and slots (https://goo.gl/QSEr8T) construct which decouples modules neatly while also allowing the future signal firing between threaded modules (mainly to join worker threads results to the main thread). But unfortunately there is still a lot of work to be done.

    Conclusion is, the Unity API being not at all accessible in other threads makes it a lot more work to implement thread safety if the game uses and depends on the engine's APIs.

    By the way, thanks for taking the time to post the suggestion. Hope it sheds some light on the multithreading aspect of Castle Story.

    PS: Fun fact, a lot of the changes to make the game work better with multithreading is similar to the work needed to make it better at Testability (as in Unit Testing). So it's definitely a good thing in the long run.
  • TomtomClancyTomtomClancy Member
    Hi Faviann,

    Thanks for the explanations, I know the requirements of multithreading, and I did know that Unity API is not thread safe, as I mentioned it in my first post.

    That's why since the very beggining of the project I though that is a Risky bet to use Unity for other thing than prototyping. I you did well.

    I know that you have to use home made C# structure and avoid any call to Unity API, to be able to use MultiThreading. And Its definitly not something trivial to setup. But not impossible (Sharing custom home made data ) to interface Unity API and multithread codes. Like you do on any rendering pipe ( Main thread fill buffers -> render pass read buffer ).

    There is no Engine that not require some modifications to fit any project even UnrealEngine or CryEngine need to recode and unplug some parts.

    I just hope that you will succeed to make the project runing smoothly, and to achieve it, running Castle Story on full available power.

    So good luck.
  • GlorvalGlorval Member
    I am not much of a programmer, but I can understand most of what you say, and for quite a few people out here our graphics cards and processors are nearly idle overall when running the game :| Kinda disappointing when my game starts becoming a lag fest and I check my performance.... and I am using like 10% of available processing power and my graphics card can handle a lot more. So while it seems there are loops to jump through, some harder than others, at some point having multi threading will be the best way to increase speed for the majority of players.
  • TomtomClancyTomtomClancy Member
    Hi all,

    Do the game still have some CPU pikes that cause lags ?
  • ShatojonShatojon Administrator Developer Backer Wiki Editor
    Hey Tomtom, there was an issue in 0.8 with garbage collection running more frequently now that we've optimized the game for more FPS, causing some lag spikes. The lag spikes are being reduced and performance improved for the next update, 0.9.

    Thanks.
  • Ennai27Ennai27 Member
    Hi, i'm on 1.1.0c and the lag is still there.
    When i start a new game it's mainly 23 fps that is just fine for this kind of game, but with time, the lag spikes increase and the fps decrease down to 7-4 fps. The one-thread issue is still there.
    I just can't continue like this, i hope you'll find a solution bec' the game is fantastic but unplayable.
    I have an octa-core, it's kinda a waste. ._.
  • ShatojonShatojon Administrator Developer Backer Wiki Editor
    Hey Ennai,

    Can you open a support ticket on http://support.castlestory.net - including your full PC specs and the save file that lags?
  • TomtomClancyTomtomClancy Member
    Hello, I have just runing the game since more than 6 month.
    I still experiement some spike (I guess GC collect) do Faviann still working on the project. to get some technical conversation ?

    #Tech
    Unity is now support C# 6, and I know since C# 4.5 you can perform a carbage collection on another thread to avoid blocking the main one and let player face to a huge spike of 1 to 2 seconds.

    I'm currently testing the new blueprint feature, but I experiment some very anoying path finding issue on vertical navigation. you can figure out by yourself by testing this blueprint.
    https://steamcommunity.com/sharedfiles/filedetails/?id=1517035217

    Have fun guys, seeking for your next update (I mean not a feature update, a core gameplay fix one).

    Thanks,

    Tom
Sign In or Register to comment.