Building an online MMO Part 2: Steps Towards a Functional Game, and a Forray Into Alternatives

Austin Hicks

2014-03-04 21:45

This is the second part of my series on my attempts at an MMO architecture. I have an incredibly basic working game at this point, though it does have problems which I believe to be solveable. The entire program, both client and server, is currently 897 lines. I am going to outline the basic components below. I will also talk about my adventures with Unity3D, Unreal Developer's Kit, and a few other products. As always, this is a description of the current state of things. I may come back in part 3 and say I had to throw it all out, but at least there is something to learn from knowing that it didn't work.

The Server

The server consists of a few mostly-separable pieces: the Server class, Place and subclasses, the MessageTransporter class, the ConnectedClientManager class, and various object types that implement gameplay.

At the lowest level is the network and objects that deal directly with it. Messages are currently JSON. The MessageTransporter knows how to encode and decode dicts and pass them across the wire. Both ends of the connection--the client and the server--use this class. In my last attempts, clients were one greenlet each, which managed state as well as writing messages. The current implementation gives each client two greenlets and makes use of Gevent queues. Both are actually simple for loops, making use of the fact that Gevent blocks greenlets when queues are empty. The actual class itself is extremely small. The interesting part is the class just above it, called the ConnectedClientManager. By overriding a function and replacing the callback to which a MessageTransporter sends messages when it receives them, this class implements filtering of messages. For simplicity, a client only sees one Place object at a time, so the ConnectedClientManager allows the client to refer to it with the special id "place". The same transformation is done for the client's body, referred to as "self". Unlike previous attempts, the ConnectedClientManager does not manipulate game state directly, instead opting to send the same messages the client can. On the creation of a new connections, it sends a new_object message to make a body and the first state_request message.

From here, the architecture can no longer be separated into levels. The next important piece, and the last part of network functionality, is the Server class. This class is completely reactive, doing nothing unless a client (dis)connects or some object sends messages. One interesting point is echo_back. If an object sends a message ith echo_back equal to True, then the message will be processed after the current one is finished.

The order in which echo_back messages are processed is the order in which they are posted, but the current message finishes processing first. This means that the queue of echo_back messages can become long. They are unlike function calls in one respect: they will only happen sometime after the current processing function returns. This defines an execution unit: when a message is processed, it is possible to know exactly what it caused to happen, thus recording reply_to information and cause-effect chains. Oddly, this is a recursive reelationship: it is possible to also know what echo_back messages cause, though such computation is decidedly nontrivial and not something I plan to implement. While I haven't done it yet, reply to information is useful for client-side interpolation.

I must confess that the following occured to me only now: This allows for infinite recursion with only a very small modification. Such recursion can never return to the body of the caller, but it can call the same function repeatedly with no limit.

It is worth mentioning at this point that no gameplay object ever calls another gameplay object directly to cause a state change, though they do sometimes touch each other to query state. All objects communicate through messages. This makes the potential need to move objects across the process boundary simple; each object knows how to make a new_object message describing its current state completely. If a router that knows how to send a message to a different process and a proxy that knows how to change the process with which the client communicates are implemented, the game can be split, possibly even across the computer boundary. Furthermore, even game ticks are messages. If an object wishes something to happen in the future, it is obliged to use the message system. As a result, The cause-effect chains described above can then theoretically be carried far into the future, well beyond the point of usefulness.

The common pattern with gameplay objects is as follows: Receive a message requesting an action, specifying state as relative values. The move message is an example, containing x and y deltas. Perform said action. Post past-tense messages, for example, moved_to with absolute x and y position information. Post further messages as needed using the echo_back system, for example to implement deletion of an object that has been killed or destroyed. This system is simple but powerful. The effect is that the server does not know about game state or gameplay. To reimplement or change something, two things and only two things must happen: the client's presenter and the server's object must be updated to understand any new or changed messages. It is my plan to make use of this facct in the near future to allow for higher-level commands and responses to the client. The current system does not distinguish between server-only messages and client-only messages. Recording such information is a good opportunity for a future optimization which I will no doubt need to implement for bandwidth reasons.

Python magic is used for message handling. All objects inherit from a magical class called MessageProcessor. This adds process_message to all subclasses, and allows for a very convenient trick. To handle a message, one need define handle_messagename. the dict representing the message will be unpacked into the function's parameters. For example, consider the following definition:

def handle_move(self, x, y, **message):
    ...some code...

Any extra information passed with the message will be disregarded and the values of interest will be unpacked into the named parameters.

The top-level object, containing all others, is a subclass of Place. This is the only object that currently knows how to handle the new_object message. In future, I may allow for any container to handle it. The server creates the first place at startup. All objects must be inside a place, directly or indirectly.

The Client

The client is currently much simpler and less interesting. Interpolation is not implemented at this point. It is also increasingly looking like I need to make use of a pubsub library such as Blinker (used in my other project, I3d, with good effect). Since it is currently so boring, I will simply provide a brief overview turned bullet points. The client is the least clean part of the code and is currently being refactored to be comprehensible.

The Client class connects to the server, using the MessageTransporter described above. The ClientConnectionManager is handled by the server, causing the client to always have two magical ids that always work: "place" and "self".
There are corrisponding presenters that know how to respond to the past-tense state update messages from the client, created when the client receives new_object using Python's importlib and getattr.
Some client-only objects exist, including the radar and the target indicator. These objects can register keystrokes and are ticked; they can be thought of as audio-only overlays.

And that's it. The client is currently decidedly non-magical. It is, of course, written with my as-of-yet unreleased game_engine package, which will probably be up on this site in the near future. Even more obviously, it uses Camlorn_audio.

Future Plans

Most immediately, the game needs multiple places and inventories. These are actually the same problem. I have opted to keep the object graph a tree and to store it separately from objects; this allows for very easy serialization, and Python provides properties to solve this problem. By serializing it from the deepest level up, reaching the node where serialization starts last, I theorize that it will be very simple to save and send subsets of the game to other computers. Objects already know how to describe themselves in a format that can be appended to messages.

The second thing that must happen is improvement of movement. The client must begin interpolation and stop spamming the server 60 times a second. In addition, the server needs to limit what it sends to the client in terms of movement, providing more sporatic updates. I theorize that the player will be unable to notice corrections in position of up to 1/100th of a unit, and probably only the most discerning players will notice anything below 1/30th of a unit. Notably, the closer the player is to a sound source, the more outstanding such corrections become. I think that by storing in reply to information, it will be possible to interpolate the position of the player's body by at least one move/moved_to pair. It is also posible to aggregate information and send it only 5 or 10 times a second, rather than 60. I do not intend this to be a first person shooter; it is my belief that only first person shooters need anything above 20 or 30 updates a second, and that this game only needs 10 at most. I will probably replace all floating point unit measurements with milliunits as integers for increased determinism.

On that note, I must define what a unit is: a meter? A foot? An astronomical unit? Something that depends on the map? I favour the last option.

Finally, it is my desire to move messages into their own module, and stop using dicts directly. The current approach is not flexible in terms of future changes. It would be useful to collect medadata: whether the message is server-only, whether it is frequent, which fields are object ids, whether it is important or can be sent over UDP, etc. I plan to investigate Enet or a custom approach using a UDP and TCP socket. I do not need such optimizations at the moment, but believe I will in future. Having messages in their own module also allows me to change how they are encoded, leaving behind the quite large approach of JSON if necessary.

Adventures with Other Options

I have spent some time over the last few days investigating various game engines that offer networking support, and ended up opting to use none of them. The core problems seem to come back to two things: either the networking is unuseable or as low-level as sockets, or they are inaccessible.

C4. Proprietary, costing $750 a user, less at the cost of lifetime updates. The problem here is accessibility: the entire game is created within an editor that appears to draw its interface using DirectX, and it is expected that you will use level design tools in conjunction with programming inside this environment. Networking appears to be low-levle, but this may not be the case. I could find little public documentation and no information on using it without the editor it comes with.
Panda3D. This one is completely programmatic. It claims to have high-level object sharing and persistence. The first warning sign of the problem with it is that the documentation on the feature is incredibly lacking. After a bit more digging, I found out that we only have client code for its advanced networking. The author, Disney, never released the server code to the public. No alternative implementation currently exists in a useable state. All that remains is a lower level message-based system; Gevent with my MessageTransporter class is much cleaner.
Unity3D. The entire game is created inside an inaccessible editor, not programmatically. The one article I found on how to go about programmatic access involved losing almost all of the benefit of Unity3D, and was incomplete. It appears to have no HRTF support; its audio is Fmod.
Ogre3d. Refers the user to external networking libraries at the level of sockets, and only handles 3d object rendering.
Unreal Developer Kit. This was the most promising. All scripts are made in an editor of your choosing, and it supports OpenAL. Networking is basically automatic. I was all set to use it. It was going to solve all my problems. It could even do stuff that is rare: occlusion of sound sources and the attenuation of environmental sounds based on the properties of the map. but it was not to be. The first thing that is done is the loading of a map. This happens even before your game launches. If you do not specify a map, you get a default one. Maps include 3d models and all the objects for your game, a well as triggers and the like. They also apparently specify the aforementioned sound properties. Problem: after two hours of Googling, I could find no way to make these maps programmatically after the game loads. The only thing I found was one Stackoverflow answer that basically boiled down to "Why would you? You're being stupid." All maps are apparently made only through the UDK editor, basically a 3d modelling program on steroids. It may be possible to get programmatic access, but I cannot find the needed info. The only other problem with UDK is that it appears incredibly oriented towards multiplayer shooters, but I could live with or work around that. It raised my hopes for a quick solution very very high. Then it smashed them and threw the pieces into a fire. I went from skeptical as to whether i could make use of one of these engines to incredibly hopeful and ready to try UDK in about an hour. Then I found out about the map thing, and am now even more skeptical. Unfortunately, I yearn for an easy solution now, after almost having a possible one.
Finally, I took a look at Cube, a simplistic multiplayer FPS. It's about 10000 lines of C++, counting headers. Someone could probably turn it into an accessible FPS in a couple weeks of work, but I don't think it even has the concept of NPC. It is also incredibly specific. I doubt it could be made into a more general sort of RPG.

If anyone has any information on using UDK without map editors, or any other engines that might work, please let me know. I'm still looking at other options.