19

Think MUDs/MUCKs but maybe with avatars or locale illustrations. My language of choice is ruby.

I need to handle multiple persistent connections with data being asynchronously transferred between the server and its various clients. A single database must be kept up-to-date based on activity occurring in the client sessions. Activity in each client session may require multiple other clients to be immediately updated (a user enters a room; a user sends another user a private message).

This is a goal project and a learning project, so my intention is to re-invent a wheel or two to learn more about concurrent network programming. However, I am new to both concurrent and network programming; previously I have worked almost exclusively in the world of non-persistent, synchronous HTTP requests in web apps. So, I want to make sure that I'm reinventing the right wheels.

Per emboss's excellent answer, I have been starting to look at the internals of certain HTTP servers, since web apps can usually avoid threading concerns due to how thoroughly the issue is abstracted away by the servers themselves.

I do not want to use EventMachine or GServer because I don't yet understand what they do. Once I have a general sense of how they work, what problems they solve and why they're useful, I'll feel comfortable with it. My goal here is not "write a game", but "write a game and learn how some of the lower-level stuff works". I'm also unclear on the boundaries of certain terms; for example, is "I/O-unbound apps" a superset of "event-driven apps"? Vice-versa?

I am of course interested in the One Right Way to achieve my goal, if it exists, but overall I want to understand why it's the right way and why other ways are less preferable.

Any books, ebooks, online resources, sample projects or other tidbits you can suggest are what I'm really after.

The way I am doing things right now is by using IO#select to block on the list of connected sockets, with a timeout of 0.1 seconds. It pushes any information read into a thread-safe read queue, and then whenever it hits the timeout, it pulls data from a thread-safe write queue. I'm not sure if the timeout should be shorter. There is a second thread which polls the socket-handling thread's read queue and processes the "requests". This is better than how I had it working initially, but still might not be ideal.

I posted this question on Hacker News and got linked to a few resources that I'm working through; anything similar would be great:

Community
  • 1
  • 1
Max Cantor
  • 8,229
  • 7
  • 45
  • 59
  • 2
    FWIW in software development there is almost never "one right way", and in real world scenarios in a game like this the database would be your primary limiting factor. The performance of the application code will most likely far exceed the performance of the database, and you would need to look into caching solutions or database sharding solutions. Sharding could be very appropriate with a game since you could shard on game areas etc. – Chris Marisic Aug 08 '11 at 14:20
  • I think the in-game resources are going to be slim enough that I can keep the entire world in memory, while maintaining an "update queue" of every change that gets written to the database in its own separate thread. That way, none of the game itself is bound on database operations, and the only "read" would be on bootup. Lots of risk in terms of error handling, but convenient as hell if I can make it work. – Max Cantor Aug 08 '11 at 15:12
  • 1
    That's plausible for a low scale solution, but I think you might be underestimating your game in regards to DB usage. You need to keep track of inventories, player and monster stats/health/location etc. Inventory database lag is one of the most common problems that plague MMO development. – Chris Marisic Aug 08 '11 at 16:27

4 Answers4

13

Although you probably don't like to hear it I would still recommend to start investigating HTTP servers first. Although programming for them seemed boring, synchronous, and non-persistent to you, that's only because the creators of the servers did their job to hide the gory details from you so tremendously well - if you think about it, a web server is so not synchronous (it's not that millions of people have to wait for reading this post until you are done... concurrency :) ... and because these beasts do their job so well (yeah, I know we yell at them a lot, but at the end of the day most HTTP servers are outstanding pieces of software) this is the definite starting point to look into if you want to learn about efficient multi-threading. Operating systems and implementations of programming languages or games are another good source, but maybe a bit further away from what you intend to achieve.

If you really intend to get your fingers dirty I would suggest to orient yourself at something like WEBrick first - it ships with Ruby and is entirely implemented in Ruby, so you will learn all about Ruby threading concepts there. But be warned, you'll never get close to the performance of a Rack solution that sits on top of a web server that's implemented in C such as thin.

So if you really want to be serious, you would have to roll your own server implementation in C(++) and probably make it support Rack, if you intend to support HTTP. Quite a task I would say, especially if you want your end result to be competitive. C code can be blazingly fast, but it's all to easy to be blazingly slow as well, it lies in the nature of low-level stuff. And we haven't discussed memory management and security yet. But if it's really your desire, go for it, but I would first dig into well-known server implementations to get inspiration. See how they work with threads (pooling) and how they implement 'sessions' (you wanted persistence). All the things you desire can be done with HTTP, even better when used with a clever REST interface, existing applications that support all the features you mentioned are living proof for that. So going in that direction would be not entirely wrong.

If you still want to invent your own proprietary protocol, base it on TCP/IP as the lowest acceptable common denominator. Going beyond that would end up in a project that your grand-children would probably still be coding on. That's really as low as I would dare to go when it comes to network programming.

Whether you are using it as a library or not, look into EventMachine and its conceptual model. Overlooking event-driven ('non-blocking') IO in your journey would be negligent in the context of learning about/reinventing the right wheels. An appetizer for event-driven programming explaining the benefits of node.js as a web server.

Based on your requirements: asynchronous communication, multiple "subscribers" reacting to "events" that are centrally published; well that really sounds like a good candidate for an event-driven/message-based architecture.

Some books that may be helpful on your journey (Linux/C only, but the concepts are universal):

(Those were the classics)

  • The Linux programming interface - if you just intend to buy one book, let it be this one, I'm not entirely through yet but it is truly amazing and covers all the topics you need to know about for your adventure

Projects you may want to check out:

emboss
  • 38,880
  • 7
  • 101
  • 108
  • Thanks for the thorough response! I expressed myself poorly, I didn't mean that I don't *want* to understand how EventMachine works. What I meant was, I don't want people to say "Just use EventMachine". So, no need to apologize! That is not what you are saying, and I appreciate the suggestion :-) – Max Cantor Aug 01 '11 at 22:06
  • By the way, I was thinking about what you wrote, and I don't see how HTTP would be the best tool for this. I loved your explanation of why HTTP servers are a good learning resource, and I agree, but the need for real-time updates being pushed from server to client make HTTP a poor choice for the task of a game server. It takes clever tricks like [Comet](http://en.wikipedia.org/wiki/Comet_(programming)) to achieve what TCP sockets get you for free. Am I off-base here? – Max Cantor Aug 04 '11 at 19:58
  • 1
    Yes, you're absolutely right there. HTTP is poor when it comes to receiving events from a server. Most of the time this is handled by polling continuously (apparently not ideal) or by something really sophisticated as Comet, so HTTP is really acting more on the pull than on the push side. I was just having a web-based game in mind when I read/wrote this, so I thought you'd have to use HTTP anyway(maybe I misunderstood). But if that's not a requirement, then you are free to choose your protocol and I'm almost certain that there is something (TCP-based) that fits better than HTTP in that case. – emboss Aug 05 '11 at 00:09
  • 1
    Like TCPServer? I can direct you to resources and suggest an architecture approach for using TCPServer with Telnet Negotiation if that's what you're looking for. – stslavik Aug 11 '11 at 16:35
  • Yeah, that would be pretty rad! – Max Cantor Aug 12 '11 at 20:12
3

I'd recommend reading a bit about unicorn web-server design. That should give you some insight into thread vs. process discussion.

http://unicorn.bogomips.org/DESIGN.html

http://tomayko.com/writings/unicorn-is-unix

Slotos
  • 398
  • 2
  • 10
2

I don't know much about Ruby - sorry - but I think the architecture needs to be driven by your requirements (motherhood, apple pie...I know).

If you're building something that needs to scale to large numbers of users, your architecture needs to reflect that - and you end up making all sorts of decisions you don't necessarily need to make if you're operating at more modest scale.

Response time also plays a big part - I don't think that is such a big deal with MUD style games, but for web servers or FPS games, it's a huge issue.

Having said that - the only systems similar to what you describe that I know in any detail use an event driven programming model - clients trigger events, the system updates its internal state, and notifies affected clients. The "internal state" is actually stored in a separate application, again communicating using sockets - this allows the app to be scaled by adding more servers to handle client interaction. Not sure you need that level of complexity.

Threading is indeed a real issue, and it creates hard-to-test code, so whilst dash-tom-bang's answer is indeed a little off topic, testability is a significant concern.

Neville Kuyt
  • 29,247
  • 1
  • 37
  • 52
-1

The right way is to use test driven development. You will then discover what you need to reinvent at the exact moment that you need to invent it.

Start with a test that "connects" and assert that the message "hello, user" is returned. Take it one step at a time from there.

dash-tom-bang
  • 17,383
  • 5
  • 46
  • 62
  • 5
    I downvoted your answer because I asked about architecture and your answer was about programming methodology, which makes it irrelevant. I'm actually a huge fan of test-driven development, and I have been using it on this very project; but its utility is orthogonal to the question of high-level server architecture. – Max Cantor Aug 04 '11 at 20:06
  • Yeah, Test Driven Development (as opposed to simply unit testing) guides your architecture, but yours is a common misunderstanding. That's cool though, no hard feelings; we can't all have winning answers. :) Anyway- my opinion is that there's no better way to learn than to define requirements and then implement to those requirements. – dash-tom-bang Aug 05 '11 at 06:01
  • 3
    I can't resist... are you really suggesting that by adhering carefully to the tenets of TDD, I will produce performant, concurrent code with no race conditions, memory leaks or unexpected behavior whatsoever, without referring to *ANY* other resources about this problem domain? – Max Cantor Aug 05 '11 at 18:22
  • 2
    Sorry, my understanding was that you wanted to "reinvent a wheel or two" and that you were not targeting a world class epic server to handle any possible loads. When *I* want to learn how to do something new this is how I do it. If you want to create something best-of-breed, well, clearly you'll need to see what others have done and build on that. – dash-tom-bang Aug 09 '11 at 18:49