Difference between revisions of "Lag"

From Armagetron
m
m
Line 1: Line 1:
 +
Have you wondered why sometimes you think you dumped the other guy only to find that your dumped yourself?  Does it ever feel like the things you do don't happen quite the same way to everyone on the grid?  There's a reason for that, and the reason is called ''lag''.
 +
 +
== Description of Lag ==
 +
 +
Lag is a pretty big topic because the word 'lag' is pretty ambiguous.  It means a number of things.  It is usually associated with your ping, but ping by itself is not a complete measurement of lag.  It's very difficult to quantify lag, so instead we'll talk about what causes lag.
 +
 +
=== Light Speed Now, Mr. Scott ===
 +
 +
As you've probably learned at one time or another, electrical signals travel at a speed very close to the speed of light.  Consider that it takes 8 minutes (approximately) for light from the Sun to reach the Earth, if you were to stretch a copper wire from the Earth to the Sun, it would take approximately 8 minutes for an electrical pulse to travel that distance.  So electrical signals don't travel instantaneously, they merely travel so fast that the conscious brain perceives it as instantaneous.  So, to start with, every time you press a button on your computer, the electrical pulse associated with that button press has to travel some distance from the button itself to a location in the computer, and then be sent out over the wire to the game server.
 +
 +
=== Hops, not beer ===
 +
 +
The signal's first hop is the game itself.  Of course, that doesn't mean anything unless you know what a hop is.  A hop is considered a point along the path where the signal has to be processed and understood, at least in part, to determine where it's going to go next.  So the first hop is your computer.  The keypress has to be sent up from the hardware to your operating system's kernel, which then has to send it to the program that wants it, in this case the game client.  The game client does some processing on it (did he turn?  Did he want to say something?), and then sends it's action down through the kernel to the network interface.
 +
 +
Follow?  This is not an instantaneous thing!  It is very very '''very''' fast, but it's not instantaneous.
 +
 +
The next hop is probably your router, if you have one.  If you're on a high speed network, you have one.  It may be called a "dsl modem", "dsl router", "cable modem", or whatever, but it's still a router.  It processes the signal to some extent, enough to figure out where to send it.  Then it sends it.  If you're running a local network attached to a cable ISP (a very common setup among players), then your first router sends it to the second router.  So now you've had 3 hops and the signal hasn't even left the house yet!
 +
 +
Then your ISP's router will pick it up, process it again, and determine where to send the signal.  Then their wholesale provider picks it up, and so on and so forth.  The signal will reach a point we generally consider the midpoint, although that's a bit of a misnomer, called the backbone.  This is the highest-speed and highest-traffic part of the internet.  It is the backbone of the internet!  From there the signal goes through another series of hops to get to the game server's house.
 +
 +
From there, it usually goes first through a high speed connection device, such as a dsl router or somesuch, and then goes to a regular router used in homes to setup home networks, and finally arrives at the game server.  Now what?  Well, now the game server processes your turn, some fraction of a second ''after'' you originally pressed the button to indicate your turn.  Is it done?
 +
 +
No, it has to send a signal back to your client when it marked the turn.
 +
 +
=== Measuring Latency ===
 +
 +
We refer to the time it takes for a signal on the network to leave its starting point, reach its destination, and have a response sent back as ''network latency''.  Network latency is one type of lag, it's the parent type.  It is the thing from which all lag is inherited.  It's the Sun Source Of Lag.  And it's measured in milliseconds, typically.
 +
 +
Armagetron Advanced gives you a number, called ''ping'', which is measured in milliseconds.  That number represents the amount of time it takes a signal from you to reach the game server and be processed on the game server, and then sent back.  The game takes regular measurements along the way and always tells you your most up-to-date ping.
 +
 +
== Reflexes ==
 +
 +
Now that you understand network latency, you should be able to make the jump to understand latency in your body's nervous system.  When your eyes process the image you see on the screen, they send a signal to your brain.  Your brain makes a decision on what to do, and then sends a signal down your arm or whatever to push whatever button it's decided to push.  We generally call this ''finger lag'', although there's certainly no requirement that fingers be involved.  Neuropsychologists have measured the time it takes for a signal to leave your brain and arrive at its destination along many paths, if you find this topic interesting you should certainly investigate it further.
 +
 +
== Bugs in the game? ==
 +
 +
Bugs in the game are usually blamed for lag.  In fact, you can tell who's at least new and/or who doesn't know jack about computers by how much they blame the game for the inaccuracies they see on the screen.  The simple fact is, Armagetron Advanced has a very solid network system that is relatively bug-free.  I'm not going to pretend it's perfect, but it's definitely good enough that if you suspect a bug and you didn't write the network code yourself, then you're probably wrong.  I'm not saying you are wrong, just that you're probably wrong.  I'd bet money that the lag you're experiencing and blaming the game for comes from somewhere else.
 +
 +
An associated comment usually comes along the lines of "but there has to be something in the game that can be done to compensate for it."  This statement frequently accompanies a list of other games where lag isn't as big of a problem.
 +
 +
=== Z-man's take ===
 +
 +
I'd like to add that the prime reason you feel more lag in AA than in, say, Counter Strike, is the difference in the game dynamics. In shooters, moving (thus defending by taking cover) and attacking (pointing your mouse and shooting) are independent actions. Taking a hit does not influence your movement immediately (it does later if you get wounded, or immediately if your pushed away by an explosion). Thus, it its possible to treat attack and defense separately in the network code and hide the lag. When you shoot at someone, you can shoot at the position you see him now, and even though your enemy really has moved away already, you'll still hit him because the server can compensate.
 +
 +
In AA, attacking is moving. Defending is moving. Getting hit has an immediate and strong impact on your movement. It's not possible to divide the different aspects of movement in the network code. (Well, it is, but the result would be that it would be impossible to survive close combat for both participants) In this sense, AA is a sports game, a beat em up to be precise. And you don't see them played over the net a lot precisely because they're heavily affected by network latencies (and because it's that much more fun if you're playing against someone in the same room).
 +
 +
== 6 pigs, 7 nipples! ==
 +
 +
You can also improve your lag by keeping other network usage down.  When you see someone's ping fluctuating in the game, it's usually a safe bet that they have a really large download going on somewhere, KaZaA or some other large downloading program.  Web browsing while playing the game will create more lag for you, which will in turn get pushed out to other players.  You have a limited amount of bandwidth available on your internet connection, and the game server has a limited amount of bandwidth available on its connection.  Some server admins will limit the number of players on their server to address this issue, but it's still not complete.  You have to behave yourself and keep your bandwidth usage low.
 +
 +
A corollary to this is when the game server in particular is sharing a connection with other servers.  For example, the Crack Pipe runs on a server that is also a web server, mail server, cvs server, and a few other things.  On top of that, it's used as an alarm clock, and as a regular desktop computer.  So it's using a fair amount of bandwidth just running, which gets pushed out to game clients.  But also under the heading of limited resources vs amount of demand, the Crack Pipe only has a 900mhz AMD Duron processor and 256MB RAM.  That's not really a lot of system resources for the amount of work it does.  So occasionally spam will be sent to the machine and it will have to spend some of its precious CPU time receiving the spam.  In the meantime, game clients suffer a little bit.  Some players have gotten to where they can identify how much spam the mail server receives while playing the game.
 +
 +
== What can be done about lag? ==
 +
 
''Lucifer sez''
 
''Lucifer sez''
  
Line 24: Line 78:
  
 
C / T > H
 
C / T > H
 
Smile
 
 
(The purpose of this post is to try to get people to realize that lag is part of the game and the best thing you can do to reduce the effect of lag on your game is to adapt to it as much as possible. Now cue Jonathan, who recently cleaned my clock while suffering insane lag on his end while I had pings below 20ms on mine....)
 

Revision as of 18:45, 28 October 2005

Have you wondered why sometimes you think you dumped the other guy only to find that your dumped yourself? Does it ever feel like the things you do don't happen quite the same way to everyone on the grid? There's a reason for that, and the reason is called lag.

Description of Lag

Lag is a pretty big topic because the word 'lag' is pretty ambiguous. It means a number of things. It is usually associated with your ping, but ping by itself is not a complete measurement of lag. It's very difficult to quantify lag, so instead we'll talk about what causes lag.

Light Speed Now, Mr. Scott

As you've probably learned at one time or another, electrical signals travel at a speed very close to the speed of light. Consider that it takes 8 minutes (approximately) for light from the Sun to reach the Earth, if you were to stretch a copper wire from the Earth to the Sun, it would take approximately 8 minutes for an electrical pulse to travel that distance. So electrical signals don't travel instantaneously, they merely travel so fast that the conscious brain perceives it as instantaneous. So, to start with, every time you press a button on your computer, the electrical pulse associated with that button press has to travel some distance from the button itself to a location in the computer, and then be sent out over the wire to the game server.

Hops, not beer

The signal's first hop is the game itself. Of course, that doesn't mean anything unless you know what a hop is. A hop is considered a point along the path where the signal has to be processed and understood, at least in part, to determine where it's going to go next. So the first hop is your computer. The keypress has to be sent up from the hardware to your operating system's kernel, which then has to send it to the program that wants it, in this case the game client. The game client does some processing on it (did he turn? Did he want to say something?), and then sends it's action down through the kernel to the network interface.

Follow? This is not an instantaneous thing! It is very very very fast, but it's not instantaneous.

The next hop is probably your router, if you have one. If you're on a high speed network, you have one. It may be called a "dsl modem", "dsl router", "cable modem", or whatever, but it's still a router. It processes the signal to some extent, enough to figure out where to send it. Then it sends it. If you're running a local network attached to a cable ISP (a very common setup among players), then your first router sends it to the second router. So now you've had 3 hops and the signal hasn't even left the house yet!

Then your ISP's router will pick it up, process it again, and determine where to send the signal. Then their wholesale provider picks it up, and so on and so forth. The signal will reach a point we generally consider the midpoint, although that's a bit of a misnomer, called the backbone. This is the highest-speed and highest-traffic part of the internet. It is the backbone of the internet! From there the signal goes through another series of hops to get to the game server's house.

From there, it usually goes first through a high speed connection device, such as a dsl router or somesuch, and then goes to a regular router used in homes to setup home networks, and finally arrives at the game server. Now what? Well, now the game server processes your turn, some fraction of a second after you originally pressed the button to indicate your turn. Is it done?

No, it has to send a signal back to your client when it marked the turn.

Measuring Latency

We refer to the time it takes for a signal on the network to leave its starting point, reach its destination, and have a response sent back as network latency. Network latency is one type of lag, it's the parent type. It is the thing from which all lag is inherited. It's the Sun Source Of Lag. And it's measured in milliseconds, typically.

Armagetron Advanced gives you a number, called ping, which is measured in milliseconds. That number represents the amount of time it takes a signal from you to reach the game server and be processed on the game server, and then sent back. The game takes regular measurements along the way and always tells you your most up-to-date ping.

Reflexes

Now that you understand network latency, you should be able to make the jump to understand latency in your body's nervous system. When your eyes process the image you see on the screen, they send a signal to your brain. Your brain makes a decision on what to do, and then sends a signal down your arm or whatever to push whatever button it's decided to push. We generally call this finger lag, although there's certainly no requirement that fingers be involved. Neuropsychologists have measured the time it takes for a signal to leave your brain and arrive at its destination along many paths, if you find this topic interesting you should certainly investigate it further.

Bugs in the game?

Bugs in the game are usually blamed for lag. In fact, you can tell who's at least new and/or who doesn't know jack about computers by how much they blame the game for the inaccuracies they see on the screen. The simple fact is, Armagetron Advanced has a very solid network system that is relatively bug-free. I'm not going to pretend it's perfect, but it's definitely good enough that if you suspect a bug and you didn't write the network code yourself, then you're probably wrong. I'm not saying you are wrong, just that you're probably wrong. I'd bet money that the lag you're experiencing and blaming the game for comes from somewhere else.

An associated comment usually comes along the lines of "but there has to be something in the game that can be done to compensate for it." This statement frequently accompanies a list of other games where lag isn't as big of a problem.

Z-man's take

I'd like to add that the prime reason you feel more lag in AA than in, say, Counter Strike, is the difference in the game dynamics. In shooters, moving (thus defending by taking cover) and attacking (pointing your mouse and shooting) are independent actions. Taking a hit does not influence your movement immediately (it does later if you get wounded, or immediately if your pushed away by an explosion). Thus, it its possible to treat attack and defense separately in the network code and hide the lag. When you shoot at someone, you can shoot at the position you see him now, and even though your enemy really has moved away already, you'll still hit him because the server can compensate.

In AA, attacking is moving. Defending is moving. Getting hit has an immediate and strong impact on your movement. It's not possible to divide the different aspects of movement in the network code. (Well, it is, but the result would be that it would be impossible to survive close combat for both participants) In this sense, AA is a sports game, a beat em up to be precise. And you don't see them played over the net a lot precisely because they're heavily affected by network latencies (and because it's that much more fun if you're playing against someone in the same room).

6 pigs, 7 nipples!

You can also improve your lag by keeping other network usage down. When you see someone's ping fluctuating in the game, it's usually a safe bet that they have a really large download going on somewhere, KaZaA or some other large downloading program. Web browsing while playing the game will create more lag for you, which will in turn get pushed out to other players. You have a limited amount of bandwidth available on your internet connection, and the game server has a limited amount of bandwidth available on its connection. Some server admins will limit the number of players on their server to address this issue, but it's still not complete. You have to behave yourself and keep your bandwidth usage low.

A corollary to this is when the game server in particular is sharing a connection with other servers. For example, the Crack Pipe runs on a server that is also a web server, mail server, cvs server, and a few other things. On top of that, it's used as an alarm clock, and as a regular desktop computer. So it's using a fair amount of bandwidth just running, which gets pushed out to game clients. But also under the heading of limited resources vs amount of demand, the Crack Pipe only has a 900mhz AMD Duron processor and 256MB RAM. That's not really a lot of system resources for the amount of work it does. So occasionally spam will be sent to the machine and it will have to spend some of its precious CPU time receiving the spam. In the meantime, game clients suffer a little bit. Some players have gotten to where they can identify how much spam the mail server receives while playing the game.

What can be done about lag?

Lucifer sez

Fix the internet.

As flippant as that is, it is the source of lag. We've experimented (we as in "server admins") with a number of solutions over the last year or so (and I assume it was happening earlier) and here's what we've found, near as I can tell.

If the machine is dedicated to the armagetron server, it will have less lag. Even my server has some interesting lag effects whenever it receives mail (it's a mail server too), and that's with the armagetronad process reniced to a low level (-15 I think it is).

If the machine is on a slow processor with a small cache, it will have more lag. This effect is amplified if the machine is not dedicated to the game and/or if you run more than one game server.

Lag is reduced by parallel processing on the server, so if you have a fancy Xenon processor or a dual proc setup you can have less lag.

Lag is reduced by running a 2.6 Linux kernel over *any* other kernel (Windows, Linux 2.4, I dont' know that any others have been tested in this way). It can probably be reduced even further by applying the low-latency patches, but since those are mostly used for audio work I don't know how well they'd work here.

The closer the server is to the backbone, the less lag. The closer the client is to the backbone, the less lag the client has. (I'm pretty far from it, hence the high pings on teh Crack Pipe, but they do look better than they used to be)

After you've done all of this, the lag will be noticeably less, but there will still be numerous complaints. This is because the internet itself is the source of all lag. So all of these measures only decrease the roundtrip time for a packet that has already crossed the internet and reached the machine.

There have been a number of improvements in this area. 0.2.7.1 had simulation improvements, and also an improvement where the server can be anchored to a certian number of simulation steps per minute. So that's why 0.2.7.1 servers have more predictable lag than older servers, and when everyone is connected with newer clients, you can really see the difference. 0.2.8_beta2 has a major network refactoring that also seems to reduce lag, even though it may not be noticeable in your ping. I don't know that it would be noticeable in your ping, and since there aren't yet that many servers running the new betas the only measure I have is my own, which has less lag than the older version.

The only improvements I can think of would be to optimize code here and there, and I don't even know which code needs to be optimized. I know that periodically some one or more developers profile various pieces of code and that there has been optimizing. I also know that most of the developers around here think about performance while coding. ANd I also know that there is code that is not well optimized (although I can't point to any specific line, if I could do that, I could optimize it, right?).

So let's assign a value, C, which represents how much work is required to thoroughly optimize the code and reduce latency to bare minimum while retaining reliability and accuracy. Now let's assign a value, T, which represents how much time the entire development team has to dedicate to exactly that. Let's assign one more variable that represents how long before really fast internet arrives in everyone's houses (faster than even broadband now) and includes faster processors, faster network cards, and generally very low latency on the hardware end. We'll call this one H, for "toy".

C / T > H