openGame Project Update Thread

Post all general things in here.

Moderator: Moderators

Real Member
User avatar
Posts: 41
Joined: 11 May 2015 03:22

Re: openGame Project Update Thread

Postby Marcos » 18 May 2015 23:00

_nOx_ wrote:I'm illuminating Stat's path with the help of a focus prism. Follow the light, Marcos, there's probably a camp-fire with slightly burnt ribs there.

I do not understand you Nox, when you talk about Illuminati refers to this image?
Attachments
Sign on the Dollar.gif
Sign on the Dollar.gif (69.52 KiB) Viewed 3814 times

Board-General
User avatar
Posts: 303
Joined: 05 Mar 2012 06:32
Location: United States

Re: openGame Project Update Thread

Postby stAtrill » 19 May 2015 03:36

This is the post I referenced earlier, with the dev progress notes (AKA a build diary, basically)!


This is something of a mega-update, so I am calling this one version 0.55:


Changelog, from the Github commit: <-(This is a link, btw, to the changelog history)
-TUNTAP class completed
-Now manages ethernet headers
-Better diagnostics
-Streamlining
-Backup loopback broadcast detection method implemented
-Loads of bugfixes
-Bug fixed that prevented injecting data on the adapter
-Bug fixed that caused crashing after connections completed
-Bug fixed that caused the injecting code to crash when passing data to
the writing thread
-Files added for eventual build to .exe
-595 lines now T.T Mind you, this doesn't tell half of the story.

I have written at least 200 more, then scrapped them while looking for workarounds to what I was sure was a bug in the library I am using to interface with windows. After filing a bug report, looking all over the web, and digging in the source code, I learned that is was partially the docs being wrong, and partially differences between python code versions (I am using the newer 3, while all of the dinosaurs are still using the old 2). Tracking this caused so much frustration that I would get massive headaches and have to get up and go do something outside.

Anyway, the Tun/Tap class is complete, and is modular in a way that would allow someone to simply cut/paste the class out and have a fully working device manager. Hopefully this helps someone else out.

I am already starting to look to the future: I am looking up protocols that can be used for chat/VoIP functionality (so far looked at Tox, and torrent DSC), and am starting to read whitepapers on hole punching. Maybe, if I can manage to my head around STUN, I will get the hole punching in sooner rather than later, as I was already having trouble with a direct connection to some of the people I tried to test with.

There are some known bugs to contend with:
-A hash ('lobby', for our purposes) can only be used once, as if someone disconnects and tries to reconnect, all other connected clients crash (connection forcibly closed). This should be easy to fix, and would prevent having to repeatedly increment the hash while testing.
-There is a relatively stealthy bug that I detected with the packet sniffer - sometimes, the tap adapter will pass two packets at once (as one packet), which we then try to send (as one packet), which then gets rejected by the destination computer for being invalid. This seems to be rare enough that it won't affect a connection, but I think I still need to do some deeper study of the OVERLAPPED structure in order to resolve it. Probably an easy fix, once I discover the cause.
-I think I may have found a temporary workaround, allowing joining and playing of GC games, by simply renaming the TAP adapter to the public IP address. This shouldn't break anything, strangely enough, since the LAN can't access it, and it essentially is another loopback address.


What is still preventing us from using this to play games:
-An easy method of distribution (i.e. exe file), as I currently have been having everyone download the development environment, and this causes problems as it is platform-specific (and people don't always know their platform, etc). There are some problems here as well, as the windows API wrapper I am using refuses to be built into an exe, so I am still looking for solutions.
-Ability to join game lobbies. This entails adding support for packets with a non-broadcast address (uber-easy), and adding routing (or verifying the configuration based workaround above). If the config workaround works, this also becomes as easy as 3 clicks (which I can automate in the program with like 2 lines).
-And, finally, a method to get around everyone's horrible connection. Adding intelligent morphology/routing is a part of this, but this is ultimately where hole-punching and upnp come into play.


That said, solutions to the above (and the high-priority bugs) are top priority. Everything else after simply ices the cake.

Expected in next update:

-Not the selectors module, as callbacks make code a mess.
-Bugfixes
-Hopefully, a pre-compiled version
-The ability to actually play a game over it (if the ports are already setup beforehand, and barring certain poor connections)

And that's the way it was,
-Statrill

Board-General
User avatar
Posts: 134
Joined: 24 Jan 2012 03:16

Re: openGame Project Update Thread

Postby _nOx_ » 20 May 2015 14:23

Marcos wrote:I do not understand you Nox, when you talk about Illuminati refers to this image?
Oh, the good old 1776 seal. Those were the days, man.

Keep the diary going, Stat. It's actually very interesting to follow.

Board-General
User avatar
Posts: 303
Joined: 05 Mar 2012 06:32
Location: United States

Re: openGame Project Update Thread

Postby stAtrill » 21 May 2015 23:01

Just a tiny update: was testing with my brother last night to try to iron out a few bugs and see if the workaround actually worked.

Something had been causing problems for a while (minor problems, but still): when I connect to a tracker, the tracker shows my IP as 26.5.0.19, rather than my real IP, which is 174.109.X.X.

Whenever I connect to anything that isn't a torrent tracker, my 174.X.X.X address is returned. Only torrent trackers get the 25.X.X.X IP. Long story short; after a bunch of testing, I was starting to suspect I was the victim of a DoD Man-in-the-Middle attack when connecting to torrents (the 26.X.X.X block is owned by the DoD and is non-routable).

This had been going on since I started writing the program, btw. Strangely, as soon as I start asking around the internet about it, trackers start returning my normal IP. I guess the best way to solve bugs due to eavesdropping is let your eavesdroppers know you know they are listening.

Although cause for temporary chills, I remind myself that that is one of the ulterior motives for writing this program: to route more and more 'legitimate' traffic over the torrent infrastructure.

Also, from our testing, the program will now start being able to close and handle closed connections. Finally, it looked like the workaround was going to work, but I quickly got sidetracked into tests trying to figure out what was going on with my connection.

Screenshots later, if it works.

-Stat

Edit: it just occurred to me that some people may not know: the 26.X.X.X block is owned by our Department of Defense Network Information Center. Almost certainly also involved in civilian wiretapping.

Board-General
User avatar
Posts: 134
Joined: 24 Jan 2012 03:16

Re: openGame Project Update Thread

Postby _nOx_ » 22 May 2015 02:35


Real Member
User avatar
Posts: 41
Joined: 11 May 2015 03:22

Re: openGame Project Update Thread

Postby Marcos » 22 May 2015 03:56


Nox, This has something to do with GC?

Board-General
User avatar
Posts: 105
Joined: 27 Jan 2012 17:21

Re: openGame Project Update Thread

Postby Sarvik » 22 May 2015 07:59

Omg, into how many blacklists we managed to put Stat with this? You need to finish it fast before black helicopters come and take you away!

Board-General
User avatar
Posts: 303
Joined: 05 Mar 2012 06:32
Location: United States

Re: openGame Project Update Thread

Postby stAtrill » 22 May 2015 22:46

EVERYBODY LOOK!!!!

The workaround works! :D

Image

I completely planned to post a screenshot of us in game, but he crashed as he didn't have the fricking update mod! I just made him download it now.

Also, I did some extra work - it now can *almost* completely handle disconnected connections, and I have been studying on both upnp and how to GET THE FRICKING THING TO BUILD AS AN EXE.

-Stat

EDIT: screenshot after he grabbed the update:
Image

Board-General
User avatar
Posts: 106
Joined: 26 Jan 2012 04:10

Re: openGame Project Update Thread

Postby Admiral Ghat » 23 May 2015 06:18

The possibility of a reliable, bug-free GC multiplayer experience...can almost taste it! :D Thanks for all the hard work Stat!

Board-General
User avatar
Posts: 303
Joined: 05 Mar 2012 06:32
Location: United States

Re: openGame Project Update Thread

Postby stAtrill » 27 May 2015 17:27

Just checking in to give some updates!

Unfortunately, I haven't been able to work on this much lately, but I have gotten some crucial functionality in (that made the game with my bro possible), in addition to making significant progress on quite a few other fronts. Basically, it can host a game in 'workaround' mode (until routing is finished), it builds as an .exe now (!!!), and it can handle dropped connections to an extent (needs to be finished).


Let's call this one version 0.6:


Changelog, from the Github commit: <-(This is a link, btw, to the changelog history)
-Switched .exe platform from cx_freeze to py2exe
-Successfully builds as .exe
-Improvements to connection manager:
-->Now can handle forcefully closed connections
-->May be able to handle normally closed connections, need to test
-->Framework in place to retry closed connections, currently unused
-Can now handle non-broadcast data
-Various minor improvements
-Improvements to diagnostics
-a modest 622 lines, so far

So this one was interesting - in the version I tested with my brother, I first had to fix dropped connection functionality and I thought I had to fix non-broadcast functionality. Of course, I first made the program able to handle forcefully closed connections (which in theory should only happen if the program is forcefully closed [i.e. crash, end process, etc]). However, I realized that anytime the program closes, the connections are force-closed, and this led to me discovering that is program isn't shutting down correctly. Oops. I need to look more into this, but at least it causes the program to properly handle the disconnection of other peers for now (nothing like a bug and missing functionality working together to give stability).

Getting the .exe build working properly was a royal pain in the ass as well, it turns out that documentation suggesting cx_freeze plays better with python 3 than py2exe does is out of date. I switched over to py2exe, as they made another big update last year that makes py2exe once again the go-to for building with python. With some minor modification to the includes, everything works as expected! This is a boon to deployment, but creates some other downsides, as currently there are some settings that you can't change without having access to the source code. Need to do some work on this.

And finally, I have finished my research on UPnP, and should be integrating it soon.

The bug list, as it stands now:
-FIXED: Force-closed connections crash openGame.
-The program does not properly shutdown, a downside (or upside) being that instead of 'soft' closing connections, they always force close
-There is a relatively stealthy bug that I detected with the packet sniffer - sometimes, the tap adapter will pass two packets at once (as one packet), which we then try to send (as one packet), which then gets rejected by the destination computer for being invalid.


What is still preventing us from using this to play games (even though it technically can play one as-is):
-FIXED: An easy method of distribution (i.e. exe file)
-FIXED: Ability to join game lobbies. This entails adding support for packets with a non-broadcast address (uber-easy), and adding routing or verifying the configuration based workaround above.
-Routing. Period. The workaround can sometimes give game-ranger-like results (namely, joining a room instantly fails), and this isn't acceptable.
-And, finally, a method to get around everyone's horrible connection. Adding intelligent morphology/routing is a part of this, but this is ultimately where hole-punching and upnp come into play.

Also, I think I discovered why GameRanger is having so much trouble with GC. For starts, GR avoids having to install a virtual network adapter (which saves hassle while install, but the adapter provides sooooo many advantages). The only way you can get around having an adapter is to hook into the APIs that the games themselves load for networking (and, as we have suspected for years, also the 3D directDraw API, causing win8 slowness), this is likely why GR needs to know where the program is installed. Or, in other words, GR currently operates in what we are calling our 'workaround' mode: they hook into the game at game start, and modify the IP addresses that the game thinks it is talking to so that the game can chat across the internet. Hole-punch the router, and you should be good to go right?

Well, no. The problem is GC uses a fuckload of different ports (this isn't necessarily good network programming style, btw) ranging from 25xxx and 26xxx to 65xxx etc. Generally, a hole punch will only open a single port (though in theory you could punch as many as you want). In other words, you can't really just hole punch your way around it, and then simply let GC talk directly across the internet. I should mention that, if GC tries to send enough information across those ports to peers, it may effectually punch itself, but until that happens, guess what it looks like: chumping. BTW, if GC manages to 'punch itself through', the holes will close again if nothing keeps them open (sometimes in a span as short as 20s), and since nothing is managing them (GC just assumes the ports are open all the time), you can conceivably start chumping again on a connection that you just opened.

Our solution will work for the same reason Tunngle was a much more stable solution: we take our virtual adapter and route ALL GC traffic over it, regardless of port. We then route and tunnel it properly to where it needs to go over a single connection/port, and we can punch that for reliable operation (or UPnP it, etc).



Expected in next update:

-Routing/Tunneling
-UPnp (aka, automatic port forwarding)

BTW, the above github link has a /build folder with an .exe version of the file in it if anyone wants to play around with the program. It needs the 3 .dll files in the same folder with it to run however, and the TunTap adapter must be installed. In some future version the program will be able to detect a missing adapter and download it for you, but until then: https://swupdate.openvpn.org/community/ ... .9.2_3.exe

Very close,
-Statrill

PreviousNext

Return to Ground Control - General

Who is online

Users browsing this forum: No registered users and 2 guests