GDR Forum Index
Podcast Podcast
Dev Dev Logs
Search Search
RSS RSS
Register Register
Log in Log in
Reply to topic GDR Forum Index -> Game Developer's Refuge -> Development Log - This And That (unfocused randomness) Page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next
View previous topic :: View next topic  
Author Message
Gil
Developer

Joined: 14 Nov 2005
Posts: 2341
Location: Belgium
PostPosted: Fri Aug 22, 2014 3:52 am    Post subject: Reply with quote

Whenever Dennis does anything with the c64 palette, a fairy is born
_________________
PoV: I had to wear pants today. Fo shame!
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Sat Aug 23, 2014 1:19 am    Post subject: Reply with quote

The world is always in need of more fairies, since most people(myself included) forget to clap their hands after sneezing.

Problem: How do I solve the dynamic loading of assets and knowing about loading progress?

I tried doing that using XmlHttpRequest and .onprogress with mixed results:
page served via http:// > works in Chrome, IE and FF
page served via file:// > works in FF, works in Chrome(only if Chrome is ran with an extra flag, same as for running Workers in an external script file), fails in IE (no workaround available)

The problem is that Chrome and IE consider that a security problem when the script is ran locally from a file:// URL.

Now I am thinking this could be solved partly (for anything that can be put into the HTML document directly like <img>) by testing for the protocol(file:) in use and appending the data to the document as a hidden HTML element but then there would be no way of determining the loading progress(only when it's finished loading) and it would not work for custom/arbitrary binary data files which is a problem.

All of this is only a problem for when I would like to provide a .zip of my application which users can download and open locally in their browser. Bundling a very light-weight, very easy to configure http-server with the .zip might work as well. That light-weight http-server would have to be really simple though, as simple as starting its executable in the application dir and having it open the installed browser and point it to http://localhost:port and serving the contents of the dir it was started in.

edit: Aha... seems like http://cesanta.com/mongoose.shtml does just that. The executable is just 144kb on Windows and if copied into any dir and ran there, it will just start serving that dir on port 8080 and point the browser to localhost:8080.

edit2: Hm... but the mongoose license does not allow redistribution, so it is not suitable for bundling it within with a .zip.

edit3: Actually... bundling a webserver is a bad idea anyway since that webserver would have to be platform specific, so that's not an acceptable solution to the "loading of arbitrary resources from URL while using the file: protocol" described above.
_________________
0xDB
View user's profile Send private message Visit poster's website
PoV
Moderator

Joined: 21 Aug 2005
Posts: 10904
Location: Canadia
PostPosted: Sat Aug 23, 2014 8:59 pm    Post subject: Reply with quote

Annoyingly, Mongoose only recently changed it's licence. Until a few years ago it was MIT licensed.

But ya, the file:// thing is pretty dumb. I'm not sure what you're trying to do, but testing locally is definitely best done with a web server. Nowadays I'm using node.js a lot more, and with some common libraries installed you can run a webserver with just 3 lines of code.

There's something also called Node-Webkit, which is supposed to combine Node with Chromium. I think it's possible to run a web server the same way, and have it serve the pages you want. Not as easy as it should be though to package an app.
_________________
Mike Kasprzak
'eh whatever. I used to make AAA and Indie games | Ludum Dare | Blog | Tweetar
View user's profile Send private message
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Sat Aug 23, 2014 11:53 pm    Post subject: Reply with quote

What I'm trying to do is loading custom binary data files(anything that does not have a well kwnon mime type and thus can not be interpreted by the browser, e.g. custom game/map data) into an ArrayBuffer or Blob. With the http: protocol and XmlHttpRequest that just works fine. With the file: protocol it fails on two out of three main browsers because of the same-origin security policy.

With the file: protocol I tried to add hidden < embed > elements to the html body as application/octet-stream but the problem is there seems to be no way to get to the actual bytes of the file attached to it, because browsers simple do not load the file, if they don't know the mime type and so they end up displaying "plugin needed to display this content" box(if the element is set to visible).

So I'm thinking a hacky workaround for the file: protocol could be to convert/store any custom binary data files into 8-bit .png files. Then instead of getting the files via XmlHttpRequest, I could append them to the html body as hidden image elements(when viewed, they'd just appear as grayscale garbage to the human eye) and access the data bytes via a hidden canvas element and a combination of .drawImage, .getImageData calls. It would work but it would require additional work at design-time for the data conversion.

...

I tried node-webkit yesterday. Inside node-webkit, as in FireFox the security policies for the file: protocal seem to be a bit more relaxed so the file: protocol works there with XmlHttpRequest. For local testing, debugging, using node-webkit is inconvenient though, because on errors it navigates away from your page and displays an error page with (mostly useless) generic Exception information on it instead of opening the debugger and highlighting the line which caused the problem in your code.

Bundleing node-webkit with my app is something I would like to avoid if possible because it adds about 30MB to the download... which seems excessive for an app that might end up being just a few hundred KB itself(including code and data). Also, bundleing node-webkit creates platform-specific downloads where just zipping up the .html and .js and any data files would be fully cross-platform already(provided end-user has a compatible browser installed).

So... nonetheless, I have to make a decision now, so I won't stall my progress any further (I would rather spend time implementing features than trying to do something the base technology does not support very well :P).

My decision is to use XmlHttpRequest for now and start a "known-issues" textfile where I keep track of quirks, so I can revisit them should I ever come as far as having something usable up and ready for sharing as a download. For pain-free local testing I will use the free version of mongoose but I will also keep testing with the file: protocol directly to keep compatibility with that as complete as possible.
_________________
0xDB
View user's profile Send private message Visit poster's website
PoV
Moderator

Joined: 21 Aug 2005
Posts: 10904
Location: Canadia
PostPosted: Sun Aug 24, 2014 12:17 am    Post subject: Reply with quote

Quick bit of digging, the consensus is that file:/// is a bad idea in general. A web server is somehow less exploitable. Its there as a convenience for local HTML documentation, not for apps.

There were a set of APIs proposed for proper file access, but they didn't catch on and are obsoleted. Firefox I think, which could explain why it was allowed for you.


At this point, I always Dev with a local webserver, and just assume there will be no local copy (not HTML anyway). But NodeWebkit is a solution, just not necessarily one for typical end users. Chrome gets regular updates every 6 weeks. That means the chromium will get stale fairly quick. Easier to just host it and run in the browser. ;)
_________________
Mike Kasprzak
'eh whatever. I used to make AAA and Indie games | Ludum Dare | Blog | Tweetar
View user's profile Send private message
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Sun Aug 24, 2014 12:56 am    Post subject: Reply with quote

I agree. Maybe at some point it will be possible to implement a very simple lightweight http server directly in client-side javascript... we could then run that in a Worker. The local app served from a file: could then first start that Worker and then switch to serving itself from there instead of from the file:.

Hm... now I am wondering... if I would run the XmlHttpRequest inside a Worker (whose script would be generated at runtime, just like the worker script for the timer)... the origin of the request would be that in-memory Worker-Blobs URL and not a file:/// anymore. The question is, do I dare to waste those hours it's probably going to take get that test running? ...the tension caused by curiosity is unbearable. So at some point I might actually try that. (edit: ok, quick google reveals that won't work either because the file to get via XmlHttpRequest would still be in a file: origin of course while the blob would be in a blob: origin... so yeah, no XmlHttpRequests for file: and no easy workaround is pretty much a given restriction to just accept)

So... for downloads... as inconvenient as that may be for end-users, the easiest way would be to include instructions and links pointing towards a simple http server to use for running the app locally.
_________________
0xDB
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Sun Aug 24, 2014 9:48 pm    Post subject: Reply with quote

nice: http://stuk.github.io/jszip/
The minified script weighs in at about 74kb and can be used under MIT license... also, I think I'll put Refugee Lib under MIT License as well as it seems to be the most sane license I've read in a long time.
_________________
0xDB
View user's profile Send private message Visit poster's website
PoV
Moderator

Joined: 21 Aug 2005
Posts: 10904
Location: Canadia
PostPosted: Mon Aug 25, 2014 6:48 am    Post subject: Reply with quote

Heh, you know, I know 74K isn't much, but here I am thinking to myself "why isn't that smaller"? ;)

And yes MIT is sensible. BSD is similar too, but I occasionally call it it BNC, and I'd rather not look dumb. ;). GPL should die in a fire. It's effectively the same as closed source, but with the source wide open in a mocking "you can't actually use this" way. Software can operate in such an environment, but it's the death touch for libraries.
_________________
Mike Kasprzak
'eh whatever. I used to make AAA and Indie games | Ludum Dare | Blog | Tweetar
View user's profile Send private message
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Mon Aug 25, 2014 7:50 am    Post subject: Reply with quote

Those 74kb added to the lib will save many kb in the process of loading resources. But yeah, it should be unnecessary for applications which generate all (game) data on the fly, so I'll tailor my data manager to only use it when it is there. On the other hand... I could maybe make that the first script in the lib and then have an initializer which loads all other scripts dynamically from a .zip which could save many more kb in the long run the bigger the lib grows.

PoV wrote:
BSD is similar too, but I occasionally call it it BNC, and I'd rather not look dumb. ;).
I don't get it.
_________________
0xDB
View user's profile Send private message Visit poster's website
PoV
Moderator

Joined: 21 Aug 2005
Posts: 10904
Location: Canadia
PostPosted: Mon Aug 25, 2014 7:00 pm    Post subject: Reply with quote

BNC is an old networking standard with coax cables, not an open source license. All there is to get is that I get it wrong.

* * *

Decided to check on an LZMA compressor.

https://github.com/nmrugg/LZMA-JS

It's about 134k raw, 104k minified. So far that ZIP lib seems a good value. :)


I'm mainly curious what the smallest good compressor I can find is. I know there's something out there that packs code in PNGs, since PNG (zlib) compression is nearly free (just some troublesome conversion steps).
_________________
Mike Kasprzak
'eh whatever. I used to make AAA and Indie games | Ludum Dare | Blog | Tweetar
View user's profile Send private message
Sirocco
Moderator

Joined: 19 Aug 2005
Posts: 9461
Location: Not Finland
PostPosted: Mon Aug 25, 2014 8:51 pm    Post subject: Reply with quote

Quote:

BNC is an old networking standard with coax cables, not an open source license. All there is to get is that I get it wrong.


Do I get bonus points for working with old equipment that uses BNC connectors at my previous job(s)? Heh.
_________________
NoOP / Reyn Time -- The $ is screwing everyone these days. (0xDB)
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Tue Aug 26, 2014 10:19 am    Post subject: Reply with quote

progress is slow... screwing around a lot between implementing features
  • improved timing in rlEngine (tries to adapt to browser timer resolution inaccuracies on the fly now to reach target LUPS with a bit of tolerance)
  • fixed instance state sharing bugs in rlEngine and rlData, multiple engines test
  • fixed error/progress/load handling in rlData (XmlHttpRequest fires progress and load events, even when the request "failed"(as in did not load what it was supposed to load) but never seem to fire error events, so it is necessary to check the http status code of the progress and load events to determine whether the request does/did what it was supposed to do)
  • started implementing rlDataManager (no zip loading/exploding yet) also still untested


Test 4: multiple engines and data loading (wip):

_________________
0xDB
View user's profile Send private message Visit poster's website
Sirocco
Moderator

Joined: 19 Aug 2005
Posts: 9461
Location: Not Finland
PostPosted: Tue Aug 26, 2014 12:27 pm    Post subject: Reply with quote

:3

Hmmm.... multiple instances of the engine? Now that gets my brain going to weird places.
_________________
NoOP / Reyn Time -- The $ is screwing everyone these days. (0xDB)
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Tue Aug 26, 2014 1:24 pm    Post subject: Reply with quote

Don't know myself if I will ever have a real use for running multiple engines on the same page, except maybe on a page running some demos to test/show library features or maybe in a multiplayer(not networked but with a shared screen) game where each player gets their own viewport. Could do all of that inside a single engine as well of course.

Also... those three simultaneous engines seem to make IE choke already while FF and Chrome handle them easily.

Other obscure use-cases could be writing a multitasking system inside a browser tab or having a network emulation layer between two engines on the same page to test message exchange in theory. Not sure if I will ever get around to implementing networking though. So far it is my understanding that the WebSockets can not just start listening for requests inside a client-side script and will always need some web-server side code to connect to.

But well... even all that could still be done inside a single engine, so I have no clue why I wanted to be able to have more than one, maybe it was just the temptation of the object oriented way of thinking.
_________________
0xDB
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Wed Aug 27, 2014 12:05 am    Post subject: Reply with quote

Seems like there is a plan to eventually support raw TCP/UDP socket access but from what I've been reading, it seems to be reserved for installable Web Applications/Browser Extensions and may never be accessible by a website directly.

http://www.w3.org/2012/sysapps/tcp-udp-sockets/#introduction
http://www.w3.org/TR/runtime/#serving-manifests
https://developer.mozilla.org/en-US/Apps/Build/Manifest
_________________
0xDB
View user's profile Send private message Visit poster's website
PoV
Moderator

Joined: 21 Aug 2005
Posts: 10904
Location: Canadia
PostPosted: Wed Aug 27, 2014 12:54 am    Post subject: Reply with quote

WebRTC fills the void of unreliable UDP/datagram communication, available in FireFox and Chrome, no known IE release (but 2 against 1 tends to do it). It's not 1:1 of UDP, but the overhead should be minimal, so it's effectively the same.

WebSockets are much the same as having normal TCP communication. Available everywhere, some browsers more recently than others.

And of course, XMLHTTPRequest wraps HTTP over TCP. All responses come with an HTTP header, so they're a bit fatter than raw TCP/WebSockets.


So AFAIC, were almost there. You'll need to thinly shim your Web 2 Native Code communication, but interoperability should be doable.

FWIW, both node.js and Chrome Store Apps (JS and NaCl/PPAPI) support TCP and UDP directly, as they have special permissions. So did Adobe Air, but they were native flash apps, with the same special permissions. The consensus seems to be that all this stuff, when executed on the web, needs to be behind an API wall. Presumably this is to make it less exploitable.
_________________
Mike Kasprzak
'eh whatever. I used to make AAA and Indie games | Ludum Dare | Blog | Tweetar
View user's profile Send private message
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Wed Aug 27, 2014 3:33 am    Post subject: Reply with quote

But WebRTC requires a server in the middle to establish direct peer-to-peer communication between clients. What I would like to be able to do is open a port directly in client-side javascript code(not without asking the user for permission though) and start listening/serving requests from other client-side javascript code, so to connect to another player, all you'd need to know is their IP just like in classic network games which did not come with centralized service networks.

Using node.js would be an option though but then at least one player would need to download it and run a server on their machine to serve as a switch for WebRTC connection requests so clients could find each other.

This would probably be most pain-free with peer.js and the also available peerjs-server which can be ran in node.js (and yay! node.js, peer.js and the peerjs-server are all usable under the MIT license).
_________________
0xDB
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Wed Aug 27, 2014 3:37 am    Post subject: Reply with quote

I probably should not even worry about networking at this early stage. There is plenty of essential library functionality to implement first which does not require any networking.
_________________
0xDB
View user's profile Send private message Visit poster's website
Alex
Developer

Joined: 05 Sep 2005
Posts: 1159

PostPosted: Wed Aug 27, 2014 6:14 am    Post subject: Reply with quote

peer.js is the whole reason why I began making my pixel editor. I always wanted an editor that could connect people together and allow them to work on the same image. I don't know how to make networked stuff, but when I found peer.js, it made it really simple to create a web app that enabled that sort of thing. And I decided to just use their free cloud server because I realized that there wouldn't be that many people using my editor.. Though, if necessary, someone could set up their own server and use my editor with it if they really wanted to.
That being said, I need to update something with my editor.
Your engine thing is looking cool- this thread talks about a lot of technical things that I don't understand though, but it's nice seeing the results you're making.
View user's profile Send private message Visit poster's website
PoV
Moderator

Joined: 21 Aug 2005
Posts: 10904
Location: Canadia
PostPosted: Wed Aug 27, 2014 8:16 am    Post subject: Reply with quote

Quote:
But WebRTC requires a server in the middle to establish direct peer-to-peer communication between clients

Well by design, with the web "someone" is serving those web-pages you're viewing in the browser. I haven't looked too closely, but Mozilla seems to suggest browsers can open connections directly.

https://developer.mozilla.org/en-US/docs/Web/Guide/API/WebRTC/Peer-to-peer_communications_with_WebRTC

That said, without some form of matchmaking, users can't find each-other. That's the way networking has always been. Doesn't matter what language, someone either tells you whom you can connect to, or as a user you're punching in IPs directly.
_________________
Mike Kasprzak
'eh whatever. I used to make AAA and Indie games | Ludum Dare | Blog | Tweetar
View user's profile Send private message
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Thu Aug 28, 2014 4:51 am    Post subject: Reply with quote

Yes, they can open connections but they will be lose ends and unfortunately hooking up to another lose end is not as easy as supplying an IP and a port as it is in TCP and UDP. So a known server to act as a switching board in between for connection setup is mandatory (unless users want to copy&paste long and ugly JSON strings back and forth a few times).

Alex wrote:
Your engine thing is looking cool- this thread talks about a lot of technical things that I don't understand though, but it's nice seeing the results you're making.
I guess the tech-talk is a consequence of developing a library first and a result from a desire to share what I learn about html and javascript as I progress. If I had started making a game with it, talk would probably shift to art, content, game mechanics and so on and be less technical.

progress:
  • improved rlEngine self adapting speedup/slowdown for cases when speed is way off(which it usually is after pausing for debugging)
  • tested and debugged initial(no zip yet) rlDataManager, switched to sequential rather than parallel loading of resources as that works much more reliably on very low bandwidths
Test 4b: data loading with data manager

_________________
0xDB
View user's profile Send private message Visit poster's website
Gil
Developer

Joined: 14 Nov 2005
Posts: 2341
Location: Belgium
PostPosted: Thu Aug 28, 2014 11:51 am    Post subject: Reply with quote

Dennis, have a look at https://github.com/rogerwang/node-webkit

It's a combination of Chromium/Node that you can bundle with your game, so you can offer people an executable. It fixes the file:// problems.

I personally use Mongoose for development (or more recently, I started using the builtin webserver in Abobe Brackets), but for distributing to users and testers, Node Webkit is king.

Also, Adobe Brackets, use it. It's the only decent program for writing Javascript.
_________________
PoV: I had to wear pants today. Fo shame!
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Thu Aug 28, 2014 2:48 pm    Post subject: Reply with quote

Hm, shared my thoughts on node-webkit earlier in this thread though that may have been lost in one of my wallsoftext.

For development I've been using SynWrite for a couple of days now and I really like it. It puts little color swatches under html color strings and when a string looks like a path to an image file and you hover with the mouse over the string, it shows a little tooltip with the image inside. :)

tiny bit of progress:
  • experimental highrestimer test in rlLogicTimer (higher precision at the cost of more CPU usage, so it is disabled for now / also in IE inside a Worker the highres timer object is not available)
  • added hacky possibility to rlEngine to share a logic timer with other engines(now used in Test 004) instead of each running their own timer (greatly improves LUPS stability for parallel engines, especially in IE)

_________________
0xDB
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Mon Sep 01, 2014 12:43 pm    Post subject: Reply with quote

Phew! What started as a simple "let me see if I can just quickly hack up something that works" thought on saturday, ended up as a three day marathon of coding... or rather debugging/fixing/testing the hacked up code: and now Refugee Lib has its own ScriptShrink which can turn any javascript file or in theory even combine a bunch of scripts into a shrunk self-expanding script.
I think it will give better results the bigger the input data is for maximum symbol/constant re-use. For very short scripts, the output is likely longer than the original because of the expansion code which gets slapped on at the end.

  • fixed "timer not stopping" when using experimental highRes timer in rlLogicTimer
  • optimized speedup/slowdown code in rlEngine again (based on a mix of percentages and absolutes now)
  • started implementing rlScriptShrink (shrinks scripts, generates shrunk self-expanding scripts)
Test 4c: testing and debugging rlScriptShrink

Now... what's next... ah yes, back to the data manager and after that, basic engine input handling(keyboard and mouse). It's sometimes a bit boring doing all this "system level" stuff but I want to build a solid base for everything else first.
_________________
0xDB


Edited by 0xDB on Thu Sep 18, 2014 3:55 am; edited 1 time
View user's profile Send private message Visit poster's website
0xDB
Developer

Joined: 26 Dec 2005
Posts: 1670
Location: Your consciousness.
PostPosted: Mon Sep 01, 2014 11:23 pm    Post subject: Reply with quote

  • fixed rlScriptShrink to replace found code tokens from longest to shortest on shrinking to prevent ambiguous substring replacement on expanding
  • ran a short test, shrinking jszip.js from 287474 chars to 82839 chars (not bad but there is probably room for improvement :) )
edit: Hm... script shrinker still has issues, although the re-expanded code is mostly valid, there are special cases where a token could put its placeholder at the start of an un-tokenized symbol. If that un-tokenized symbol contains any base64 chars following the token placeholder, those could be interpreted as the token index, leading to unpredictable fuck-ups upon token expansion. So I have to make sure, only whole words can be replaced by a token placeholder. e.g. a file containing a single instance of 0xDEADBEEF and later multiple instances of 0xDE which get tokenized and assigned a placeholder @a then the current replacement code will turn 0xDEADBEEF into @aBEEF. If not placeholder @aBEEF exists in the found tokens, this is not a problem but if there is a placeholder like that it might later get expaned as <unpredictable>BEEF where unpredictable is whatever token was stored under base64 index aBEEF. So... the script shrinker is turning into quite the project in itself. :)

edit2:
  • modified base64 code in rlScriptShrink to use chars `# instead of +/ (which are valid javascript operators and could potentially be unintendedly matched on expansion of symbols)
  • improved rlScriptShrink performance and fixed un-intentional sub-symbol tokenization by allowing only whole symbols to be replaced
Shrinker seems stable now(yah... until it barfs at some obscure script but I'll fix that when it happens) as it can successfully shrink all scripts currently in Refugee Lib and I tested shrinking of webgl-utils.js and jszip.js as well.

I even found some bugs in my previous scripts thanks to the shrinker. Some javascript oddities would allow omitting a semicolon in certain places if a newline was used instead but code like that will make the javascript engine barf when the newline is removed and the script singlelined.
(webgl-utils also contained one instance of that omitted semicolon (edit: reported that and it got fixed within a couple of hours :) ))
_________________
0xDB


Edited by 0xDB on Sun Sep 14, 2014 4:41 am; edited 1 time
View user's profile Send private message Visit poster's website
Reply to topic GDR Forum Index -> Game Developer's Refuge -> Development Log - This And That (unfocused randomness) Page Previous  1, 2, 3, 4, 5, 6, 7, 8  Next

Use this link to get a Sign-On Bonus when you get started!

All trademarks and copyrights on this page are owned by their respective owners. All comments owned by their respective posters.
phpBB code © 2001, 2005 phpBB Group. Other message board code © Kevin Reems.