OpenJosh http://openjosh.com Random thoughts from Josh Marshall (the other one) posterous.com Sun, 25 Mar 2012 20:26:00 -0700 ...In Which We Triumph Over Barcraft Streaming Woes http://openjosh.com/in-which-i-triumph-over-barcraft-streaming-wo http://openjosh.com/in-which-i-triumph-over-barcraft-streaming-wo

We just finished up a Barcraft at Pluckers here in Austin, and I thought I'd do a post on the video issues we had and the goofy hack I pulled together to make the technical side work.

Before all the text, here are some shiny pictures:

Note: All those familiar with Barcraft / Starcraft skip the next few paragraphs.

<barcraft:summary>

If you are unfamiliar with a Barcraft, it's simply a group of awesome nerds getting together to watch an eSports event. What means eSports? It's competitive video gaming with cash prizes, sponsorships, professional video gamers, etc. Yes, it's real, yes it's gaining popularity, and yes, it's geeky. As geeky as watching competitve poker, chess, or, in my opinion, some of the "traditional" sports. 

At any rate, it is possible (although rare) for a player to become so good at certain video games that they can play it full time, enabled by sponsorships and tournament winnings. See this site to see the winnings of some of the best Starcraft players, and remember this doesn't include salaries paid by the team / sponsors.

Starcraft is a complicated strategy game, something like real time Risk + chess where the pieces can shoot each other. In summary, you gather resources to build up an army to defeat an opponent's army. 

Anyway, we get a big group of people together to watch the larger tournaments.

</barcraft:summary>

Pluckers is a great place for a Barcraft -- lots of TVs, a bar, a dining area, and a patio. Everyone loves wings, and most people love a fried Twinkie. So when the Barcraft outgrew the last location, Andrew (our tireless leader and the producer of a Starcraft documentary) pulled strings and now we have room for nearly 300 people for the biggest events.

Dsc00196

However...

As I assume is common at sports bars, Pluckers does not have a well-engineered A/V solution. Approximately two thirds of the televisions are wired to use the same composite (old-school RCA / standard definition) source, but the other televisions support either HDMI or VGA, and have to be hooked up individually. Some of them are just inaccessible.

At the last Barcraft, this limitation wasn't known in time to try to fix it. So, to support all the televisions for the ~250 people who showed up, we had four or five laptops, each streaming from the MLG live site, hooked up to the various "clumps" of televisions. This meant that the streams were anywhere between 5 and 45 seconds apart, and we saturated the Plucker's network and had terrible buffering problems, usually during the best parts of the games.

The situation remained the same as this Barcraft approached, so I tried a number of different solutions. I tried wireless HDMI transmitters, but the consumer ones were garbage and renting a decent system was over $600. I looked into HDMI -> cat5e systems, but the pricing was similar and we would have had network cables and HDMI splitters all over the place.

Finally, I decided we could just "re-stream" the screen contents of a single computer streaming from the live site. The other laptops would then "re-broadcast" the internal stream, ensuring the majority of the traffic occurs within the internal network. Additionally, the televisions all show the exact same source, and are mostly synced (at least within half a second or so).

Recipe for Successful Hackery

If you find yourself in a similar situation, here are the technical bits to recreate my setup. You're going to need a few ingredients:

  • A "source" laptop / computer (for these instructions, running Ubuntu Linux, but you can do it with Mac and probably Windows as well)
  • One or more "destination" laptops hooked up to televisions, projectors, what have you.
  • VLC installed on every machine.
  • FFMPEG installed on the "source" laptop (optional, but the way I prefer)
  • All laptops connected to the same network, via wired connections.
    • We tried wireless for a while, but get enough people using smartphones and laptops within the same space and you are asking for interference, dropped frames, and artifacts.

If you are following closely, and have Ubuntu on your "source" box, run the following command to install the necessary pieces:

sudo apt-get install vlc ffmpeg pavucontrol

Once that's complete, you can drop the following script contents into a file called "screenshare.sh" (or whatever you want):

 

 

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/sh
                                                                                
PORT="5544"
FRAMERATE="20"
VIDEO_SIZE="1280x720"
VIDEO_BITRATE="2000k"
AUDIO_BITRATE="512k"
AUDIO_FORMAT="mp2"
                                                                                
ffmpeg -f alsa -ac 2 -i pulse -acodec "$AUDIO_FORMAT" -ab "$AUDIO_BITRATE" \
-f x11grab -r "$FRAMERATE" -s 1600x900 -i :0.0 -vcodec mpeg2video -g 200 \
-s "$VIDEO_SIZE" -vb "$VIDEO_BITRATE" -r "$FRAMERATE" -f mpegts - | \
cvlc - --sout "#rtp{sdp=rtsp://:$PORT/stream}" --sout-keep

You'll need to run chmod a+x screenshare.sh so you can execute it, and change any parameters to tweak the output quality. If you are video savvy, you'll notice I'm doing a high(ish) bitrate MPEG2 stream instead of an H.264 stream. The reason for this is that if you are streaming from a web site, it is 99.9% likely to be H.264, and 99.9% likely to be through Flash. (note: I just completely made up those numbers.) Watching H.264, especially through Flash, and especially at HD / high quality, is already utilizing a fair chunk of your system's resources -- I decided that throwing a fast-but-large MPEG2 stream down the internal network was just a smarter choice. Feel free to argue, and post your improvements in the comments if you are awesome.

Note that you can certainly accomplish this on Mac (and probably Windows), and you might not even need ffmpeg. I like to control all aspects of the stream, and ffmpeg is my buddy for life, so this is how I rolled.

Once you have that set up, simply run ./screenshare.sh on the "source" laptop and you should see ffmpeg printing out framerates, bitrates, etc. Now, armed with the IP address of your "source" laptop, go to each "destination" laptop, open VLC, click Media -> Open Network Stream... and enter rtsp://10.10.10.10:5544/stream in the URL field, replacing 10.10.10.10 with the IP address of your source computer.

Dsc00150

After a few seconds, you should have marvelous 720p video streaming, nearly synced, to all of your devices. Note that this isn't terribly scalable -- in the end, we had a total of four laptops streaming (and a few mobile phones, oddly enough) and the network handled it beautifully, but your results may vary depending on what you are doing. (It is not advised to download torrents during this process. :))

The End

Anyway, that's the long and short of it. We had a blast, had 100-something people show up, and once we got the kinks worked out, the stream was solid. We still need to work out some quality issues (I hate streaming 720p only to end up with half the televisions showing standard definition), and obviously it would be nice to not have to deal with any of this. However, until Pluckers upgrades its A/V system, this should work nicely for us.

Let me know if you find this useful, or if you have improvements / issues getting it to work.

Shoutout to Mothership, a new gaming-centered store in Austin, and SteelSeries for sponsoring this month's Barcraft. Also to Andrew, who does all the real work and toils without rest to ensure we all have an awesome Barcraft experience.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Sat, 18 Jun 2011 12:01:00 -0700 A Python Script for Teleporting your Mouse! http://openjosh.com/a-python-script-for-teleporting-your-mouse http://openjosh.com/a-python-script-for-teleporting-your-mouse

<TLDR>Get the script here.</TLDR>

(This is mostly for Linux users -- Mac users look elsewhere.)

So, I finally got around to playing with x2x. For those of you who are unfamiliar, it allows you to control a remote machine with a local machine's input devices (mouse, keyboard). Some of you are saying "Aha! That's what they invented VNC for!" x2x is different -- it doesn't actually display what's on the other screen, but rather treats the remote monitor / TV like another attached display.

It is great for our controlling our home HTPC when it's running Google Music, RDIO, XMBC, etc. while I'm working on my laptop. I can just scroll over to the right and I'm controlling the TV, which is miles better than a crappy remote or ridiculously slow VNC refreshes when playing 1080p video.

Anyway, after browsing a blog post that wrote a couple of bash script wrappers, I thought, "Hey, Python can do that, maybe a bit better."

So, here's my github gist for the mouseporter script that starts an x2x session. You'll need Python 2.6+, x2x on both the remote machine and local machine, and faith in code downloaded from the intarwebz. :)

All you have to do (on Ubuntu, anyway) is:

 

Here's an embedded version of the Gist for copying / pasting.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Sun, 15 May 2011 11:08:00 -0700 Simple Socket.IO Chat Room Creator http://openjosh.com/yet-another-socketio-chat-example http://openjosh.com/yet-another-socketio-chat-example

So, yeah, I guess there's something to all this Node.JS hype. :)

I've been toying around with different modules (express, zeromq, socket.io) for a side project idea, and I have to say it's been a lot of fun. CoffeeScript + Node.JS is even better. I think Python may be getting a bit jealous. (I still love you Tornado. :))

Anyway, I decided to throw the goofy chat room creator (like EVERY OTHER Socket.IO "experiment") up on a server and github. (It's not in CoffeeScript though.)

<friends>It's UBERCHAT -- It's new, it's (not) German, it's extra tough!</friends>

Really basic, supports room creation with random id room URLs, simple private messaging, away / active, etc. Also supports crashing, I'm sure. There's no persistence or anything clever yet, so if the one instance of Node goes, so do all the rooms. :) Gogo redis... later.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Sat, 07 May 2011 16:35:00 -0700 Best Poboys in Austin http://openjosh.com/best-poboys-in-austin http://openjosh.com/best-poboys-in-austin

I usually post about Python and dev stuff, but I figured I'd share my new food obsession.

At 3rd and Congress, there's a trailer called Turf n Surf that serves fantastic poboys. My personal favorite is the Kiss yo Mama, which has bacon wrapped grilled shrimp, fresh slaw, and loads of awesome.

Anyway, if you are in the area, check em out.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Wed, 04 May 2011 08:09:47 -0700 Espresso Vim Plugin http://openjosh.com/espresso-vim-plugin http://openjosh.com/espresso-vim-plugin

Totally forgot to post this a few days ago when I put it up on Github. I wrote a quick little CoffeeScript –> minified Javascript compiler plugin for vim the other day, out of total laziness. It uses the CoffeeScript syntax library from the original CoffeeScript vim project. I call it Espresso because I think I’m clever.

You’ll need to have CoffeeScript and JSMin available in the path, and I’ve only tested it in Linux, so let me know if it explodes on you.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Sun, 01 May 2011 12:14:00 -0700 Introduction to Tornado http://openjosh.com/introduction-to-tornado http://openjosh.com/introduction-to-tornado

So I did an introduction to Tornado talk at the recent Austin Web Python Users' Group (or AWPUG as it is better known). It was a lot more fun for me than I bet it was for those who had to hear me yammer on. Thankfully, Travis Swicegood followed with an excellent talk on Class-based views in Django 1.3.

Someone had a Flip video camera going, so the intarweb may soon be blasted with my incomprehensible jabbering.

Anyway, here are the slides from my talk:

Also, I made a fake site for people to see how one might construct an async API-dependent website in Tornado pretty quickly. It depends on Torn, a project a few of us are working on which shamelessly apes a subset of django-admin.py, just for Tornado sites.

As an aside, someone asked me a question about why you would pick Tornado over gevent, when gevent will let you use any WSGI framework on top of magically asynchronous sockets. I mumbled something about "explicit versus implicit" and the Zen of Python, but in retrospect I totally bombed that question.

While the explicit nature of Tornado IS awesome, as is the performance of epoll, etc., Tornado is genius in the fact that it somehow manages to provide very low-level, simple access to all parts of the HTTP request-response cycle and yet simultaneously includes most helpers you might need for building a slick web application. By overwriting a few methods, I can create a custom authentication system in a few lines, using whatever data backend I choose.

You can do the same with Django, but the difference is that Django's shortcuts tend to make you follow Django conventions. Tornado provides the tools to easily create your own. Tie that in with slick async single-sign-on libraries for Twitter, Facebook, Google, OAuth, etc. and that's just one simple example of one of Tornado's many benefits. Personally, I just tend to work faster in Tornado, with more flexibility.

Anyway, that's what I should have said. :)

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Sun, 01 May 2011 11:15:46 -0700 Laziness FTW http://openjosh.com/laziness-ftw http://openjosh.com/laziness-ftw So, I'm horrible at maintaining a blog. I'm going to be oblivious to my own weaknesses and assume it's because managing my own Wordpress installation is just too time consuming (or some other equally inaccurate reason I'm too lazy to post), and I've ported all my stuff to Posterous now. I can just email a post now, so surely I'll be more proactive, right?

...yeah...

Anyway, my massive library of something like ten posts have been imported here, and all the code formatting, etc. has disappeared. Let's see how long I go before forgetting to update for another year and a half...

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Tue, 10 Nov 2009 06:51:00 -0800 URL Checker http://openjosh.com/url-checker http://openjosh.com/url-checker

I wrote a quick Python script designed to search a file / remote address for URLs and return the HTTP status codes for each one. It's quick and dirty, and the regex needs some tweaking, but for the most part it works. The reason I didn't just use a link checker is that I was actually testing RSS feeds, so this was designed to grab URLs throughout the feed as opposed to just A tags. It lists anything in the 40x range. Here's the source code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
from httplib import HTTP, HTTPConnection
from urlparse import urlparse

def get_page(url):
    parsed = urlparse(url)
    conn = HTTPConnection('%s' % parsed[1])
    conn.request(&quot;GET&quot;, parsed[2])
    response = conn.getresponse()
    data = response.read()
    return data

def get_urls(text):
    import re
    matches = re.findall(r'http://[^s&lt;&gt;&quot;']+', text)
    return list(set(matches))

def check_url(url):
    url = url.strip()
    parsed = urlparse(url)
    request = HTTP(parsed[1])
    request.putrequest('HEAD', parsed[2])
    request.endheaders()
    reply = request.getreply()
    return reply[0]

if __name__ == '__main__':
    import sys
    import os
    source = sys.argv[1]
    data = ''
    if os.access(os.path.abspath(source), os.R_OK):
        print 'GETTING LOCAL FILE.'
        data = open(os.path.abspath(source), 'r').read()
    else:
        print 'GETTING REMOTE FILE.'
        data = get_page(source)
    print 'SEARCHING FOR URLS.'
    urls = get_urls(data)
    codes = {}
    print 'CHECKING %s URLS...' % len(urls)
    for url in urls:
        code = '%s' % check_url(url)
        if code not in codes.keys():
            codes[code] = []
        codes[code].append(url)
    print 'RESULTS:'
    print '========'
    for code, paths in codes.iteritems():
        if 399 &lt; int(code) &lt; 500:
            print 'There were %s %ss.' % (len(paths), code)
            for path in paths:
                print '* %s' % path
        else:
            print 'There were %s %ss.' % (len(paths), code)

...and here's an example of usage:

$ python checkurls.py http://www.google.com
GETTING REMOTE FILE.
SEARCHING FOR URLS.
CHECKING 21 URLS...
RESULTS:
========
There were 14 200s.
There were 2 302s.
There were 1 404s.
* http://www.google.com/ig%3Fhl%3Den%26source%3Diglk&amp;amp;amp;amp;usg=AFQjCNFA18XPfgb7dKnXfKz7x7g1GDH1tg
There were 1 405s.
* http://www.google.com/reader/view/?hl=en&amp;amp;amp;amp;tab=wy There were 3 301s.

You can also pass in a local file as the first parameter:

$ python checkurls.py file.htm

If you have any thoughts, improvements, etc. just post them in the comments and I'll update the script. I may make a "recursive" one eventually, so that it actually could function as a link checker, but I don't feel like adding that right now. :)

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Tue, 03 Nov 2009 02:53:14 -0800 Updated jsonrpclib http://openjosh.com/updated-jsonrpclib http://openjosh.com/updated-jsonrpclib I've added automatic class translation, a simple standalone JSON-RPC server, and a configuration object to the jsonrpclib. You can check it out at the project page, where I've also posted a few new usage examples. I'll be updating the blog with more specific examples and things to keep in mind soon.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Mon, 19 Oct 2009 03:47:00 -0700 Tornado RPC Library on Google Code http://openjosh.com/tornado-rpc-library-on-google-code http://openjosh.com/tornado-rpc-library-on-google-code

I finished my JSON-RPC and XML-RPC handlers for Tornado, and so I've thrown them up on Google Code. Now people can test my JSON-RPC v2.0 implementation on my jsonrpclib library. :) Which is just what everyone was dying to do, I'm sure. Usage is pretty simple, it's designed to be a.) pretty transparent and b.) easily ported between XML-RPC and JSON-RPC to compare speed, network transfer, etc. times. For example, here's the complete code for XML-RPC client (after installing Tornado and the TornadoRPC library):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from tornadorpc.xml import XMLRPCHandler
from tornadorpc import private, start_server

class Handler(XMLRPCHandler):
    def add(self, x, y):
        return x+y

    def ping(self, obj):
        return obj

    @private
    def private(self):
        #should not get called
        return False

start_server(Handler, port=8080) [/code]

To use JSON-RPC, you'd just change any instance of xml/XML to json/JSON. There are several benefits to using JSON, such as smaller transfer size, keyword arguments, notifications, etc., but to use it you are required to install my jsonrpclib implementation.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Sun, 18 Oct 2009 07:09:00 -0700 Added my JSON-RPC library to Google Code http://openjosh.com/added-my-json-rpc-library-to-google-code http://openjosh.com/added-my-json-rpc-library-to-google-code

Recently, I've been hacking together a JSON-RPC implementation for Python. It follows the xmlrpclib syntax (because it's pretty darn simple to use and extend), but also adds some of the new, slick features of the JSON-RPC v2 (proposed) specification like:

  • Batch submissions
  • Keyword arguments
  • Notifications (both via batch and single)

So, unsurprisingly, I called it jsonrpclib. I didn't see anything called that yet, so I hope I'm not hopping on someone else's name. But Google Code didn't have it, and if it's not on Google, then it doesn't really exist, right? :)

Project Page for jsonrpclib - http://code.google.com/p/jsonrpclib/

In some simple (read: probably inaccurate) speed tests, my library usually came out on top, about 1.1x faster than the "official" JSON-RPC implementation, and about 1.5-1.7x faster than the xmlrpclib library. I've also created a JSON-RPC handler for Tornado (what I used to test it), and I'll be uploading it plus the XML-RPC Tornado handler to Google Code in the next few days, after I put some shiny paper on it. I didn't see any other 2.0 server implementations yet (maybe because it's just a proposed specification :)) so I figure maybe it will be worthwhile until I can add a standalone JSON-RPC server to the library a la SimpleXMLRPCServer.

Anyway, so usage should be really familiar (this is in a python console):

1
2
3
4
5
6
7
8
9
10
11
>>> import jsonrpclib
>>> server = jsonrpclib.Server('http://localhost:8181')
>>> server.add(5, 6)
11
>>> server._notify.add(5, 6)
>>> batch = jsonrpclib.MultiCall(server)
>>> batch.add(10, 17)
>>> batch.ping({'key':'value'})
>>> batch._notify.add(50, 40)
>>> batch()
[27, {'key': 'value'}]

I do have to say inserting a notification into a batch seems a little silly to me since you have to do intelligent parsing afterwards to dismiss the missing notification, but people smarter than I are working on the spec, so I'll dig for now. If you play with it, please post in the comments about your experience! Also, add an Issue to the project page if it a.) acts oddly b.) just doesn't work for you c.) replaces your hard drive with a toaster oven.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Thu, 15 Oct 2009 07:26:00 -0700 It's a Twister - Now with More XML http://openjosh.com/its-a-twister-now-with-more-xml http://openjosh.com/its-a-twister-now-with-more-xml

Holy community code, Batman. Facebook has been launching open source projects like crazy in the past few months. (This would be a Good Thing -- I think they launched two today just while I refreshed the page!) Things like Cassandra, Thrift, and Hive are great additions to the community, and hopefully the rewards will go both ways. One of the projects I've been playing with is Tornado, FriendFeed's Python web framework. It's quite simple to dive into if you are familiar with any of the dozens of web frameworks available for Python (like web.py, Pylons, CherryPy, etc.), and it's very fast. (But, in the immortal words of LeVar Burton, "don't take my word for it.")

So, of course, the first thing I said is that we need to hack on some additional libraries and functionality to slow that sucker down. :) Actually, I use XML-RPC exhaustively in my projects, and it was only fitting that I decided to try tacking on simple (read: not the full specification) XML-RPC to Tornado, just to get my feet wet. Turns out, it's really simple. Unlike some frameworks, you always have access to the entire request body, so you can parse it to your heart's content. (Just the thing for a lonely Saturday night, eh?)

Below is my "quick" implementation, free for perusal / critique / a gripping read. Make sure you have the Tornado library installed -- it's really quite simple (unlike some Facebook open source projects), just download the tar and run (Mac / Linux):

tar xvzf tornado-0.2.tar.gz cd tornado-0.2
sudo python setup.py install

Other than that, my code shouldn't require any libraries that won't come with a default Python install. (Tornado's site does recommend simplejson and PyCurl for full functionality.) Keep in mind, this is a script you found on the internet. It shouldn't delete your files or leave your oven on, but let's just say it's a bad choice for a production environment. :)

Also, this has been rigorously tested -- ran it once on Linux -- so enjoy those bugs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
import tornado.httpserver
import tornado.ioloop
import tornado.web

def private(func):
    # Decorator to make a method, well, private.
    class PrivateMethod(object):
        def __init__(self):
            self.private = True
        __call__ = func
    return PrivateMethod()

class XMLRPCHandler(tornado.web.RequestHandler):
    """
Subclass this to add methods -- you can treat them
just like normal methods, this handles the XML formatting.
"""

    def post(self):
        """
Later we'll make this compatible with "dot" calls like:
server.namespace.method()
If you implement this, make sure you do something proper
with the Exceptions, i.e. follow the XMLRPC spec.
"""
        import xmlrpclib
        try:
            params, method_name = xmlrpclib.loads(self.request.body)
        except:
            # Bad request formatting, bad.
            raise Exception('Deal with how you want.')
        if method_name in dir(tornado.web.RequestHandler):
            # Pre-existing, not an implemented attribute
            raise AttributeError('%s is not implemented.' % method_name)
        try:
            method = getattr(self, method_name)
        except:
            # Attribute doesn't exist
            raise AttributeError('%s is not a valid method.' % method_name)
        if not callable(method):
            # Not callable, so not a method
            raise Exception('Attribute %s is not a method.' % method_name)
        if method_name.startswith('_') or
                ('private' in dir(method) and method.private is True):
            # No, no. That's private.
            raise Exception('Private function %s called.' % method_name)
        response = method(*params)
        response_xml = xmlrpclib.dumps((response,), methodresponse=True)
        self.set_header("Content-Type", "text/xml")
        self.write(response_xml)

class TestXMLRPC(XMLRPCHandler):
    def add(self, x, y):
        return x+y

    def ping(self, x):
        return x

    def _private(self):
        # Shouldn't be called
        return False

    @private
    def private(self):
        # Also shouldn't be called
        return False

if __name__ == "__main__":

    application = tornado.web.Application([
        (r"/", TestXMLRPC),
        (r"/RPC2", TestXMLRPC)
    ])

    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8080)
    tornado.ioloop.IOLoop.instance().start()

To test it, just save the contents as a file (like hello.py), start it up with python, and test it with the following in a Python console:

>>> import xmlrpclib
>>> hello = xmlrpclib.Server('http://localhost:8080/')
>>> hello.add(5, 6)

You should get 11, unless you are passing in exceptionally large values of 5. :)

I'll be updating this script as I go along, making it more robust, more to spec, etc. Random Exceptions, believe it or not, are not the way to fail an XML-RPC connection.

Feel free to blast at me with the utter failings of this script and the ineptitude of even starting on this path (that's why they built Thrift, right?), or, you know, just comment friendly-like.

Update: All these source code offereings might have something to do with that ridiculous patent infringement lawsuit announcement where Facebook was ordered to "show their code" because some dude(s) managed to fool the U.S. Patent Office into giving him a patent that covered, you know, sharing information through the Interweb thingy. This would be a perfect time to rant against the insanity of software patents, but I'll save that for another day. (Likely tomorrow.)

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Wed, 14 Oct 2009 08:15:48 -0700 Quickie: Updating Site and Ubuntu http://openjosh.com/quickie-updating-site-and-ubuntu http://openjosh.com/quickie-updating-site-and-ubuntu So, I've started creating a new theme for this blog. It's pretty simple, and knowing me, I'll change it up a lot as time goes by, but it works for now. Forgive some ugly / broken stuff as I sift through all the CSS / PHP and make the necessary changes. I really do like how simple it is to create / augment Wordpress themes. The system is very intuitive, and you can scale from a small blog to a decent-sized multi-user site easily. Plus, the plugin system is quite developer-friendly. You can't create the level of customization you can achieve with a real web framework, but it's fun for baby sites. :) Anyway, the other part of this evening was spent grabbing the new Ubuntu Desktop Beta (9.10). Originally, I could not get any of the Live CDs to work, and I was about to post a mean bug report when I noticed that I was getting strange RAM reporting on boot up. Lo and behold, after running a memory diagnostic, it turns out one of my sticks went bad. Darn you, cheap power supply! Pulled it out, and within twenty minutes I was rocking a new, 64-bit beta distro. There is a Koala in my computer now. Anyway, if you want to run 64bit Linux, and you happen to frequent tiny sites like YouTube, Vimeo, ON Networks, and Hulu, then you probably will want the Flash plugin. You can wrap the libraries in ndiswrapper, sure to provide hours of fun, or you can just grab Adobe's 64-bit beta Flash 10 player from here. After that, just copy it into your Firefox directory: [sourcecode language="sh"] tar xvzf libflashplayer-10.0.32.18.linux-x86_64.so.tar.gz sudo cp -r libflashplayer.so /usr/lib/mozilla/plugins/ [/sourcecode] 64-bit Ubuntu is really quick, by the way. I have enjoyed just pummeling the system (made better by the fact that I already reduced the amount of RAM!) and watching it yawn at me.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Tue, 13 Oct 2009 08:06:03 -0700 Sacrilege and Zeroconf http://openjosh.com/sacrilege-and-zeroconf http://openjosh.com/sacrilege-and-zeroconf (Yes, I will get back to CouchDB soon. It's my blog, I can post what I want. :)) Recently, I've been toying around with Zeroconf, the "look ma, no hands" network service discovery better known as Bonjour on Mac and Avahi on Linux. It actually does a lot of different things, from "headless" network address negotiation to host name resolution, all built on top of (mostly) the same stuff that drives DNS across the web. It's really cool stuff, and also really late for me to be getting into the game. Anyway, the stuff I am most interested in is the "service discovery", or the way two iTunes clients see each other without anyone entering in IP addresses. This is solid user-friendly tech, and this style of thinking should be implemented in more software. So, when I found pyzeroconf, a pure python implementation of the service discovery stack, I was stoked. Until I tried running it, and all of the tests failed no matter what system I ran it on. I was no longer stoked. Thus I began reading, and this is always a dangerous thing. Below is just some fun I've been having. It is a horrific test implementation of multicast UDP service discovery. About the only thing it shares in common with Bonjour is that it uses the same multicast address. I understand that many educated network programmers will see this and run screaming -- don't worry, this is just to get a feel for the way it all works. Actually, any critiques / advice / links would be most welcome, as anyone who comes stumbling along here via the alleys of Google will also learn from it. This should run in any default install of Python. There are some caveats, as this is just something I've thrown together. First: it doesn't actually do anything with the service, it just prints out a list of every compatible service it finds (including itself). Second: I only tried it once in Windows, but while my Ubuntu box and Macbook saw the Windows service, Windows wouldn't see any, even itself, so I have some hacking / learning to do if I want to support the devil. Finally -- and this is where it gets really funny -- I have had a few times where if I leave them running for too long I can't hit the outside internet. At all. So, yeah, not a perfect implementation at all yet, but I'm moving along. :) Eventually, I'll tack on a graphical user interface and, you know, conform to some actual specification as opposed to throwing strings around. Without further ado, my Frankenstein: [sourcecode language="py"] import socket from threading import Thread, Lock import time import sys # Multicast Globals MC_IP = '224.0.0.251' # Oh no he didn't MC_PORT = 5354 # Oh yes he did def message(text): # Just an stdout.write wrapper sys.stdout.write(text) sys.stdout.flush() class Broadcast(Thread): def __init__(self, client): Thread.__init__(self) self.die = False self.client = client self.lock = Lock() def run(self): self.emit('UP') self.last_emit = time.time() while True: if self.die: self.emit('DOWN') break # So the emit thread stays alive time.sleep(3) def emit(self, status='ALIVE'): self.lock.acquire() if status not in ('UP', 'DOWN', 'ALIVE'): raise ValueError('Status "%s" not valid.' % status) message = '_woot._tcp.local.|%s|%s|%s' % (self.client.port, self.client.hostname, status) sock = socket.socket( socket.AF_INET, socket.SOCK_DGRAM ) sock.sendto(message, (MC_IP, MC_PORT)) sock.close() self.lock.release() class Browser(Thread): def __init__(self, client): Thread.__init__(self) self.client = client self.die = False def run(self): while True: if self.die: break sock = socket.socket( socket.AF_INET, socket.SOCK_DGRAM ) sock.bind((MC_IP, MC_PORT)) data, addr = sock.recvfrom( 1024 ) sock.close() self.parse(data, addr) def parse(self, data, addr): """ DATA STRUCTURE: _woot._tcp.local.|port|hostname|STATUS STATUS can be UP, DOWN, or ALIVE """ if not data.startswith('_woot._tcp.local.'): return data_list = data.split('|') if len(data_list) 1: port = int(sys.argv[1]) client = Client(port) client.run() [/sourcecode] Save it as something (I like client.py personally) and run it just by: [sourcecode language="sh"] python client.py [/sourcecode] Optionally, you can add a port to the end of it, and that is the port the actual client will run on. The system should be smart enough to tell the other systems about it, but if it doesn't work, hey, it's a pre-alpha script file from the internet -- give me a break. :) You should get something like the following if it is by itself: [sourcecode language="sh"] Starting up: josh-laptop-ubuntu:1337 Searching for services. 1 SERVICE(S(): * josh-laptop-ubuntu - 192.168.0.120:1337 | UP [/sourcecode] ...with the SERVICE(S) list printing out to the terminal every few seconds. DO NOT run two of these on the same computer, I haven't done it and I'm not at all certain what would happen. If you want to test multiple ones, use different computers or at least different virtual machines with local addresses. Feel free to shout at me in the comments!

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Fri, 09 Oct 2009 20:05:00 -0700 Quickie: Using Vim for Actionscript http://openjosh.com/quickie-using-vim-for-actionscript http://openjosh.com/quickie-using-vim-for-actionscript

Okay, we all know vim is the best IDE out there. But what is a fella to do when he wants to use the best IDE with the best closed-source vector-based browser plugin scripting language? Why, hack at his vim install! :) First, download the actionscript.vim file from this place. The actionscript.vim file is a syntax file, so it should go in the appropriate folder as you see fit. However, note that in OS X the .vim folder seems to have no impact, so you have to put it in the /usr/share/vim/vim72/syntax folder. (Use sudo, and you may need to use a different vim folder if it has a newer version.) Next, we have to hack at the filetype.vim file in the /usr/share/vim* folder in order to disable the default treatment of *.as files. Change the lines that say: [sourcecode language="sh"] " Atlas au BufNewFile,BufRead *.atl,*as setf atlas [/sourcecode] ...into the following series of lines: [sourcecode language="sh"] " Atlas au BufNewFile,BufRead *.atl setf atlas " ActionScript au BufNewFile,BufRead *.as setf actionscript [/sourcecode] Voila! Opening .as files in vim should work now.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Fri, 09 Oct 2009 07:26:42 -0700 Compiling CouchDB http://openjosh.com/compiling-couchdb http://openjosh.com/compiling-couchdb So, in the last post I gave an introduction to the concepts behind CouchDB and other document store systems. For this post, I had intended to overview compiling on both Mac OS X and Ubuntu 9.10 Beta. However, on the Mac front, it turned out there was already project with a nicely cooked binary that would be significantly easier to install, and, let's face it, just more Steve Jobs than compiling from source. If you are running Mac OS X Leopard (10.5), you can download the CouchDBX binary here. The application should be dropped into your /Applications folder, a fairly standard procedure. Once you run it, you'll be greeted with the CouchDB web interface, which is actually embedded into the CouchDBX program. Not necessarily the way you'd want to run it in production, but you probably aren't serving your apps from a standard Leopard install anyway. It should look like the following: [caption id="attachment_12" align="alignnone" width="586" caption="CouchDBX Mac OS X Application"]
Media_httpblogjoshmar_hhxli
[/caption] So let's talk Ubuntu. I am certain you are more likely to serve CouchDB production apps from a Linux server, instead of a graphical interface, so we'll go through compiling a fresh install on Ubuntu Server 9.10 Beta. (I'll update this page once it is released if there are any significant alterations.) Obviously, this is Ubuntu, so you could install from the repositories if you wished. However, the last few iterations of Ubuntu 9.10 have had some issues with CouchDB -- if you are going to install from the repos, you must "tweak" a few things afterwards. Keep in mind -- these are steps I am taking to try and make it work, it is probably not the "recommended" method by the Ubuntu "Masters of the Universe", and so hopefully it will be patched soon. [sourcecode language='sh'] sudo apt-get update sudo apt-get install couchdb # The last command will end up failing. sudo mkdir /var/log/couchdb sudo apt-get install xulrunner # You will get a new, different couchdb failure sudo ln -s /usr/bin/xulrunner /usr/bin/xulrunner-1.9.1 [/sourcecode] ...and even after all that, couchdb doesn't seem to start up due to a permissions / port issue that I'm still working out. So, that's the "easy" path if you want pre-built binaries. :) Will most likely be working by / shortly after release, I'm sure. So, to compile. My commands will be run in a fresh install as a virtual image in VirtualBox. This process should work for any version of Ubuntu 9.10 - (Desktop or Server, 32-bit or 64-bit), but just so you know this is on the Server / 64-bit image. By the way, one of the things I love about Ubuntu 9.10 -- this screen on the installation process: [caption id="attachment_11" align="alignnone" width="612" caption="...one-click cloud install? Sweet."]
Media_httpblogjoshmar_jqgej
[/caption] Anyway, once you have logged in via SSH or whatever, run the following commands, tweaking as necessary: (A huge thank you to till's blog post, from which these commands were either tweaked or stolen, you pick.)
  • sudo apt-get update
  • sudo apt-get install build-essential autoconf automake checkinstall libtool help2man
  • (That gets all of the build tools.)
  • sudo apt-get install erlang libicu-dev libmozjs-dev libcurl4-openssl-dev
  • (That gets all the dependencies.)
  • mkdir Source
  • cd Source
  • wget http://mirror.cloudera.com/apache/couchdb/0.9.1/apache-couchdb-0.9.1.tar.gz
  • (Or another mirror, you can go to couchdb.apache.org to find a download link.)
  • tar xvzf apache-couchdb-0.9.1.tar.gz
  • cd apache-couchdb-0.9.1/
  • ./configure
  • make
  • sudo checkinstall -D --pkgname=couchdb --pkgversion=0.9.1 --maintainer=email@address.org --pkglicense=Apache
  • sudo ln -s /usr/local/etc/init.d/couchdb /etc/init.d/couchdb
  • sudo update-rc.d couchdb defaults
  • sudo ln -s /usr/local/etc/logrotate.d/couchdb /etc/logrotate.d/couchdb
  • sudo vim /usr/local/etc/default/couchdb
  • (Change COUCHDB_USER=couchdb to COUCHDB_USER=root)
  • OPTIONAL:
  • sudo vim /usr/local/etc/couchdb/local.ini
  • ("Uncomment" the ;address = 127.0.0.1 line and change it to address = 0.0.0.0 to enable access from any ip / hostname. You can change the port too, if you'd like.)
...and you should be done! The server should start up by itself whenever the system starts, or you would normally use the sudo /etc/init.d/couchdb start (or stop, or restart) commands, but for this first time you really should just type sudo couchdb , because the following message just sums up the entire application:
  • Apache CouchDB 0.9.1 (LogLevel=info) is starting.
  • Apache CouchDB has started. Time to relax.
In a web browser, navigate to http://HOST_IP:5984/ (HOST_IP will be 127.0.0.1 if you are on the same system, otherwise you'll need to determine your internal network). You should see something like the following:
  • {"couchdb":"Welcome","version":"0.9.1"}
Note that this message is in JSON, and that it is working. :) Next time, we'll dig into the meat of CouchDB and talk about how to start storing and retrieving your documents!

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall
Thu, 08 Oct 2009 07:20:16 -0700 CouchDB and Me http://openjosh.com/couchdb-and-me http://openjosh.com/couchdb-and-me So, as per the advice of my friend Yuce, I have begun blogging about various types of software, technology, and whatever else I see fit to project onto the masses. I decided to start with a recent "crush", CouchDB. Relational databases are simple. Anyone with experience using spreadsheet software understands the basic functionality - a series of rows conforming to a specific, pre-defined series of columns and data types. A lot of data can be easily organized in this manner. Most data can be roughly shoved into it. However, ubiquitous does not necessarily mean appropriate. CouchDB is just one example of a document store, which, as opposed to a relational database, means you can insert an object with almost any structure, with wildly disparate data types. These are inserted into a single "database" (which should be renamed to something more akin to 'documents with some sort of unified meaning') and can be retrieved in different ways. You can grab all of the documents, or group documents by certain attributes, or extract data from various objects and correlate it into entirely new data structures on the fly. This is accomplished with the Map / Reduce ideology, where (in brief) you Map (or group) document data by a key or attribute, and then optionally Reduce (or correlate / extract / etc.) by some alternate algorithm. (For more information on Map / Reduce, see here and here.) This is wildly different than simple data abstraction and union done with traditional relational databases, and provides an almost unwieldy amount of freedom. CouchDB is interesting in the confluence of the following factors: access interface, storage notation, retrieval language, and replication process. It uses a simple RESTful interface, meaning that data can be extracted with plain URLs, and data can be inserted with simple POST calls. The data itself is created, and stored, in JSON, something with which most every web developer worth his mouse has interacted. As such, there is almost no need for a data-to-object mapper --many languages (PHP, Python, obviously Javascript) include this feature either by default or with simple additional libraries. The Map / Reduce logic is written in Javascript and is stored with the document database. Finally, replication is a simple URL call, with a source (local or remote) and destination (local or remote), which performs at a fairly rapid rate. These factors combine to create a system that is a.) accessible to a wide range of web developers, b.) trivially implemented in a variety of languages, and c.) easily replicated / transported / backed-up. Any web developer should be able to hack a JSON object and write the simpler Map / Reduce logic, and subsequently open storage possibilities from an overly-fragmented table design into a simple, extensible document paradigm. In future blog posts, I'll write about compiling CouchDB on different systems, the potential pitfalls this new approach creates, and my own personal adventure with the system. Already I'm rediscovering that age-old technology problem -- when you really like your new, shiny hammer, every problem starts to look like a nail.

Permalink | Leave a comment  »

]]>
http://files.posterous.com/user_profile_pics/1196921/josh_painted.jpg http://posterous.com/users/herZJT15WRyH0 Josh Marshall joshmarshall Josh Marshall