nginx and ws/wss proxy

There are many recipes for nginx and ws proxy. However, for some websocket servers, they may not work! Some websocket systems, e.g., crossbar.io, check the URL and respond differently for a http:// URL than for a ws:// URL. But, at the time of writing this (Oct 2014), nginx does not allow ws/wss targets in rules.

What can be done? Well, one could patch nginx and get the patch pushed through upstream. A more immediate path is to use SNAT/DNAT to forward the traffic to its ultimate destination, that is to do the work at the network level, rather than the application server level.

SSE

SSE (Server Sent Events) have some real advantages for pub-sub type applications, but come with one cost that I have not seen discussed much. First, the advantages, no timeouts, simple to use, and with chrome, simple to debug (you can simply go to the URL and watch the stream). But, in many ways, SSE can be thought of as an abuse of HTTP; something originally intended for use in streaming video or audio is being subverted to a long running data stream. This seems like it would cause no problems, but, it is inherent in multi-media that the user will seldom want more than one or two streams open. Most web browsers have a hard limit of six to eight connections open to any FQDN. For data applications, this can be a real problem! If a user starts to open mulitple tabs to the same application, the user may run out of connections quickly. Once this limit is reached, further requests simply block, with no notice to the user. This can be partially overcome by creating extra DNS entries, say an entry per app, and using CORS, but this is an additional administrative burden. Thanks to gavinc on #pyramid for calling my attention to this issue.

Also, if you need a good python server for SSE, look at cyclone-sse.

Because of the problem of connection exhaustion, it is usually better to move to websockets. Having a websocket server will be necessary. In the Python world, one good option is crossbar.io.

I have not personally used it, but a MQTT broker such as mosquitto and a client such as paho looks like another good option. Note that, as in the case of crossbar.io, websockets are actually used behind the curtain.

Pyramid, Angular, and POST

Angularjs does not send its POST data form-encoded. Pyramid, actually webob, does not decode the submitted POST data into the conventional request.params structure. After some discussion, this is probably the correct decision. form-encoded data should have distinct text keys, and textual values. JSON encoded data does not necessarily have either.

There is a convenient option, though. request.json_body is a nicely decoded python object and is what you want to be working with on the server side.

Pyramid and Angularjs - Authentication

If you are using Pyramid and Angularjs and have resources requiring authorization, here is one way to handle the authentication process.

SPF Rant

If you can use SPF, please do so, but you can't if you have joined the chain of fools. Detailing the chain.

Invoking Slow-Running Jobs from Pyramid

First, why worry? If you have a site that has light usage and the slow job is not really slow, say under 10 seconds, and you have a patient set of users, and all the processing can occur on your Pyramid machine, then there is nothing really needed. If you are in a situation where any of these conditions are not met, then you probably need to think about how to get parts of the task out of Pyramid -- whether to save threads, to prevent timeouts, or just because the work doesn't happen there.

There are many options, but the classic method to decouple the server-side work from the Pyramid response cycle would be to use a message queue. And there are a lot of options available here. I used to recommend ZeroMQ, but now prefer redis, as it is simply easier to work with.

Celery is another popular option as a memory queue. It is much more full-featured than simply using RPUSH and BRPOP (or RPOP) in redis, but, also much more complex.

PIL why you such a pill?

One of these things is not like the other. Rescaled from 532x276 to 80x39 by ImageMagick, PIL, PIL (Image.ANTIALIAS), netPBM, and phantomjs resp. I'm a sucker. Every time I forget how awful the PIL rescale is.

Critiques: ImageMagick's command line is hard to use, but it gives good results here. PIL, with its default settings, is awful: creating "jagglies" and dropping the lower black line entirely. With Image.ANTIALIAS, it is a bit better; most of the jagglies are gone, but the buttom line is still missing, the stems are not consistent, and the lotus flowers have odd asymmetries. netpbm gives good results, in some ways a bit better than ImageMagick, but adds some near-white "sparklies"; stems of lotus leaves are better than ImageMagick, but it can only be used on a very light background. phantomjs gives very good results, although there appears to be a missing pixel at the left border; the stems of the lotus leaves are a bit better than ImageMagick, in that they are more uniform in appearance, it is easier to invoke than ImageMagick, but slower (about 300 milliseconds on my laptop) and with some heavy requirements.

For this particular application, the ultimate solution was phantomjs.

Every Pyramid is an Island

I gave a talk at ploneconf on 3 Nov 2011. It was better received than it deserved to be. Thanks for all who attended for their kindness and for the great questions! Slides are available. At the last minute, I learned that the device resolution was lower than I expected. An 800x600 pixel version is here.

A Rolodex program implemented in Pyramid

This is a Pyramid program written for my wife. It is a good example program for a certain style development. To give the buzzwords, it is an imperatively configured application, using direct SQL calls controlled by context managers, and using (trivial) traversal as the URL mapping mechanism. If that jargon is confusing, don't worry. It is intended as a quick summary for people who have already skimmed the Pyramid main documentation. You don't need it to understand the program itself. Read the details here.

Source can be downloaded as rolodex.tar.gz. A demo version is running at jpenny.im:8080/. Installation instructions have not been written yet. My apologies.

A Visitor Web Login System

Another Pyramid program, this one is like a hotel web registration form, where the user has to sign into the site before he can reach the Internet, well, anyway, before he can reach the web. This is could be interesting to anyone who needs such a system. expecially if they are using squid as a proxy server. It is also interesting because it is a useful program not much more complicated than a "Hello, World". The Pyramid portion of the application is only 64 lines, including email, comments and blank lines, with a template 51 lines long. The squid portion is 51 lines.

I just realized that this should have been called kraken. Subby and Goss are just around the corner...

Writeup is at: http://jpenny.im/weblogin_doc/

Why Pyramid

In late 2008, my boss and I were discussing Zope 2, and the fact that it had pretty much lost all momentum. I was asked to select a new web application framework. I started by listing the things I needed in a framework:

More...

A Note on Handling Form Posts in Pyramid

Some background. Pyramid allows the programmer to declare a renderer that will be invoked when a dictionary is returned from a view callable. The renderer will usually be some kind of template, and it will usually produce HTML that can be rendered in the browser.

Classically, browsers work in a strict call-response cycle. The browser makes a request and the server is responsible for making a valid response. And in web applications, the cycle looks like: browser makes a GET request. The response is a form that the user fills in. After it is filled in, the browser makes a POST request. Ideally, the server will process the form, change any stored information, and REDIRECT the browser to some page. (For more information on why the redirect is done, you can start at: http://en.wikipedia.org/wiki/Post/Redirect/Get)

But, what happens if the POST data has an error in it. Somehow the original form has to be "re-painted" and sent back to the user. Users get really cranky if the have to type in the whole thing again! What is a good way to do this in pyramid?

More...

Beginning Web Application Development

Don't get me wrong, web applications have a compelling story versus native applications. Notably, the "distribution problem" of how do you get updates to users is pretty much gone. And, while the incompatibilities between browsers can be infuriating, still, there are at most a dozen that matter.

But, the number of technologies involved can be staggering. A good web developer will have a working knowledge of HTTP; an intimate knowledge of HTML, including "HTML5"; will be able to work with CSS; will have a working knowledge of JavaScript, and probably better knowledge of some JavaScript libraries; will be able to work with persistence engines, either directly or using some sort of adapter library; will probably need to know at least one server framework well; and will need to know whatever languages the server framework uses. In addition, some domain specific languages, such as templating languages will probably need to be learned. And the developer will have to be able to fluently switch between all of these artifacts, as they will typically all be in use simultaneously.

More...