currently do custom web applications
and network administration
for a manufacturer. I work mainly in python,
but have worked with at least a dozen languages.
The writings on this website are mine, and
have neither been reviewed nor approved by
the company for which I work.
I have long used Zope. As Zope is mostly a zombie,
these days, I switched to bfg, and then to pyramid.
This site is mainly about pyramid,
but, I reserve the right to ramble on
about anything that crosses my mind.
I can be reached as jpenny @ jpenny.im.
nginx and ws/wss proxy
There are many recipes for nginx and ws proxy. However, for some websocket
servers, they may not work! Some websocket systems, e.g., crossbar.io, check
the URL and respond differently for a http:// URL than for a ws:// URL.
But, at the time of writing this (Oct 2014), nginx does not allow ws/wss
targets in rules.
What can be done? Well, one could patch nginx and get the patch pushed
A more immediate path is to use SNAT/DNAT to forward the traffic to its
ultimate destination, that is to do the work at the network level, rather
than the application server level.
SSE (Server Sent Events) have some real advantages for pub-sub type
applications, but come with one cost that I have not seen discussed much.
First, the advantages, no timeouts, simple to use, and with chrome, simple
to debug (you can simply go to the URL and watch the stream). But, in
many ways, SSE can be thought of as an abuse of HTTP; something originally
intended for use in streaming video or audio is being subverted to a long
running data stream. This seems like it would cause no problems, but,
it is inherent in multi-media that the user will seldom want more than
one or two streams open. Most web browsers have a hard limit of six to
eight connections open to any FQDN. For data applications, this can be
a real problem! If a user starts to open mulitple tabs to the same application,
the user may run out of connections quickly. Once this limit is reached,
further requests simply block, with no notice to the user.
This can be partially overcome
by creating extra DNS entries, say an entry per app, and using CORS, but
this is an additional administrative burden. Thanks to gavinc on #pyramid
for calling my attention to this issue.
Also, if you need a good python server for SSE, look at cyclone-sse.
Because of the problem of connection exhaustion, it is usually better to
move to websockets. Having a websocket server will be necessary. In the Python
world, one good option is crossbar.io.
I have not personally used it,
but a MQTT broker such as mosquitto and a client such as paho looks like
another good option. Note that, as in the case of crossbar.io, websockets
are actually used behind the curtain.
Pyramid, Angular, and POST
Angularjs does not send its POST data form-encoded. Pyramid, actually
webob, does not decode the submitted POST data into the conventional
request.params structure. After some discussion, this is probably the
correct decision. form-encoded data should have distinct text keys, and
textual values. JSON encoded data does not necessarily have either.
There is a convenient option, though. request.json_body is a nicely
decoded python object and is what you want to be working with on the
Pyramid and Angularjs - Authentication
If you are using Pyramid and Angularjs and have resources requiring authorization,
here is one way to handle the authentication process.
If you can use SPF, please do so, but you can't if you have joined the chain of fools.
Detailing the chain.
Invoking Slow-Running Jobs from Pyramid
First, why worry? If you have a site that has light usage and the slow
job is not really slow, say under 10 seconds, and you have a patient set of
users, and all the processing can occur on your Pyramid machine, then
there is nothing really needed. If you are in a situation where
any of these conditions are not met, then you probably need to think about
how to get parts of the task out of Pyramid -- whether to save threads, to
prevent timeouts, or just because the work doesn't happen there.
There are many options, but the classic method to decouple the server-side
work from the Pyramid response cycle would be to use a message queue. And
there are a lot of options available here. I used to recommend ZeroMQ,
but now prefer redis, as it is simply easier to work with.
Celery is another popular option as a memory queue. It is much more
full-featured than simply using RPUSH and BRPOP (or RPOP) in redis, but,
also much more complex.
PIL why you such a pill?
One of these
things is not like the other. Rescaled from 532x276 to 80x39 by ImageMagick,
PIL, PIL (Image.ANTIALIAS),
netPBM, and phantomjs resp. I'm a sucker.
Every time I forget how awful the PIL rescale is.
command line is hard to use, but it gives good results here. PIL, with its
default settings, is awful: creating
"jagglies" and dropping the lower black line entirely.
With Image.ANTIALIAS, it is a bit better; most of the jagglies are gone,
but the buttom line is still missing, the stems are not consistent, and the
lotus flowers have odd asymmetries.
netpbm gives good results,
in some ways a bit better than ImageMagick, but adds some near-white "sparklies";
stems of lotus leaves are better than ImageMagick, but it can only be used on a
very light background. phantomjs gives very good results, although there
appears to be a missing pixel at
the left border; the stems of the lotus leaves are a bit better than ImageMagick, in
that they are more uniform in appearance, it is easier to invoke than ImageMagick,
but slower (about 300 milliseconds on my laptop) and with some heavy requirements.
For this particular application, the ultimate solution was phantomjs.
A Note on Handling Form Posts in Pyramid
Some background. Pyramid allows the programmer to declare a renderer
that will be invoked when a dictionary is returned from a view callable
The renderer will usually be some kind of template, and it will usually produce
HTML that can be rendered in the browser.
Classically, browsers work in a strict call-response cycle. The browser
makes a request and the server is responsible for making a valid response.
And in web applications, the cycle looks like: browser makes a GET request.
The response is a form that the user fills in. After it is filled in, the
browser makes a POST request. Ideally, the server will process the form,
change any stored information, and REDIRECT the browser to some page.
(For more information on why the redirect is done, you can start at:
But, what happens if the POST data has an error in it. Somehow the original
form has to be "re-painted" and sent back to the user. Users get really
cranky if the have to type in the whole thing again! What is a good way
to do this in pyramid?
Beginning Web Application Development
Don't get me wrong, web applications have a compelling story versus
native applications. Notably, the "distribution problem" of how do you
get updates to users is pretty much gone. And, while the incompatibilities
between browsers can be infuriating, still, there are at most a dozen that
But, the number of technologies involved can be staggering. A good
web developer will have a working knowledge of HTTP; an intimate knowledge
of HTML, including "HTML5"; will be able to work with CSS; will have a
directly or using some sort of adapter library; will probably need to
know at least one server framework well; and will need to know whatever
languages the server framework uses. In addition, some domain specific
languages, such as templating languages will probably need to be learned.
And the developer will have to be able to fluently switch between all of
these artifacts, as they will typically all be in use simultaneously.