Published January 27th, 2010, updated January 28th, 2010.
Like any modern language, Python comes along with a nice traceback module. This module gives you stack traces from the line of code where an exception is raised up to the next try-except clause. So, you can easily catch exceptions and write stack traces into a debug log. This debugging technique is pretty handy to drill down bugs and I use it a lot in prototyping.
Using the traceback module is straight forward for evident programming mistakes. However, real bugs are context-sensitive and they can hardly be reproduced without the actual data that was processed when an exception was raised. If you can reproduce a specific bug, you can add some logging code in front and inspect the variables the next time the bug is triggered. But if a bug occurs once in a blue moon, you’d be better in logging the data the first time an exception raises.
import traceback
def erroneous_function():
ham = u"unicode string with umlauts äöü."
eggs = "binary string with umlauts äöü."
i = 23
if i>5:
raise Exception("it's true!")
try:
erroneous_function()
except:
print traceback.format_exc(with_vars=True)
Here’s my solution; an improved Python traceback module the logs variables from the local scope next to the affected code. You can find a working copy in our Mercirual repository (see the below).
Traceback (most recent call last):
File "test.py", line 16, in <module>
Local variables:
erroneous_function = <function erroneous_function at 0x7ff6d82b...
__builtins__ = <module '__builtin__' (built-in)>
__file__ = "test.py"
traceback = <module 'traceback' from '/srv/www/vhosts/dev.teamr...
__name__ = "__main__"
__doc__ = None
erroneous_function()
File "test.py", line 13, in erroneous_function
Local variables:
i = 23
eggs = "binary string with umlauts \xc3\xa4\xc3\xb6\xc3\xbc."
ham = u"unicode string with umlauts ???."
raise Exception("it's true!")
Exception: it's true!
I am not sure if it is the “right” solution as sensitive information might be logged. This might have security implications for some real-world scenarios where webapps report stack traces to the end user (e.g. by using cgitb in production).
Credit: this code was inspired by format_exc_plus by Bryn Keller.
2010-01-28: there’s an active discussion on python-dev.
/*
* Hello Brandley, tonight I've tried to figure out how to do proper
* JSON POSTs using "Content-type: application/json" and serialized JSON
* data in the content portion (body) of my HTTP requests. This is
* specified in the Twitter API for status updates and Identica JSON
* webservices (found via Mark Pilgrim).
* After some tests, I had to recognize that jQuery does not support JSON
* encoding in the core distribution and aside of $.getJSON(), there
* is no $.postJSON().
* Below is an proposed update. As it relies on your json plugin, I'd ask
* you to add it to your code base so that other jQuery users can benefit
* of it.
*/
$.postJSON = function(url, data, callback) {
return jQuery.ajax({
'type': 'POST',
'url': url,
'contentType': 'application/json',
'data': $.toJSON(data),
'dataType': 'json',
'success': callback
});
};
Dear Googlebot, I want to tell you that you can find my latest source code over at hg.sickos.org. That’s where my friend dlat has installed a Mercurial source code management server. I’ve already migrated some projects there and would like you to index these pages. Please note that some projects have individual pages in our Wiki, just follow this category page. See you again!
Published June 18th, 2009, updated December 8th, 2009.
I’m currently working on a pet project where I want to convert favicons (in Windows .ico format) to Web standard .png format. I started using the Python Image Library (PIL) for this, which supports plenty of image formats and which is Python’s standard way for doing image manipulations. A basic any_to_png function using PIL looks like this:
This is straight forward, but I had to find out that PIL’s .ico support is pretty outdated while Microsoft has updated the specs. Modern .ico files have switched from .bmp to .png format and added alpha masks. There is a patch available that brings you .png support, but alpha masks are still broken.
img = Image(StringIO(buf), 'ico')
if not i.select((16,16)):
i.alpha(True)
i.scale((16,16))
return i.dump('png')
You see, Ian Stevens‘ CDLL bindings are a straight forward implementation of the MagickWand C API using the CDLL wrapper library. I’ve added some missing functions, documentation and clarified the licensing issues (now available under a BSD license) and I think this is a clean and elegant solution for a long standing problem. You can find a snapshot of Python MagickWand here and the latest source code there. Enjoy.
2009-12-08: We’ve cloned python-magickwand and accept patches. You can find our latest work in our mercurial repository. Today, we’ve commited a magickwand6.5 patch, please look at the hg changelog for details.
XMLStarlet is a command line interface to the Gnome XML- and XSLT libraries (libxml2, libxslt). It supports XPath queries and other usual permutations to XML data on the command line. I’ve stumpled upon this tool some years ago and today, I wanted to use it in practice. However, it looks like development has deceased and the last (yet incomplete) release is from 2005.
So far, you can easily query XML data, but there is not way to append formatted XML elements (like parts of a tree) somewhere to the tree. Is anyone using XMLStarlet in real-world scenarios like shell scripts? Are there any better tools for DOM manipulations that are handy within shell scripts?
btw: I’ve fixed the x86_64 compile errors on RHEL5 here.
Creating backups of open files was a challenging endeavor in the past. The main problem here is inconsistency. This is because the original data could get changed while it is read by the backup software. If this happens, it can result in missing references or references pointing at incorrect data. A typical example would be any kind of database. Here we have a lot of indices that are stored aside of the raw data. When the backup software reads the index and an update occurs, the original data cannot be read anymore.
A simple strategy to deal with this type of problem is the creation of offline backups. They are fairly simple to implement and they ensure exclusive access by the backup software. Though, this approach is inelegant as it requires a downtime for every backup. To overcome this limitation, many applications spawned custom backup mechanisms to enable online backups. These mechanisms include simple dumps, logshipping, single- and multi-master replication and others. Although, they enable online backups, they are proprietary and require application specific know-how.
A more generic approach is the use of file system snapshots. They create a static copy of the original data within milliseconds. This copy can be backed up while the system is online and the original data gets changed. On Linux, this snapshot functionality is part of the Logical Volume Manager (LVM). It is included in the standard Kernel since version 2.4.x and most modern Linux distributions activate it in their default installation.
To create a file system snapshot, one basically requires the file system to be on a logical volume (LV) and some free space on the underlying volume group (VG). Then, one can tell the logical volume manager to snapshot the particular volume. The LVM holds all IOs and creates a new copy-on-write (COW) volume within milliseconds. This new snapshot volume can be mounted and backed up safely, as it does not change when it is written to the original volume.
Of course, there are good reasons to use the backup methods that are recommended by some software vendor. But there are also many situations where a generic approach is preferable. Think of small databases, virtual machines, mail servers and other applications that store custom index files. I’ve seen a lot of these situations in the wild and for many times, I desired to have an easy snapshotting functionality. This lead to various snapshotting scripts, consolidation, adoptions and so on. After all, I’ve created dsnapshot which I want to introduce here.
The dsnapshot script provides a high-level interface to the Linux Logical Volume Manager. It uses its block-level snapshot support to create directory snapshots. In contrast to block-level snapshots, directory snapshots resemble the file system layer. Thus, you can snapshot any directory that is on a logical volume and you don’t have to worry about the actual logical volumes, mount points and paths.
Published July 9th, 2008, updated September 30th, 2008.
301 is an uri redirector. It allows you to create short links for complex web addresses. Just submit a longish uri at 301.sickos.org, and you will get a short link that points to the original address. You can pass this on twitter, in irc or whereever you want to avoid complex web addresses.
This service was inspired by tinyurl.com and monkey.org/sl. In contrast to their services, 301 comes along with full python source code. This give you freedom to run your own 301 service and adopt it to your needs. Get the source code at benjamin-schweizer.de/files/301/.
Published June 13th, 2008, updated September 18th, 2008.
User authentication on unix systems typically relies upon password files or directory services. Both contain logon names, user ids, passwords, the location of your home directory and other information. The choice of the right authentication backend typically relies upon the amount of users you have to manage and your system environment.
If you have decided to use simple password files, you can create different files for various services. This gives you the opportunity to separate system users from service users. Further, this enables you to delegate administrative rights to certain people.
However, user management still requires you to twiddle with command line tools. This is fine if you are a unix lover, but if you want somebody with little command line experience to manager your users, you probably prefer a user interface that guides the unexperienced and reduces the risk of crashing the system.
This is exactly what htpasswd_editor does. It provides a text user interface for htpasswd(1) files and can easily be integrated with popular software like the Apache Web Server, VSFTP Daemon and other PAM-enabled programs (using pam_pwdfile).
update 2008-09-17: there’s a new bugfix release available
Entropy is a password generator. It generates two kinds of passwords: i) low entropy passwords that humans can easily remember and ii) high entropy passwords as commonly used in stored sessions. The low entropy passwords are generated from the Basic English vocabulary by C.K. Ogdeni. The high entropy passwords are random alpha numeric passwords where similar looking characters are stripped.
This application is writte in Python and supports both, a CGI interface for your web server and a command line interface. From a security perspective, I strongly recommend the command line version after reading the source code.
Published November 28th, 2007, updated March 7th, 2008.
The Web Watchdog notifies you when a website gets updated. Instead of continuously returning to blogs, forums or discussion boards, you can ask the Web Watchdog to do this for you.
Published October 17th, 2007, updated April 10th, 2008.
Ipmap is a GTK-based IP address grapher, inspired by an XKCD comic and glTail. It reads data from standard input and maps IP/size pairs on a grid (see the screenshot). Due to this simple interface, it is easy to create filters for a variety of data sources. The program comes along with some example filters, including tcpdump ouput, Apache/ProFTPd’s access logs and Squid logs.
Published June 6th, 2007, updated February 29th, 2008.
“The pickle module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure.” Pedit is an interactive editor for such data structures, making them handy on the command line. The code is work-in-progress but yet useful…
Published May 9th, 2007, updated February 29th, 2008.
This is a proposed patch for squid_auth_ldap, the authentication helper of the Squid Cache. It increases the maximum filter length to 32k and thus, enables long search filters. Wow:-)
# patch for squid-2.6.STABLE11, enables long search filters
# author: Benjamin Schweizer
# diff -r squid-2.6.STABLE11/helpers/basic_auth/LDAP/squid_ldap_auth.c \
squid-2.6.STABLE11-sickos/helpers/basic_auth/LDAP/squid_ldap_auth.c
32a33,34
> * 2007-05-09: Benjamin Schweizer
> * - Increased buffer size to support long search filters.
343c345
< char buf[256];
---
> char buf[32768];
566c568
< while (fgets(buf, 256, stdin) != NULL) {
---
> while (fgets(buf, sizeof(buf), stdin) != NULL) {
653c655
< char filter[256];
---
> char filter[32768];
Published April 18th, 2007, updated March 7th, 2008.
This is a wiki plugin for the Unix editor Vim. It enables you to use syntax highlighting, hyperlinking etc. within your favourite text editor. Based upon Tim Hemel’s original wiked, the code perceived many changes to fit the needs at sickos.org.
We use it here (sickos.org) for server documentation, notes, logbooks, howtos and many other tasks where we savor syntax highlighting and hyperlinking. In contrast to Tim Timewaster’s original wiked, we have found that many wiki-like syntax highlightings are useless within a text environment. So, we have limited on these features:
**bold**
//italic//
__underline__
[[link]]
Words within brackets are interpreted as hyperlinks and automagically added to the tags file. You can jump to the target using Vim’s tag feature (as in ctags/etags). This is achieved by either entering
:tag <link>
into the ex status line or by the keyboard combination <ctrl>-<+>. (This is <strg>-<alt-gr>-<9> on German keyboard layouts, see the vim manual for details on tagging).
To install this wiked release, simply copy .vim/ and .vimrc/ to your home directory. If you understand what you do, you can also integrate it with your existing configuration; you need at leat “syntax on” and “filetype plugin on” in your vimrc and the Perl scripts and .vim files at the appropriate places.
Published January 18th, 2007, updated February 29th, 2008.
This is a patch for GnuDIP’s minidip server. It switches the password storage format from plain text to md5 hashes, so you can copy passwords from GnuDIP to MiniDIP and vice versa. Furthermore, this allows you to use arbitrary characters in your passwords without breaking the format of the config files. Use GnuDIP’s encpass utility to encrypt passwords.
Published November 16th, 2006, updated November 25th, 2009.
This is a plugin for the Munin monitoring system. It graphs the cache efficiency of your Squid proxy servers and shows nice graphs for average byte and request hits.
Published April 24th, 2006, updated February 29th, 2008.
This is a caching and scaling proxy for images that are accessible via http. It is designed for the use in weblogs where you want to refer to images on foreign hosts without the need of manual mirroring or scaling. After an initial fetch, the (scaled) pictures remain in the local cache directory. It is tried to reload the image when the cache expires, otherwise the old image is delivered.
Published October 5th, 2005, updated February 29th, 2008.
Preface
wp_flickr_sidebar is a plugin that shows a moblog in a WordPress sidebar. It can be configured to display any Flickr feed with a specified width and number of pictures. The plugin supports caching and resolution-dependent fetching.
Installation
Shorts steps for the impatient:
extract the files to the plugin directory (currently one file)
drop a copy of magpierss in /magpierss (or symlink it or fix the constants)
ensure that you have libgd2 and the php bindings installed
activate the plugin somewhere on the WordPress setup page
configure the plugin using the WordPress options dialog from within the admin menu
add <?php wp_flickr_sidebar_feed() ?> to your template/sidebar
If something went wrong (missing libs, wrong permissions, typos), you should see an error message as long as the bug is in stage 1. To see error messages in the standalone jpeg section, set this line as a comment:
// (header("Content-type: image/jpeg");)
When you’ve fixed your problem, come back here end tell us, please.
q: after commenting the header line, i see an error message that says there is no imagecreatefromjpeg()
a: install/update the gd graphics library and php bindings q: it does not work anymore a: flickr changed the uri schema, update to version 0.2
Published May 11th, 2003, updated February 29th, 2008.
wconsd is a serial console server for Windows NT/2k/XP. It maps local com ports to an IP socket and makes it accessible with any TCP/IP terminal. I use it with PuTTY to set up routers when no IP address is assigned… PuTTY just feels better than Hyperterminal or TeraTerm;)
Published December 29th, 2002, updated February 29th, 2008.
Echolot is a network station monitor. It sniffs ARP packets on ethernet network devices and stores them in an IPTraf-compatible database. These unique addresses could be used to find known hosts on any LAN (a popular FTP server at a LAN party), or to find unknown hosts on a company LAN (such as intruders).
Published January 28th, 1997, updated February 29th, 2008.
My first hack:-) Back in 1996 I checked out a method to by-pass the password verification programm in the software “Klassendatei”. Dropping the sequence (0×03, 0×76, 0×83) to the dos keyboard buffer and running kd0.exe opened the doors. Here’s the code:
Dear Googlebot, I want to tell you that you can find my latest source code over at 
