BitPim

An application in Python

Roger Binns

rogerb@rogerbinns.com

bitpim.org

Motivation

New cell phone, new features

Phonebook

Calendar

Wallpapers

Ringtones

Messaging

Voice and Text notes ...

bitpim.org

I am human!

I will not use a number pad

Sync products limited

'Tickbox' phone support

Phonebook only

Windows only

Single machine licenses

Stupid quirks

I can do way better J

bitpim.org

bitpim.org

Scale

30,000 lines of Python

2,500 lines of description files

turns into 41,000 lines of generated code

500 lines of C/C++

20,000 words of online help

bitpim.org

Specifications

Expose and use all features of my cell phone

No protocol documentation at all!

Let me use it on as many machines as I want

cross platform

Interoperate with other data sources

No data islands

bitpim.org

More specs

Let others join the party

Open source

Able to add other phones

Do not be a PIM

Only have UI where no other program does

Be possible to plug into other programs

deal only with cell phone interfacing

bitpim.org

Even more specs

Easy to use

No installation hurdles

No prerequisites

No DLL hell

No RPM dependencies hell

Easy to diagnose what has happened on user machines

Lack of documentation means learning from the field

bitpim.org

The users don't care

Users don't care what language you use

Users don't care how hard it is to write

Users don't care what development methodology you use

Users don't care about you being consistent with a platform they don't use

bitpim.org

The users do care

Users do care that your program does what you claim

.. and they want to use it for as little time as possible

Goals, not tasks

bitpim.org

I care

I care about productivity

No drudge work thanks!

I care about refactoring

Knowledge will change over time

I care about ease of talking to other libraries & components

Productive!

Active community

bitpim.org

Preaching to the choir

The only solution is

(And some Python libraries)

bitpim.org

bitpim.org

Python!!!

Gui options

Tkinter û

Lowest common denominator

Draws widgets itself

Unpleasant prior experience

Stagnant

Qt û

High common functionality

Not open source on Windows

Draws widgets itself

bitpim.org

Gui options

GTK û

Linuxy

Draws widgets itself

wxWidgets/wxPython ü

High common functionality

Uses native widgets where possible

Very active community

Pleasant prior experience

bitpim.org

Distribution

Has to appear as a 'normal' application on each platform

Two steps

Freezing Python code (confusingly called installers)

Installer

bitpim.org

Freezing

Gather all Python modules, binary modules and Python interpretter

And data files you specify

Launcher stub

Loads Python interpretter

Set path for binary modules

Load Python code from archive (.zip)

Can be placed anywhere – no need for installation

bitpim.org

Freezing - Platform specific

Windows – py2exe

Icons

version_info resource

COM integration/modules

Linux – cx-Freeze

Strip RPATH from wxPython shared libs

Mac – BundleBuilder

Some manual specification of shared libraries to include

bitpim.org

Installation – Platform specific

Windows – InnoSetup

Start Menu

Uninstall

Control Panel

Mac – dmg

Icon

bitpim.org

Installation – Platform specific

Linux – RPM

provides, requires (be careful of rpmbuild being too helpful!)

Files out of the way, launcher shell script

Menu icon

$LD_LIBRARY_PATH (cx-Freeze can do this for you)

bitpim.org

Not that hard!

bitpim.org

Serial Ports

pyserial library

Windows (win32all)

Linux & Mac (posix)

Interface doesn't know/care about platform

Full functioned

Data rates

Flow control

DSR/CTS etc

bitpim.org

User friendly serial ports

bitpim.org

User friendly serial ports

bitpim.org

Months of fun!

Windows – scan registry

Different on Win9x, Win2k, WinXP

Good detail (drivers, dates etc)

Linux

Device nodes in various directories

Kernel module associated with major

Mac

Device nodes in one directory /dev

No other information

bitpim.org

USB

Aka Python/C integration is easy

Libusb provides good USB access on Windows, Linux and Mac

SWIG generates Python code wrapping C/C++

Language neutral but best at Python

bitpim.org

SWIG

SWIG is really good

Does a good job with raw C/C++ header

Is actually simpler than it seems

Can pick up cues from parameter names (encourage C library implementors!)

Binary strings

bitpim.org

C/C++ problems highlighted

Who owns memory?

Global pointers

Data owned by other data

May need C++ reference counter

Poor API design

Provide Pythonic layer

bitpim.org

Multi-tasking

Can't do two things at the same time

Waiting for GUI event

Talking to device

Talking to network

Two approaches

Event driven

Threading

bitpim.org

Event driven

Twisted/asyncore

Scales well

Harder to write (“inside out”)

Every library has to work the same way

bitpim.org

Threading

Simpler (initially)

Things can change underneath you

Harder to deal with exceptions

bitpim.org

And the winner was ...

Threading

Decouple threads (and data) as much as possible

wxPython event loop integrates well

Use Queue.Queue to send messages

wx.PostMessage to send to GUI event queue

bitpim.org

Sample Code - GUI

def OnGetFile(self):

...

self.makecall(getfile, path, self.OnGetFileResult, path)

def OnGetFileResults(self, path, exception, data):

if exception is not None: raise exception

...

def makecall(self, callfunc, args, resultfunc, resargs):

self.queue.put( (callfunc, args, resultfunc, resargs) )

bitpim.org

Sample Code – thread loop

result=exception=None

callfunc, args, resultfunc, resargs = queue.get()

try:

result=callfunc(*args)

except:

exception=sys.exc_info()[1]

wx.PostEvent(RESULT_EVENT,

(resultfunc, resargs, exception, result))

bitpim.org

Threading and SWIG

// deal with data being returned

int usb_bulk_read_wrapped(usb_dev_handle *dev, int ep, char *bytesoutbuffer, int *bytesoutbuffersize, int timeout)

{

int res;

Py_BEGIN_ALLOW_THREADS

res=usb_bulk_read(dev, ep, bytesoutbuffer, *bytesoutbuffersize, timeout);

Py_END_ALLOW_THREADS

if (res<=0)

*bytesoutbuffersize=0;

else

*bytesoutbuffersize=res;

return res;

}

bitpim.org

Threading gotchas

Python doesn't have thread priorities

Nor do the event driven frameworks

Cannot interrupt a thread

Have to poll variable

Use setDaemon for blocking calls (eg accept)

Mismatch in producer/consumer rates can have dire effects

Update GUI from idle handler

bitpim.org

Threading techniques

Assertions

against thread.get_ident()

Class/instance wrappers

Forward requests to correct thread

Thread pooling

Checking

bitpim.org

Outlook

If you can do it easily from VB, you can do it easily from Python (win32all)

Dynamic module generation at runtime (cached)

makepy can generate static module

Binary distribution favours latter

bitpim.org

Outlook - code

import outlook_com

app=outlook_com.Application()

mapi=app.GetNamespace(“MAPI”)

contacts=mapi.GetDefaultFolder(constants.olFolder)

for i in range(len(contacts.Items)):

item=contacts.Items[1+i]

print item.FullName

print item.MobileTelephoneNumber

bitpim.org

Evolution

vCards stored in berkely db files (bsddb module)

ebook api

Significant versioning issues

Significant dependency issues

bitpim.org

vCards

No mature Python vCard modules

No data source implements vCards correctly anyway

530 lines for vFile parser and vCard data converter

Correctly deals with every correct and broken vCard I could find

bitpim.org

Wisdom

Use the highest level language you can afford, even if you have to write it yourself

Intent not implementation

Higher productivity

Less hand written code

Easier to tune

bitpim.org

Protocol description

Each field is some number of bytes

Each field has a type

Marshalled type (eg lsb integer)

Python type (eg string)

Some fields are conditional on the values of others

bitpim.org

Protocol Requirements

* RESPONSEHEADER header

1 UINTlsb blockcounter

1 BOOL thereismore

* STRING name

IF thereismore

4 UINTlsb filesize

2 UINTlsb datasize

* DATA data

bitpim.org

Description language

Use Python syntax!

tokenize module to parse

650 lines of code to generate Python

2,500 lines of protocol description

41,000 lines of generated code

with comments

with checking

with introspection

bitpim.org

Protocol example

PACKET foobarrequest:

“a comment”

* requestheader {'command': 0x04} +header

1 UINT {'constant': 0} +blocknumber

if self.blocknumber==0:

* STRING {'terminator': 0, 'pascal': True} filename

* LIST {'elementclass': speeddial} +speeddials

bitpim.org

Protocol Visualisation

bitpim.org

Internal representation of data

Dicts

Keys

Unique ids

Order

Lists of dicts

Really hard to add later

Be database ready

SQLite

Gadfly

bitpim.org

File formats for saving data

Human readable is great for debugging, testing and interoperability

XML

Python 'code'

Concurrent access

Pretty printed dicts/lists works for me

(Except concurrent access)

bitpim.org

Data versioning

Versioning

Backwards and forwards compatible

if version==1:

... convert to 2 ... version=2

if version==2:

... convert to 3 ... version=3

if version==3: ... we are happy ...

if version>=4:

... do something user friendly ...

bitpim.org

Programmer doc

User documentation

HTML is king

Microsoft CHM

Proprietary archive of compiled HTML

wxWidgets help

ZIP archive of HTML

HelpBlocks

Produces both

Preprocessor

bitpim.org

User doc

Web site

Troubleshooting

Capturing exceptions

sys.exc_info() -> type, value, traceback

Frame variables

Python Cookbook

Can transfer across threads

Keep original traceback

ex.original_exception=sys.exc_info()

bitpim.org

Secure remoting

Express methods and parameters

Types important

int

string

dict

list/tuple

So are exceptions

bitpim.org

Choices

Home grown

IDL

Corba

XDR

XML based

XML-RPC

SOAP

bitpim.org

XML-RPC

<?xml version='1.0'?>

<methodCall>

<methodName>add</methodName>

<params>

<param>

<value><int>1</int></value>

</param>

<param>

<value><int>2</int></value>

</param>

</params>

</methodCall>

bitpim.org

XML-RPC

Has exceptions (aka Faults)

Interoperates with other languages

No None

Well supported in Python standard library

HTTP transport in stdlib

No security (https, authentication)

One request per connection

bitpim.org

Attempt 1

http authentication

Every request

m2crypto (openssl wrapper)

Very difficult to prevent connection closes

See “useless destructors” thread on c.l.p

X.509 certificates

bitpim.org

Attempt 2

Paramiko (SSH library)

SSH has named channels

Simple certificates

Authentication on connection establishment

bitpim.org

Auto-remoting methods

__getattr__ to work out name used

__call__ to invoke with args

See xmlrpclib._Method

Be careful!

A tale of __str__

bitpim.org

Printing

Html, the universal panacea

Automatic flow

Use templates

Easy preview

wx.Html lacks style sheets

Can also use in UI

bitpim.org

Publish/Subscribe

Similar to MVC

Decouples code

Threading

Be careful

Weakrefs

bitpim.org

Debugging

Print statements

They tend to stay a long time

logging module

Python debugger

import pdb ; pdb.set_trace()

assert

Do not throw away exceptions!

pychecker

bitpim.org

Do early

Decide on undo implementation

Transaction log

Decide on forwards compatibility of stored data

Have a test plan

Stress/boundary

Normal usage/regression

Have a position on i18n/l10n

Use XRC for user interface

bitpim.org

Standing on the shoulders of others

Python ð language

wxPython ð gui

pyserial ð com ports

Python-DSV ð csv files

HelpBlocks ð Help authoring

SWIG ð Python wrappers

GCC/MinGW ð C/C++ compilers

SourceForge ð Project hosting

libusb/libusb-win32 ð usb access

paramiko ð ssh

win32all ð Windows APIs

ffmpeg ð multi-media format conversion

InnoSetup ð Windows installer

py2exe ð Windows Python Freezing

cx-Freeze ð Linux freezing

bitpim.org

Conclusion

Happy users & developers

Python productivity

Open source

Cross platform

Plug

Also try Dotamatic

bitpim.org

This presentation was given at the Silicon Valley - San Francisco Bay Area Python Users Group (baypiggies) on July 8th 2004.

The purpose of the talk was to show the developer side of a non-trivial application written in Python. It highlights issues and choices, and what I picked for BitPim and why. (Other choices may be applicable for other projects).

Two spelling corrections have been made, as well as some explanatory notes that give more context of what particular slides are about.

The phone was an LG VX4400 from Verizon Wireless obtained in January 2003.

Verizon Wireless uses CDMA as do many other carrier in the US. There is no standard synchronisation mechanism as there is with most GSM phones, and hence there is less software available, and less choice.

This BitPim phonebook, running on Windows. Note that it shows the wallpaper associated with a phonebook entry – something none of the other software packages available do.

This is the wallpaper view, shown on the Mac. It helps demonstrate the cross platform nature of BitPim.

And wallpaper on Linux. Yes, BitPim really is cross platform.

The embedded filesystem available on most CDMA phones. No other software package lets you see this, and this also demonstrates why I started BitPim (to access more functionality of my phone).

The monthly popularity of BitPim over time. The dips in the middle were partially due to SourceForge's statistics system having issues.

Yes, I made a mistake of including “pim” in the program name when I didn't want it to be a pim!

The “Python” is flashing

Because of the loader stub, you need a platform specifc freezer. These are the best of breed on each platform IMO.

This is how many lines of file were used for the freezing and installation. It includes all files (for example the spec file for rpm and the InnoSetup file for Windows).

pyserial doesn't have code for listing available serial ports, so I wrote my own. This is the result shown on Windows.

And the result on Linux. The Mac display is almost identical.

It was not a pleasant experience doing that code!

The C library libusb/libusb-win32 provides USB access on all platforms. I used SWIG to get a Python binding.

There are other high quality tools for doing Python/C bindings, but SWIG was the most appropriate for BitPim.

It is also possible to combine an event driven program (Twisted/asyncore) with threads but I just went with straight threading for BitPim.

This demonstrates splitting a function into two (OnGetFile) and how the request is sent to a worker thread (makecall)

This is the loop in the worker thread which does the call, captures the results and exceptions and passes them back.

This demonstrates how easy it is to allow threading (the bold lines). It also demonstrates the use of parameter names to give cues to SWIG (bytesoutbuffer and bytesoutbuffersize are combined by SWIG to represent a binary string in Python – ie two seperate parameters at the C level become one at the Python level)

The first point is a “secret” that should be shouted from the rooftops!

Note the subtle line where Items is indexed. In COM some items start indexing at zero (standard C and Python) and others start at one (VB default).

One question was why code generation into classes was used instead of just running off the descriptions at runtime. The simple answer is because I have always done it that way, and it seemed right. Sometimes old habits from other programming languages die hard!

This is what the description language ultimately looked like. Note comments (which can also appear at the end of a line) as well as how parameters are passed to fields.

This shows a protocol analyser which was trivially possible because of the use of a high level description language.

Be careful about using data fields for the keys of dicts. For example if I used the name of a phonebook entry, there would be trouble the first time someone knows two John Smiths.

Python code (a file you just exec) is great, and fine for BitPim. However do not use them where the date could come from a third party or is in an area shared with other users.

This screenshot demonstrates the output of the epydoc tool and how helpful it is for programmers.

This is the output from pyxr which shows cross references to standard Python modules as well as other functions and classes. It helps programmers less familiar with Python or your code see what is being referenced.

This is the wxWidgets help viewer running. It looks/functions the same on all platforms.

The preprocessor in HelpBlocks is used to automatically generate the footer on each page pointing to the parent, previous and next pages.

This is the same HTML munged and displayed on a website.

This screenshot shows frame variables included in an exception trace. It makes debugging and troubleshooting a lot easier than with just a standard stack trace.

This is how an XML-RPC request is encoded. The response is very similar. It demonstrates the simplicity of XML-RPC.

XML-RPC works well for BitPim. You can also use SOAP which is like a second system effect on XML-RPC. Note that both can be used at the same time – there are several examples in the Python Cookbook.

c.l.p is the comp.lang.python newsgroup. The existing Python standard library closes connections all over the place making it hard to prevent that from happening when I wanted multiple requests per connection. The discussion thread has more details on when destructors run in Python (if they run) and various advocating on why programmers should have to manually close/release resources such as connections, and equally ardent argument that the language environment should figure it out.

Ultimately I couldn't get XML-RPC to work with authentication over HTTPS with multiple requests per connection.

My second attempt turned out to be very easy (although part of that is due to the hard lessons from the first attempt).

Just like xmlrpclib._Method, I didn't make any attempt to not remote methods beginning with underscores. Consequently I ended up with a hard to debug situation where an exception was causing an exception in an exception. This was due to the exception display code displaying frame variables, which in turn called __str__ on each variable, with the __str__ method being remoted on a failed network connection thereby causing another exception.

These are all the components BitPim uses. They are all compatible with open source programs (even GPL ones like BitPim) as well as proprietary programs. It demonstrates the great building blocks available to Python programmers, no matter how you ultimately distribute/sell your program.

BitPim

2004-06-11T13:48:51