BitPim

• An application in Python
• Roger Binns
• rogerb@rogerbinns.com

Motivation

• New cell phone, new features
  • Phonebook
  • Calendar
  • Wallpapers
  • Ringtones
  • Messaging
  • Voice and Text notes ...

I am human!

• I will not use a number pad
• Sync products limited
  • 'Tickbox' phone support
  • Phonebook only
  • Windows only
  • Single machine licenses
  • Stupid quirks
• I can do way better J

Phonebook screenshot - Windows

Wallpaper screenshot - Mac

Wallpaper screenshot - Linux

Embedded Filesystem screenshot

Popularity

Scale

• 30,000 lines of Python
• 2,500 lines of description files
  • turns into 41,000 lines of generated code
• 500 lines of C/C++
• 20,000 words of online help

Specifications

• Expose and use all features of my cell phone
  • No protocol documentation at all!
• Let me use it on as many machines as I want
  • cross platform
• Interoperate with other data sources
  • No data islands

More specs

• Let others join the party
  • Open source
  • Able to add other phones
• Do not be a PIM
  • Only have UI where no other program does
• Be possible to plug into other programs
  • deal only with cell phone interfacing

Even more specs

• Easy to use
• No installation hurdles
  • No prerequisites
  • No DLL hell
  • No RPM dependencies hell
• Easy to diagnose what has happened on user machines
  • Lack of documentation means learning from the field

The users don't care

• Users don't care what language you use
• Users don't care how hard it is to write
• Users don't care what development methodology you use
• Users don't care about you being consistent with a platform they don't use

The users do care

• Users do care that your program does what you claim
• .. and they want to use it for as little time as possible
  • Goals, not tasks

I care

• I care about productivity
  • No drudge work thanks!
• I care about refactoring
  • Knowledge will change over time
• I care about ease of talking to other libraries & components
  • Productive!
• Active community

Preaching to the choir

• The only solution is
• (And some Python libraries)

Gui options

• Tkinter û
  • Lowest common denominator
  • Draws widgets itself
  • Unpleasant prior experience
  • Stagnant
• Qt û
  • High common functionality
  • Not open source on Windows
  • Draws widgets itself

Gui options

• GTK û
  • Linuxy
  • Draws widgets itself
• wxWidgets/wxPython ü
  • High common functionality
  • Uses native widgets where possible
  • Very active community
  • Pleasant prior experience

Distribution

• Has to appear as a 'normal' application on each platform
• Two steps
  • Freezing Python code (confusingly called installers)
  • Installer

Freezing

• Gather all Python modules, binary modules and Python interpretter
• And data files you specify
• Launcher stub
  • Loads Python interpretter
  • Set path for binary modules
  • Load Python code from archive (.zip)
• Can be placed anywhere – no need for installation

Freezing - Platform specific

• Windows – py2exe
  • Icons
  • version_info resource
  • COM integration/modules
• Linux – cx-Freeze
  • Strip RPATH from wxPython shared libs
• Mac – BundleBuilder
  • Some manual specification of shared libraries to include

Installation – Platform specific

• Windows – InnoSetup
  • Start Menu
  • Uninstall
  • Control Panel
• Mac – dmg
  • Icon

Installation – Platform specific

• Linux – RPM
  • provides, requires (be careful of rpmbuild being too helpful!)
  • Files out of the way, launcher shell script
  • Menu icon
  • $LD_LIBRARY_PATH (cx-Freeze can do this for you)

Not that hard!

Serial Ports

• pyserial library
  • Windows (win32all)
  • Linux & Mac (posix)
  • Interface doesn't know/care about platform
• Full functioned
  • Data rates
  • Flow control
  • DSR/CTS etc

User friendly serial ports

User friendly serial ports

Months of fun!

• Windows – scan registry
  • Different on Win9x, Win2k, WinXP
  • Good detail (drivers, dates etc)
• Linux
  • Device nodes in various directories
  • Kernel module associated with major
• Mac
  • Device nodes in one directory /dev
  • No other information

USB

• Aka Python/C integration is easy
• Libusb provides good USB access on Windows, Linux and Mac
• SWIG generates Python code wrapping C/C++
• Language neutral but best at Python

SWIG

• SWIG is really good
• Does a good job with raw C/C++ header
• Is actually simpler than it seems
• Can pick up cues from parameter names (encourage C library implementors!)
  • Binary strings

C/C++ problems highlighted

• Who owns memory?
• Global pointers
• Data owned by other data
  • May need C++ reference counter
• Poor API design
  • Provide Pythonic layer

Multi-tasking

• Can't do two things at the same time
  • Waiting for GUI event
  • Talking to device
  • Talking to network
• Two approaches
  • Event driven
  • Threading

Event driven

• Twisted/asyncore
• Scales well
• Harder to write (“inside out”)
• Every library has to work the same way

Threading

• Simpler (initially)
• Things can change underneath you
• Harder to deal with exceptions

And the winner was ...

• Threading
• Decouple threads (and data) as much as possible
• wxPython event loop integrates well
• Use Queue.Queue to send messages
• wx.PostMessage to send to GUI event queue

Sample Code - GUI

• def OnGetFile(self):
• ...
• self.makecall(getfile, path, self.OnGetFileResult, path)
• def OnGetFileResults(self, path, exception, data):
  • if exception is not None: raise exception
  • ...
• def makecall(self, callfunc, args, resultfunc, resargs):
• self.queue.put( (callfunc, args, resultfunc, resargs) )

Sample Code – thread loop

• result=exception=None
• callfunc, args, resultfunc, resargs = queue.get()
• try:
• result=callfunc(*args)
• except:
• exception=sys.exc_info()[1]
• wx.PostEvent(RESULT_EVENT,
• (resultfunc, resargs, exception, result))

Threading and SWIG

• // deal with data being returned
• int usb_bulk_read_wrapped(usb_dev_handle *dev, int ep, char *bytesoutbuffer, int *bytesoutbuffersize, int timeout)
• {
• int res;
• Py_BEGIN_ALLOW_THREADS
• res=usb_bulk_read(dev, ep, bytesoutbuffer, *bytesoutbuffersize, timeout);
• Py_END_ALLOW_THREADS
• if (res<=0)
• *bytesoutbuffersize=0;
• else
• *bytesoutbuffersize=res;
• return res;
• }

Threading gotchas

• Python doesn't have thread priorities
  • Nor do the event driven frameworks
• Cannot interrupt a thread
  • Have to poll variable
  • Use setDaemon for blocking calls (eg accept)
• Mismatch in producer/consumer rates can have dire effects
  • Update GUI from idle handler

Threading techniques

• Assertions
  • against thread.get_ident()
• Class/instance wrappers
  • Forward requests to correct thread
  • Thread pooling
  • Checking

Outlook

• If you can do it easily from VB, you can do it easily from Python (win32all)
• Dynamic module generation at runtime (cached)
• makepy can generate static module
• Binary distribution favours latter

Outlook - code

• import outlook_com
• app=outlook_com.Application()
• mapi=app.GetNamespace(“MAPI”)
• contacts=mapi.GetDefaultFolder(constants.olFolder)
• for i in range(len(contacts.Items)):
• item=contacts.Items[1+i]
• print item.FullName
• print item.MobileTelephoneNumber

Evolution

• vCards stored in berkely db files (bsddb module)
• ebook api
  • Significant versioning issues
  • Significant dependency issues

vCards

• No mature Python vCard modules
• No data source implements vCards correctly anyway
• 530 lines for vFile parser and vCard data converter
• Correctly deals with every correct and broken vCard I could find

Wisdom

• Use the highest level language you can afford, even if you have to write it yourself
  • Intent not implementation
• Higher productivity
• Less hand written code
• Easier to tune

Protocol description

• Each field is some number of bytes
• Each field has a type
  • Marshalled type (eg lsb integer)
  • Python type (eg string)
• Some fields are conditional on the values of others

Protocol Requirements

• * RESPONSEHEADER header
• 1 UINTlsb blockcounter
• 1 BOOL thereismore
• * STRING name
• IF thereismore
• 4 UINTlsb filesize
• 2 UINTlsb datasize
• * DATA data

Description language

• Use Python syntax!
• tokenize module to parse
• 650 lines of code to generate Python
• 2,500 lines of protocol description
• 41,000 lines of generated code
  • with comments
  • with checking
  • with introspection

Protocol example

• PACKET foobarrequest:
• “a comment”
• * requestheader {'command': 0x04} +header
• 1 UINT {'constant': 0} +blocknumber
• if self.blocknumber==0:
• * STRING {'terminator': 0, 'pascal': True} filename
• * LIST {'elementclass': speeddial} +speeddials

Protocol Visualisation

Internal representation of data

• Dicts
  • Keys
  • Unique ids
• Order
  • Lists of dicts
  • Really hard to add later
• Be database ready
  • SQLite
  • Gadfly

File formats for saving data

• Human readable is great for debugging, testing and interoperability
  • XML
  • Python 'code'
• Concurrent access
• Pretty printed dicts/lists works for me
  • (Except concurrent access)

Data versioning

• Versioning
• Backwards and forwards compatible
• if version==1:
• ... convert to 2 ... version=2
• if version==2:
• ... convert to 3 ... version=3
• if version==3: ... we are happy ...
• if version>=4:
• ... do something user friendly ...

Programmer doc

Programmer documentation - pyxr - screenshot

User documentation

• HTML is king
• Microsoft CHM
  • Proprietary archive of compiled HTML
• wxWidgets help
  • ZIP archive of HTML
• HelpBlocks
  • Produces both
  • Preprocessor

User doc

Web site

Troubleshooting

• Capturing exceptions
  • sys.exc_info() -> type, value, traceback
• Frame variables
  • Python Cookbook
• Can transfer across threads
• Keep original traceback
  • ex.original_exception=sys.exc_info()

Exception with frame variables screenshot

Secure remoting

• Express methods and parameters
• Types important
  • int
  • string
  • dict
  • list/tuple
• So are exceptions

Choices

• Home grown
• IDL
  • Corba
  • XDR
• XML based
  • XML-RPC
  • SOAP

XML-RPC

• <?xml version='1.0'?>
• <methodCall>
• <methodName>add</methodName>
• <params>
• <param>
• <value><int>1</int></value>
• </param>
• <param>
• <value><int>2</int></value>
• </param>
• </params>
• </methodCall>

XML-RPC

• Has exceptions (aka Faults)
• Interoperates with other languages
• No None
• Well supported in Python standard library
• HTTP transport in stdlib
  • No security (https, authentication)
  • One request per connection

Attempt 1

• http authentication
  • Every request
• m2crypto (openssl wrapper)
• Very difficult to prevent connection closes
  • See “useless destructors” thread on c.l.p
• X.509 certificates

Attempt 2

• Paramiko (SSH library)
• SSH has named channels
• Simple certificates
• Authentication on connection establishment

Auto-remoting methods

• __getattr__ to work out name used
• __call__ to invoke with args
• See xmlrpclib._Method
• Be careful!
  • A tale of __str__

Printing

• Html, the universal panacea
  • Automatic flow
  • Use templates
  • Easy preview
  • wx.Html lacks style sheets
• Can also use in UI

Publish/Subscribe

• Similar to MVC
• Decouples code
• Threading
  • Be careful
• Weakrefs

Debugging

• Print statements
  • They tend to stay a long time
• logging module
• Python debugger
  • import pdb ; pdb.set_trace()
• assert
• Do not throw away exceptions!
• pychecker

Do early

• Decide on undo implementation
  • Transaction log
• Decide on forwards compatibility of stored data
• Have a test plan
  • Stress/boundary
  • Normal usage/regression
• Have a position on i18n/l10n
• Use XRC for user interface

Standing on the shoulders of others

• Python ð language
• wxPython ð gui
• pyserial ð com ports
• Python-DSV ð csv files
• HelpBlocks ð Help authoring
• SWIG ð Python wrappers
• GCC/MinGW ð C/C++ compilers
• SourceForge ð Project hosting
• libusb/libusb-win32 ð usb access

Conclusion

• Happy users & developers
• Python productivity
• Open source
• Cross platform
• Plug
  • Also try Dotamatic