Code Vault

Was writing some code today and ended up with “Dictionary<string, List<Dictionary<string, string>>> resultDictionary”

Does anyone else think that this is hideous? It’s a dictionary keyed by strings and holding lists of dictionaries keyed by strings and holding strings. And it is what you get when you have a statically typed language(C#) and no native dictionaries.

The code is trying to build a tree to encode as JSON to pass to javascript to represent as a tree widget…try saying that ten times.

I know what you’re thinking…and no, the code is no longer there, it only survived for a few mins before another iteration replaced it with something less icky..I tend to use evolutionary steps and refactor as i go.

My workplace has a subscription to SpringerLink. So I’ve been having a blast looking at the different books over there. I found lots of books that are on my Amazon wish list and so i was like a child at a candy store.

The problem is that the books are provided via separate pdf files for chapters. I am not sure why they do that, I think to make it tougher to pirate the files? Although that doesn’t really make any sense. Regardless, it is a bit annoying and I’d like to have the books as one pdf to transfer over to my trusty Sony Reader(more about it later) to read them while on the go.

So i hacked up a python script to rename the individual files by their chapter number making it easier to join. I then hacked up another script that reads these files and joins them in correct order into one file. Sadly, the scripts are only tested on Linux and will probably only run there. The first version used pyPDF and was probably cross-platform, but it had some problems with the pdfs so i reverted to using pdftotext that, as far as i know, only runs under a GNU Linux system.

This is what the code looks like for the script that does the renaming.

#! /usr/bin/python
import os
import os.path
from subprocess import Popen, PIPE
import glob
import sys
import re
import shutil

def main(argv):
    fullPath = os.path.abspath(argv[1])
    print "Looking at %s" % fullPath

    for fileName in glob.glob(os.path.join(fullPath, '*.pdf')):
        try:
            fullFileName = os.path.join(fullPath, fileName)
            print "Converting %s" % fullFileName

            pOpenCls = Popen(("/usr/bin/pdftotext", fullFileName, "-"), stdout=PIPE)
            convertOut = pOpenCls.stdout

            titleContent = ""

            for i in xrange(1, 6):
                titleContent += convertOut.readline()

            titleContent = titleContent.replace('\n', '')

            digitSearch = re.search('\d+', titleContent)

            if digitSearch != None:
                newChapterPdf = os.path.join(fullPath, digitSearch.group(0) + ".pdf")

                while os.path.exists(newChapterPdf):
                    newChapterPdf = newChapterPdf + "dup.pdf"

                shutil.move(fullFileName, newChapterPdf)

            else:
                print "File is not a chapter, checking if front or back"
                if fileName.find("front") != -1:
                    firstPdf = os.path.join(fullPath, "0.pdf")

                    while os.path.exists(firstPdf):
                        firstPdf = firstPdf + "0.pdf"

                    shutil.move(fullFileName, firstPdf)

                elif fileName.find("back") != -1:
                    lastPdf = os.path.join(fullPath, "9999.pdf")

                    while os.path.exists(lastPdf):
                        lastPdf = lastPdf + "9.pdf"

                    shutil.move(fullFileName, lastPdf)

        except OSError, ex:
            print ex
            print 'Failed', fullFileName

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print "Supply directory as argument"
        sys.exit()
    main(sys.argv)

This is what the code looks like for the script that does the joining. pyPDF is needed for this to function.

#!/usr/bin/python
import sys
import glob
import os
import os.path
import pprint
from pyPdf import PdfFileWriter, PdfFileReader

def compareByActualNumbers(x, y):
    strippedX = os.path.basename(x)[:-4]
    strippedY = os.path.basename(y)[:-4]

    if not (strippedX.isdigit() and strippedY.isdigit()):
        return 0

    xInt = int(strippedX)
    yInt = int(strippedY)

    if xInt < yInt:
        return -1
    elif xInt == yInt:
        return 0
    elif xInt > yInt:
        return 1

def main(argv):
    fullPath = os.path.abspath(argv[1])
    outputFileName = os.path.join(fullPath, "fullBook.pdf")
    print "Looking at %s" % fullPath

    outputPDF = PdfFileWriter()

    pdfList = glob.glob(os.path.join(fullPath, '*.pdf'))
    sortedPdfList = sorted(pdfList, compareByActualNumbers)

    pp = pprint.PrettyPrinter(indent=2)
    pp.pprint(sortedPdfList)

    for chapPdf in sortedPdfList:
        pdfInput = PdfFileReader(file(chapPdf, "rb"))
        for i in xrange(pdfInput.getNumPages()):
            outputPDF.addPage(pdfInput.getPage(i))

    # finally, write "output" to document-output.pdf
    outputStream = file(outputFileName, "wb")
    outputPDF.write(outputStream)
    outputStream.close()

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print "Supply directory as argument"
        sys.exit()

    main(sys.argv)

You can download the scripts from the My Projects section. The code will be licensed under the GNU GPL.

Note to publishers, you really should step into the 21st century. Especially for the technical books, I really don’t want to pay shipping costs that are more than the price of the book, the printing costs and all the other unneeded crap…I just want the content. This is especially true for technical books that get outdated very quickly such as books about frameworks.

Michael Ivey recently wrote about his git work flow over at My Git Workflow and it turned out to be a pretty useful article and was just the encouragement that i needed to blog about my own experience. Although I’ve used git before, I much rather use bzr whenever possible. I’m not into religious wars about tools, so use whatever you like.

I work as an independent programmer(freelance programmer if you prefer) and so I am often working on several projects at the same time, mostly on my own, but sometimes with 1 or 2 more people involved.

Feature branches

I use standalone branches in bzr so they will have their own directories. I create a parent folder with the name of the project, say KillerIdeaX. And inside it, i would have the project’s branches. One called KillerIdeaXtrunk which is the stable version of the project. And other branches called feature branches for…well, features.


mkdir KillerIdeaX
cd KillerIdeaX
bzr init KillerIdeaXtrunk
bzr branch KillerIdeaXtrunk KillerIdeaXFeatureA
...hack....
bzr merge KillerIdeaXFeatureA KillerIdeaXtrunk


If the project is small enough, i would just have the trunk folder and i work on that. But usually, the project will have a number of distinct features. So i create feature branches and work on those and merge them back into trunk after testing. The trunk branch is also where i take snapshots from to deliver to the client. So i always know that what i am delivering is a stable build.

If i am working with other people, i use an approach that is similar to feature branches. Whenever i receive code from the client(or co-worker), i create a standalone branch such as KillerIdeaXWithFooFeature and thus be able to quickly see what is different using bzr commands(such as diff and others).

Say Cheese! Tag!

I mentioned taking snapshots from the trunk to deliver to the client for testing but i didn’t explain how. What i usually do is simply export the code from trunk(which would always give me a stable build) and then tag that version so that i know which version the client is talking about when i get the inevitable email telling me that the whole thing is broken.


...receive angry email from client demanding a prototype...
bzr export KillerIdeaXtrunk KillerIdeaX.zip
....email KillerIdeaX.zip to client with usual warnings about software prototypes...
bzr tag "0.1"
....now we can both argue about same codebase....


I need more shelf space

When you begin using any version control system, it might be somewhat challenging to learn to code via separate changes as entities. So one feature would be one change, one bug fix would be another change and so on. Changes are grouped logically. As in, what are the modifications that logically fit together. That way, it becomes much easier to roll back certain changes or to chase down bugs or retrieve an earlier version of the codebase…etc.

While this is a proper programming practice, it sometimes can be hard to follow. So let’s say you’re working on some feature and while testing it, discover an unrelated bug. The proper thing to do would be to branch out from your last revision, work on that branch to fix the bug and then merge the changes back in and then go on working on your feature. This makes sense, though, only if the bug fix is significant enough. Often, fixes are one liner changes and so it will be much faster to simply fix in place. Of course, doing that would ruin your ability to separate changes and so rolling back your feature for what ever reason also means losing that bug fix.

This scenario is excellent for the shelve command in the bzrtools plugin(recent versions of bzr have this command built-in in a slightly modified way). The shelve command interactively asks you about the changes in the source code(by hunk) and asks you if you want to keep it or shelve it. Shelving changes simply removes them and stores them in a temporary location until they’re un-shelved.


....working in KillerIdeaXFeatureA branch, discover unrelated bug....
bzr shelve
....choose to shelve all changes related to feature...
....introduce one liner fix for bug...
bzr commit -m "fixed silly bug related to foobar problem"
bzr unshelve
...continue working on feature....
bzr commit -m "implemented awesome feature zazzle"


Note that using the shelve command, you can shelve and un-shelve any change you want. So you could’ve introduced the one line fix and then before committing the feature shelved that instead for example.

Can you see it?

As you can see, there’s a lot that you can do to optimize your process and using a distributed version control system means that the administration overhead of a centralized one is no longer there. So you really have no excuse not to make use of its power…even if you’re a solo programmer.

(I tried to stay away from bzr terminology in this article to give you a general idea of the tool. But do check out the great documentation that comes with it.)