I mostly write throw away code

I'm pretty sure Github (any other social code sharing site is also guilty) has changed my coding habits forever. What was once a mostly private activity followed by some sharing is now extremely public. Why do I choose to push code there which I've spent few hours working on. Well, it's hard to resist the energy. You follow all your friends and see all the projects they've been working on and who's committing to what. Regardless of how good anything that is being committed might be, you want to share too. Not committing for even a few days feels like you're falling behind in a race. I don't even know when this race finishes; maybe it's just an endless sprint to be something. Surely my willingness to produce and share the amount of code that I feel is necessary to keep up is taking a tole on my quality of work; how could it not?

Most of my code lacks documentation. I'm not even talking about formal docs; it's lacking those too. No, it usually lacks even the slightest organized overview of what it does and why anyone would really want to use it. A few months later, I might not even remember why I wanted to use it and chances are I don't anymore. This code should never have been released. On the insanely small chance someone else stumbles onto it, they have a near zero clue of what it's for.

I usually wing the API design. Heck, flying by the seat of your pants never hurt anyone right? This berserk idea that I can get an API design right just by thinking it up on the spot is so wrong. If anyone were to tell me they just thought up a new API and coded it up immediately, I'd know they got it wrong. Why don't I bother setting the same standards for the code I share? I'm stuck maintaining bad API's if anyone else uses it and I'm stuck using the bad API if I use it. Its sadistic and dumb. I have absolutely not taken the time to design a modular, transparent, discoverable library. This is not something that can be done overnight, but I sure act like it can.

Tests. Pfft.

So, why do I share throw away code? Well, sometimes an idea or two in it is worth sharing. Most of the code is short enough that it can be read in a few minutes. I guess I am really sharing so others can read it, not use it. At the end of the day, I love the ideas that I produce more than the code that represents them. Ideas are representable so many ways, but code speaks the clearest to me; other programmers might agree. At the end of the day, most of what I'm sharing is bad code representing interesting ideas and if you take an honest look of what's hosted on Github, you would see everyone else is doing that too. There is only so much time for doing things the right way. There is a lot of time for sketching ideas and experimentation. Being good is knowing the difference and when it matters.

Beginnings of a python analyzer

Analyzing the source code your running is a massive task. An enormous amount of effort goes into search for and fixing software faults. One thing I've always wondered is how would an analyzer (static or runtime) look if it were examining python. Being a dynamically typed language, python can be extremely tricky to analyze from just its syntax. However, thanks to Armin Ronacher it does have one excellent feature which could allow deeper analysis; direct access to a parsed ast. With this module you can examine, manipulate or mangle the language any way you see fit.
I've decided to attack the number one bug creator in my opinion, passing None to a function. If this were C, that would be like passing NULL. I can't really see why there would be a good place to do that and in my experiences its usually causing a bug. Beyond the basic reasons, its just semantically wrong to be passing around nothing. It sort of defeats the point of providing arguments at all. In python, we've got really nice default argument syntax so if you must have None for a argument, make that the default and don't pass anything. If you can't avoid this due to some API then I'm sorry.
Anyway, here's how I went after it; first I created a class which extends NodeTransformer and added this method:

def visit_Call(self, node):        
 self.generic_visit(node)        
 for x in chain(node.args, node.keywords):            
  x = x.value if hasattr(x, 'value') else x            
  if isinstance(x, Name) and x.id == 'None':            
   warn("Passing None Argument to function", RuntimeWarning)
 return node

Then I just use that method in a simple function to create the ast and walk it:

with open(sys.argv[1]) as f:    
 source = f.read()    
 ast = parse(source, sys.argv[1])    
 node = StaticTransformer().visit(ast)

Now when I call this script and pass in any compilable Python code, if there are any warnings I'll know about them and be able to fix them before the code gets used elsewhere. It is easy to add more methods to check ast nodes, you just follow the visit_<Node> pattern for naming the functions in your NodeTransformer class. I've created a gist with the code that I've written so far, feel free to fork and contribute!

Happy New Year

To all my readers, if there are any, have a Happy New Year and a great 2011!

The code is the documentation

Yeah that's right, it is. If you're a working programming, you spend a good amount of your time writing in a language few people understand and then a bunch of time writing in english to help people understand what the hell your run-on overly complicated login function does. What you probably did is believe that you'd document it later and then you tried to be clever. That was a mistake; you're not Shakespeare or Chaucer. Your cut and paste, cargo-culted crap reads like a 4th grade essay on the death penalty. There needs to be a serious change on how we look at the work products produced from programming.

When you buy a cabinet, a chair, or anything from a master craftsman, do they design it with some clever way to sit in it and then write a manual in terrible english prose to explain how you should be sitting in it? NO!... Well at least not most of them, maybe some douche-bag avant-garde retro-bauhaus designer would, but who cares. Let's face it, in life most of what you do will not be sophisticated or clever or even fun. It will be work. The difference is not between how you deal with the opportunity for creativity but the opportunity for craftsmanship.

There are a few things that I've noticed in my short term in industry

• You will do more craft than art in life (even if you're an artist)
• If you do something at the limits of your comprehension, you will not understand it later
• Cleverness leads to sadness

After writing those down a few days ago, I've had some time to reflect on what that all means. It also gave me time to decide that the first and last concepts, while important, are just not as useful as the second; I'm throwing them away for now. Really the most important thing you need to remember is:

If you do something at the limits of your comprehension, you will not understand it later

On second thought, maybe this post was the limits of what I can understand; I should stop trying to say something profound...

Being lazy in Python

One of my favorite features of Haskell is its lazy evaluation of list comprehensions. This makes it possible to take the nth position item of a list without first having to calculate the entire list. Pretty sweet if you want to reason of infinite data sets such as the Fibonacci numbers or primes. In a perfect world, I'd be writing Haskell all the time. However, that's just not possible. Python is far more popular, is installed by default on many OSes and can make some tasks very simple. So really, I want lazy list evaluations in Python. The only way to implement this without hacking up the actual interpreters object space is to mess with iterators. So I've implemented the Fibonacci sequence as an iterator. Essentially, I use the next parameter to move over the list of Fibonacci numbers and calculate the next one. I then store any numbers which I have already calculated to prevent extra work. I then use that pre-calculated list along with further calculation to implement list subscripting and contains. One thing to note is that itertools.takewhile will continue iteration for one through the failing condition, not up to. Hence the existence of a rollback function. Overall it was a fun exercise on using iterators and who knows, maybe it's actually useful.

from itertools import takewhile


class Fib(object):
    i = 0
    items = list()

    def __init__(self):
        self.n = 0
        self.n_1 = 1

    def __contains__(self, item):
        if item in self.items:
            return True
        else:
            self.items += list(takewhile(lambda x: x <= item, self))
            self._rollback()
            return self.items[-1]

    def __getitem__(self, key):
        if key <= len(self.items):
            return self.items[key - 1]
        else:
            self.items += list(takewhile(lambda x: self.i < key - 1, self))
            self._rollback()
            return self.items[-1]

    def __iter__(self):
        return self

    def next(self):
        next = self.n + self.n_1
        self.n = self.n_1
        self.n_1 = next
        self.i += 1
        return next

    def _rollback(self):
        """
        When iteration gets stopped by takewhile we've gone one too far
        so it needs to be backed up one
        """
        self.n = self.items[-2]
        self.n_1 = self.items[-1]
        self.i -= 1

Update: I've got a gist going of this which includes some bug fixes for slicing.

GSoC's Over

Well, it was fun, but as of today it's pencils down. I've tagged my repo with a snapshot of my summer's work. I'll be continuing dev of this project so don't despair! In addition, I will probably be speaking at pdxfunc in October about the project. Hope to see you there!

Abstraction is Complexity

I've been thinking more and more about the issues associated with abstraction. I am becoming convinced that abstraction is not always a good choice and certainly should not be taken lightly. When developing structures that other developers will rely upon, you're coding for a much more critical and savvy audience. Overly complex and needlessly heavy-handed code is always the wrong way to do it. Your efforts in API design might be completely destroyed because of a few small choices. Why wrap up simple things like object constructors, when you can expose them instead? Don't abstract unless you have to and when you do don't abuse the privilege.

I've been working with JS a lot more and I think the patterns used in that language to give the appearance of classical inheritance are terrible. Let the language be what it is, not what you wish it was. Most directly, I'm talking about a JS charting library. It is clearly an API designed around object configuration, not object manipulation. In my attempts to provide a simpler API for less experienced developers to use, I've discovered the quirks that are inherent to the choices its designers made. I believe there is too much data hiding. If I wanted to configure an object on the fly, I'll need to reconfigure the whole thing because the API hides my direct access to these configuration functions. I see no reason for these to be hidden, but for these reasons I will abandon my efforts to simplify things further.

API designers, please trust your users more. I've learned this lesson the hard way. It is better to keep an API simple and require the user to do a little more work, than to wrap it up in a leaky abstraction; and lets face it, all abstractions are leaky. If you're going to abstract something, make it opaque and simple! Too much sekret sauce might make things look nice, but when it all goes wrong it won't be pretty.

APIs should provide all functions that are needed to manipulate the underlying public data structures. Don't make things private unless you need to; In fact, don't make it private unless its for internal use only. For example, if you have a chart with axes, and there is an Axis constructor, make that public. Selfish API's don't make me want to use your library. We're using your code to make our lives simpler not harder. If you've gone out of your way to hide important details --I'm looking at you Javascript!-- no one will thank you.

Remember, you don't know what your audience is assuming, maybe its better not to assume at all. And in the case that you do keep assumptions in your code, list them out somewhere! Maybe seeing the number of assumptions made is a good exercise in facing the complexity you've created, since your users will also need to keep track of them too. Now, I will grant that this opinion is a bit harsh. The ideal is probably somewhere in between. Without assumptions code is tedious and tedium leads to mistakes. However, more is not always better so I will leave you with this thought: magic is capitalizing on assumptions, but the best tricks are usually the simplest.