eat your own shit in copious amounts

Over the past 6 months I’ve become acutely aware that a majority of corporate America must run so incredibly inefficiently. There are probably only a handful of companies that run smoothly, everything else suffers from split brain-esque issues. I knew this to be true prior to working in a company that suffers this, after all this is why I started a company of my own, I just didn’t realize the causes for it; I think now I do.

As your size increases you’re expected to master lots of different disciplines. For example software management, content screening, report creation, HR, QA, server hosting, the list goes on for a while. Purchasing these services is expensive, and unfortunately since people do not purchase them readily that market place hasn’t entered a race to the bottom – the only exception to this, in my fairly limited experience, is IT support, which usually amounts to backing up of data before reinstalling windows on the machine. Offering these services in bundles to companies is hard because of the level of bespoke business rules involved, you cannot apply QA tests A, M and X to all products from companies a, m and x.

Software Engineers, being people we are, try their damnedest to fix this. We have built things like Continuous Integration, Unit Testing, Test Driven Development, etc. These provide some concept of baseline which we operate upon, specialization being incorporated. System admins have automated update processes that they can employ from many linux vendors/distributions and MS, etc. You can outsource your HR (but it costs a ton). So there is work going on in this space.

So why is it so inefficient? Am I being ignorant to believe this problem should be solved by now? After all computers have already changed the face of our world indelibly and for the better. Perhaps this is the long tail of the problem?

I would assert that the problem is in fact due to the explosion of computing and the relatively attractive salaries it provides. I have a graph that I draw every so often when explaining why I think computing is going to hit an inflection point where the number of people required to perform our computing jobs drops below the number of people claiming to be able to perform our computing jobs. At that point offered salaries will decrease and the jobs will go to the people willing to ask for less.

The argument, in my mind, stands up because computing is becoming easier; Ruby exists – that simple fact means that my mother could probably throw together a website without too much thought (note: I mean a functional website versus a flat one), this is not a criticism of ruby, merely a statement that computing is becoming easier, we’re achieving a primary goal of making computers an extension of our lives.

However, the general dumbing down of computing means lower quality people enter the work force and as such for the specialized items that are not easy yet (QA, Reporting, HR, etc) are being handed to them. Hiring departments rarely know how to tell a talker from someone that can also walk it, so the signal:noise ratio is high. Net effect underqualified people get these jobs that need to go to specialists, productivity goes down; and it is only going to continue in that vein.

Now I just need a solution to this problem!

I think one of them, and the source of the title of this document, is to just do what you do and do it a lot and do it well. Find other ways to do the things you don’t do as a core competency – start outsource everything that you’re not good at. Embrace trust in other parties – let them be responsible for hiring and have a strict agenda by which they must abide – if bugs hit the real world – replace them with others. By creating that type of market we’ll, hopefully, end up in a utopian world where only the people that are good at what they do are doing what they should be doing!

 

Compiz on Ubuntu

I’m not entirely sure who I shit on in my last life, but my new company has me using PC hardware. I’ve managed to get a semi-decent machine (Compaq 8510w) maxed out with 4GB of mem.

With all of this luscious memory available to me I had to do something, so I installed compiz – it’s so pretty! I’ve not switched on any bells and whistles yet – but the high res on this monitor and the NVidia Quadro supporting the rendering it’s pretty fun.

learning NIO via clojure pt. 2

So it turns out that my assertion about not needing to clear the set returned by selector.selectedKeys() was incorrect. My assumption was based upon a broken implementation of my clojure! In fact it turns out that not only must you alter the state of the socket (in the case of OP_ACCEPT) but you must also clear the set. The assertion that NIO was crap was, however, not incorrect! As far as a framework goes it is far too complex without any clear reasoning for why.

Interestingly I found a better tutorial online, irony of it is that its one of the top hits but a combination of the URL and writing style make it impenetrable – I dug in deep and started converting to clojure the contents of the tutorial (clojure code)

learning NIO via clojure

Short summary of recent events:

• Heysan got acquired
• I work for the acquirer (Good Technology / previously known as Visto Corp)
• Good acquired InterCasting Corp
• Met Ben from ICC
• Moved to new appt with Ben

One of the first things I was tasked with at Good was to get IM working in the Intercasting product using heysan as the backend; intercasting had been using Openfire. This involved us building a protocol which essentially replaced Jabber + XEP-0100 + Gateways. It’s called GIMP. I’ve not done any direct socket stuff like this before, at least not in Java, I was put on to the Apache MINA Project. In short, 3-4 months later – it’s really not a good library.

Part of living with Ben has involved me getting into the clojure scene a little. The combination of MINA being crap and clojure being interesting has me re-writing GIMP in clojure, this meant I have to learn NIO. I’ve learned some interesting things about NIO in the process:

• It sucks pretty bad.
• Most examples on the internet suffer because of that, i.e. most of them are inaccurate

The latter point forms about 50% of the reason for this post. 25% is about clojure, the remainder was the update above :)

The stage I’m at is still very early on, I have a socket and I want events that tell me connections are being made. In NIO you work with selectors, they “select” the socket (this is basic Socket 101). When an event takes place it goes into a queue which you read from the selector using the method .selectedKeys() – it returns a Set<SelectionKey>. Most examples show you removing the key after processing it, this is not required  (at least so far it isn’t..); this caused a lot of distraction for me because Clojure generally has immutable collections, thus remove seemed weird to require. The event you register with your Selector determines what you must do to each SelectionKey to ensure it is removed from the selector; otherwise you end up with a the SelectionKey not ever being removed.

Consider, you register OP_ACCEPT, you must accept the connection for it to leave the queue. This wasn’t immediately clear to me, perhaps I’m stupid and missed this. If you don’t do this that connection attempt event stays in the selector and thus in the inifinte loop you have which selects on the socket you end up with a tight loop. The set you get from .selectedKeys() is populated at the time of calling so only if you’re going to iterate it multiple times do you need to remove events from it (as many of the examples online show).

I’ve two implementations of a very simple system which accepts a connection and prints out a message. The exercise of getting this done in clojure was pretty difficult for a couple of reasons:

• There are examples in Java (though they are often inaccurate they work)
• clojure removes a few constructs which make “sense” (for loops for example)
• the clojure equivalents are often purposefully lazy (incase of large data sets)

Implementations: (Java) (Clojure)

You’ll see the (doall map …) stuff in the recv function, well that forces the realization of the sequence. I believe this is possible using the for form, I’ll work that out later.

I’m thinking that this socket stuff might form the basis of me attempting to create a jabber library, I’ll probably not get that far. The only real java library I managed to be get on with was smack, and that was bad for our usage. We had a server side instance of it thus could have benefited from them creating sockets using NIO. I actually think the functional form makes a lot of sense for something like jabber, we’ll see. Anyways, expect a few posts detailing what I find about NIO and clojure.

mport java.nio.channels.*;

import java.net.*;

import java.util.*;

import java.io.*;

public class Server {

private ServerSocketChannel server;

private Selector selector;

public static void main(String[] args) throws Exception {

Server s = new Server(args[0], Integer.valueOf(args[1]));

s.recvLoop();

}

public void recvLoop() throws Exception {

for (;;) {

System.out.println(“About to select”);

System.out.println(String.format(“found %d elements”, selector.select()));

Set<SelectionKey> keys = selector.selectedKeys();

for (SelectionKey key : keys) {

//SelectionKey key = it.next();

//it.remove(); //required

keys.remove(key);

Socket s = ((ServerSocketChannel)key.channel()).accept().socket();

PrintWriter out = new PrintWriter(s.getOutputStream(), true);

out.println(“k, thx, bye”);

out.close();

}

Thread.sleep(500);

}

}

public Server(String host, int port) throws Exception {

server = ServerSocketChannel.open();

server.configureBlocking(false);

server.socket().bind(new InetSocketAddress(host, port));

selector = Selector.open();

server.register(selector, SelectionKey.OP_ACCEPT);

}

}

I now have gears

Google gears has always been an interesting concept to me. The interest is more from a “WTF?!” perspective than “I wan’t to code for that” perspective. I’m noticing more and more sites using it. WordPress is using it – called “Turbo” – and both Google Reader and Google Calendar are using it. No Gmail support yet?

I’m fortunate to pretty much always be online. Everywhere and expect to use my laptop I have a net connection. For the cases were there isn’t a net connection I have my phone. I do my laptop and phone setup for modem use – but I’m such an avid mobile enthusiast that I’ve not had to use it yet.

Does anyone have any good examples of Gears in use?

My dad has written some software (windows app) that lets his kids train on their math skills. He has seen them increase in math dramatically since they’ve been using it. I think its a great idea and prime for a web app. There is even the potential for a vertical social network where kids can anonymously compare themselves against their class – see how their doing, etc. Schools could use it to tout their proficiency and rates themselves against other schools in the district. As far as performance related payments for teachers go it provides the ability to view which teacher has aided their class the most.

Is gears a candidate for this? Kids using online educational software means they spend more time in front of the machine and theoretically more time able to be led astray by perverts, predators, etc. All the flexibility of a web app running without a network connection does sound pretty appealing.

anger at ownership

It was perhaps last year that I first wrote to Brad Feld about something I considered ‘META-WEB’. The idea behind it was that information stored on various sites should be accessible. Sites should have control over what they want to publish, but overall most of the information should be query-able. Think of it as some mass OAuth + MySQL type setup. Each site can auth a user on another site using their credentials and after that has happened you’re able to load information into your site.

The problem with this idea is ownership. Facebook et al believe they own your data. They want to believe that because it is better for them in the long run for that to be true. Facebook has spent lots of money gathering, storing, srting your data – so why should they give this to other sites, competitor sites, for free?

It was there that I was stuck. I was hoping through Brad Feld to be put in contact with smart people that could help me think my way over this hurdle, however due to work / time constraints I couldn’t approach the situation with any seriousness.

Now a year later, an age in computing, the problem rears its head again and I’m unable to do something that should be fairly simple. Google Voice has my contacts but I cannot call; quite simply Google Voice’s contacts are from GMail and therefore they’re by nature email contacts. Who I call is a subset of who I email, so they’re on to a winner, however as it stands none of my contacts in gmail have a number associated with them.

I decided to fix this.

It’s not easy.

It’s not hard.

It’s just freaking annoying because of ownership by Google and by Facebook. Here is how it breaks down:

1. Decide to scrap facebook’s mobile web site to get phonebook numbers
2. Decide to match against google voice contacts by email address
3. Facebook encodes email addresses into pictures (use SVG to scan?)
4. GoogleTalk doesn’t send telephone number along with Jabber VCard
5. GoogleTalk doesn’t allow setting data using Jabber VCard

Annoying ownership – it seems the path of least resistance to be getting what I want is to create a mobile web site which goes and scraps facebook, gets contacts and their numbers and logs into google voice’s mobile web site to trigger phone calls to your contacts.

Until there is an API this type of mash up will have to do. I only wanted to use Jabber so that I could give the numbers back to Google – however between them not allowing helpful information to be sent and recieved using VCards and Facebook doing what they can to make using their data as hard as possible I’m unable to make the world a nicer place.

To salt the wound I was hoping to use Google App Engine to host this; I’ve been hankering for some python and also I’m looking at how to save cost everywhere for small idea’s. They do not allow you to override the User-Agent using their URL lib. I understand why – but it means I cannot exploit their user agent detection to get access to their mobile web interface. As it stands, mobile web is becoming my favoured way of building up server side mashups – no javascript therefore easy to script HTTP calls into it.

Hopefully by the end of this week I have a usable mobile web site which lets you call your facebook contacts for free!

write less, be better

http://asserttrue.blogspot.com/2009/03/how-to-write-fast-code.html

This is a great article and I’m only a few lines in :)

Something of an underlying thought going through my mind the past few months is `write fewer lines`. Some part of that is about choosing good libraries. Some part of that is about not writing excessive checking. Some part of that is abstracting. Most part of it is how to be a good team member.

At heysan we recently released to a select number of people our IM-invite widget. It’s pretty neat. A core design principal throughout it was to use tools which let me develop on my age old mac and run code on prod without having to worry about environment changes. The tools selected to achieve this were ant, junit, java, spring, hibernate. All together it worked pretty well; where I needed a certain type of feature I was sure to find a spring project for it (spring-security in this case), where I needed to do database based stuff I had hibernate to help me out (desktop ran hsqldb and live runs postgresql). Net effect was I had to write less code; and using tools that ship alongside spring made that more true. I have way more experience using struts but I didn’t want to have to write the struts / spring integration stuff so I used spring mvc.

The more lines of code you write at the birth of a project the more you have to describe as the project matures. If I write 100 lines in a method I have to write at least 10 lines in the java-doc method comment. I hate, in general, writing comments so its in my best interest to write fewer lines in the method body. Using products which have good documentation means that I can reference them when explaining certain things to a new person on the project.

I have noticed a disconcerting trend towards googling for answers when reading documentation would suffice. I do it. I know many others that do. I think I do it far less than I used to. I’ve learned where to go to find good docs, I’ve learned what tools to use that have good docs. It seems people approach problems as though they’re new and pure; versus “does my library already handle this”, I find people always write stuff new versus converting an existing class/concept to handle the new data – almost like refactoring is evil or something… However the scariest thing about this trend is that it seems because you found the solution online you do not write any comments – not even referencing the page you found it.

Message of the day – write comments, write fewer lines of code, write test cases, make things easier to support. You’ll spend more time working out why something is broken than working on cool new products if you don’t.

twitters struggle

I have never really wanted to weigh into the the twitter-making-money-debate. I like the twitter team (their office is way too quiet, but they’re nice people!). I will just say this one thing however: “if their focus is on making money, how do they intend to do that if they have uptime issues?”.

About once a day I get the fail whale. I do not mind at all. If you see their growth then as a techie you have to sympathise with their struggle. Overall I think that they cannot focus on making money until they are stable; Niklas Zennstrom once told me not to move onto the next thing until you’ve finished your first thing. I’m not sure @biz and @ev are being realistic when they say that twitter can focus on monetization this year. Perhaps they can flesh out idea’s – but any fast moving start up needs to work on new idea’s otherwise divisions build and frustration mounts.

Strategically I think that means twitter will start using that valuable (fb valued them at $300mm-$500mm) stock of theirs to buy teams / products that have worked out how to make money using twitter. While keeping their entrenched employees focused on building a more stable platform.

perhaps if I write this down, I’ll remember it!

Dyslexia is an interesting issue to have, one of the few our scientists know so little about. I mean, our brain as a whole is still black-boxy for the large part, and how things like dyslexia really stand out as “simple” problems that our scientists have yet to solve.

I imagine it will become more complex as we head towards media-dyslexia. For example I suffer dyslexia of some sort when it comes to boolean logic; so I imagine my dyslexia is computer-media related. Whenever I mean to type “not equals” I end up doing equals. I catch myself pretty quickly and I have adapted the way I code, probably for the better, to avoid complex compound decisions such as:

if (!equals(test1) && (test2 || test 3 && test 4))

Try as I might these type of things always feel pot luck as to whether I get them right on the first attempt – usually taking three or four goes. I have found moving away from this type of complex compound boolean logic the safest option in the long run.

My second form of dyslexia isn’t at all akin to dyslexia at all – but I struggle to find another word that can describe it accurately that isn’t the term “plain stupid”. I subscribe to the singular named table school. I.e. user over users, campaign over campaigns. I recently discovered with Hibernate that leads to an interesting issue when being used with postgres.

I had a mapping table mapping my user object to the table USER. I quickly realized that hibernate couldn’t insert into this table because of the name, Will pointed out that you can quote the table name in hibernate, I changed my mapping to be &quot;USER&quot; (xml mapping file). All my other tables were lower case so I decided to add &quot; before and after the table name. Now I had all upper case tables and my schema looked pretty.

I then deployed both web applications I’m working on at the same time and noticed that I got two sets of tables one set with capitalized names and one set with lower cased names (minus USER which was just the upper case variant). If I deployed my applications with a decent length of time in between (stop tomcat, drop database, create database, deploy one app, start tomcat, stop tomcat, deploy second app, start tomcat) I ended up with just upper case table names. I decided to see if this behaviour was the same if I lower-cased my table names, changed all my &quot;TABLENAME&quot; to &quot;tablename&quot; and deployed my applications at the same time – all lower case table names.

spooky.. huh.

I checked the logs and couldn’t see anything that really stood out at me. Googled a bit and couldn’t find anything either. Hopefully if you find this its because you have the same problem and this helps you. If not… thanks for reading this entry!

update: turns out you’re meant to use back ticks (`) to mark where hibernate should quote the table/column names for you.

eclipse vs intellij

At heysan we bought intellij 7 licenses about 18 months ago. Before that I had written and maintained heysan using netbeans and vi. Intellij was a really revolutionary step forwards, it worked so well. It was one of the few tools that I can honestly say worked as I expected it to.  If I wanted to set a default, there would be a screen in the right sort of place within its menu system that had a text box with a label that read pretty much what I thought it would. It was great.

I had ended up accustomed to Netbeans from my time at the cloud. Prior to that I had used eclipse.

Eclipse and Netbeans are pretty much on par with one another; Kevin preferred us to use netbeans as it allowed a greater level of flexibility when it came to setting up a complicated EAR-type project. Netbeans’ usage of ant is pretty flexible, so since kevin was a vi-extremist he wanted us to use an IDE that didn’t interfere with the way that he wanted to setup his project.

I’ve adopted his project setup by-in-large – its clean and clear. I’ve changed some things to work in a way that I prefer, also I do not create EAR’s so I do not need the level of hierarchy. Netbeans stuck after I left cloud as Eclipse was just far to stody, I’d go as far as saying I *really* hate Eclipse.

So why I started my most recent project in Eclipse is beyond me! I could have used vi (no code completion kinda sucks, also no in-ide code testing with clickable results is a bit limiting), I could have used Netbeans (no reason not to) but I chose to see what eclipse was like after 3-4 years away from it (flirtomatic used it).

Whenever there is a *really* good paid for piece of software I often find myself expecting there to be a just as good free product, but it seems in the world of IDE’s this is not true. I find that so interesting. I have been reading for a long time that eclipse is so much better than it used to be, that they eclipse foundation had started moving in the right direction, WTP was the solution to all the web app creation issues that we’d been suffering with.

It is not true.

At all.

I expect eclipse suffers the ‘too many buckets’ issue. It went from being a Java IDE to a Java framework on top of which you build IDE’s. I understand from Jonatan that eclipse-PHP is pretty good. However as a Java-IDE it truly sucks. I am willing to accept that a portion of its suckage is related to me not knowing which plugin to install. I’m willing to accept that eclipse is suffering from my non-eclipse-esque project layout. I’m willing, in fact, to accept that everything that I hate about eclipse is down to some sort of problem I have. That level of accepting, however, speaks volumes of just how bad eclipse is – why must I learn about 10 different third party plugins, configure each Junit test by hand to use a common environmental variable, repeatedly sync eclipse with my filesystem when I pull changs from our repository, add build path jars to the build path explicity rather than just add the directory they are in.

Clearly the argument of you get what you pay for could be applied here. However so many OSS things are great and are free. I’m fetching netbeans at the moment, I really have had it up to my neck with Eclipse, but I’m not going to jump ship just yet, I know there will be at least a half dozen niggling issues that will cause a slow down in the project I’m working on. I’ll just have to battle with eclipse until I’ve some time to move.. netbeans will be sat there, patiently waiting to stop me buying an intellj license each year for the rest of my java-coding-life.