Functional Programming is Fun(ctional Programming)

The ball keeps rolling. Microsoft has has F#, a functional language for the Common Language Runtime (CLR, the .NET virtual machine) for a while. Then there's LINQ, the functionally-inspired generic data query mechanism built into C# 3.0. Now Microsoft is involved with Haskell, one of the better-known purely functional languages of recent vintage.

One reason for the increasing interest in and support of functional programming languages is that they have some characteristics that are very beneficial when attempting to scale to a large number of cores. In particular, since all data is immutable, there are no locks. Also, unlike imperative languages (which includes Java, despite its object-orientedness), programs written in functional languages say what to do, not how to do it, which gives the compiler and runtime system freedom to, say, run parts of a loop on different cores. Anders Hejlsberg, one of the designers of C#, talks about that aspect of functional languages here (starting around 20:45).

For applications that need to run efficiently on many cores, the general migration path I see is from using threads explicitly, to using tasks (e.g. Java's Executor framework, where the application doesn't explicity create threads itself), to writing in a concurrency-oriented fashion, whether in a purely functional language using some hybrid approach. For the hybrid approach, here's a write up of some ways to do concurrency-oriented programming in Java.

(This post is based on an email I sent to my group at work. It's a Java shop.)

Hang 'Em High

For such a crisp thinker as Willem Buiter, these words, with which I agree, are extremely harsh:

Too many bank insiders have exploited their monopoly of information and the control it bestows on them, to enrich themselves by robbing their shareholders blind. There has been a spectacular failure of corporate governance. Boards have foresaken their fiduciary duties. Surely, even the liability insurance taken out by board members ought not to shelter those who are guilty of, at best, such willfull negligence and dereliction of duty? Where are the class actions suits by disgruntled shareholders? Where are the board members in handcuffs?

Now that there is no meat left on the shareholder drumstick, the rogue managers and employees are going after a piece of the really juicy bird - the ever-patient tax payer. I hope they choke on it.

Governance lies as the heart of the mess. It was broken before the house of cards collapsed, and it's broken now, too, because the people responsible for the status quo are either still in power, or not on trial, or both.

I'm Looking Forward to This

Be Honest But Mysterious

Your boss wants to know where the reports that you were supposed to have on his desk are? Be honest with him: "I don't have them." But, instead of full disclosure, leave a little to the imagination: "And I'm not sure why." If he presses, be nice but firm: "I don't feel comfortable giving them to you yet, but I'm flattered that you're looking for them." "Who is this wild creature who doesn't have the reports?" he'll be wondering. Sure, it will feel uncomfortable for you at first, but, trust us, it'll drive him crazy. He'll probably have to contact you every hour. You won't be able to get rid of him!

First Impressions of Structural Computing

A lot of research is computer science is sort of boilerplate. Like when a graduate student incrementally improves a known solution to a given problem in an established area of research. Even when it's hard, that's the easy stuff, it seems. The hard stuff involves creating a new area of research altogether. That's what Peter J. Nürnberg is up to with his idea of "structural computing." So far, a couple of pages into a couple of his papers, and really not being familiar with the hypermedia corpus, the first thing that comes to mind is a data structure in which every element is connected to every other element by way of common data structures -- e.g. array, list, hash table -- so as to enable on-the-fly views of the data to be created. And this suggests the need to transform one view into another view arbitrarily, which in turn reminds me of category theory, data provenance, and the stuff that Benjamin Pierce has been working on, but not necessarily in that order and not to imply a relationship among those things that doesn't exist.

What was Universal thinking?

I love The Big Lebowski as much as the next guy, but the marketing department at Universal Studios really went too far with the their 10th Anniversary Limited Edition DVD. Come on. A bowling ball as a DVD case? Why not just give us the two disks with the bonus materials, etc, etc, and be done with it? What value does the bowling ball add exactly? What's worse, there's not even a symbol on the bottom of the plastic ball indicating how to recycle it.

And on the occasion of my 39th birthday, a colleague sent me this:

Once a cuckoo told me it was my birthday and flew away.
I asked myself Have you no regard for yourself to celebrate your birthday?
Then the cuckoo came back and told my brain Birthdays are for fathers and mothers.

Yesterday I turned 39. No surprises or revelations, just a sense of what it's like to keep growing older, an understanding that the tired phrase life goes on is tired because it's true. Of course, one's sense of time fluctuates, expanding and contracting, depending on the intensity of experience at that moment, depending on how much the world is defying or agreeing with you, but in all cases, things keep happening. Somewhat like no matter where you go, there you are. Both phrases are trivially true in an all-too-universal way. Perhaps the thing that distinguishes the old from the young is the knowledge that the truth of these phrases isn't trivial, because having a sense of how they are true only comes with experience, and experience only comes with age, and age implies death.

Back when Todd Mytkowitz and I were looking into the feasibility of using Amazon Web Services to run a not-for-profit scientific compute service, Todd explained the idea of the service to some scientists at a conference at the Santa Fe Institute. The most surprising part of their response was that a pay-as-you-go model doesn't always fit well with the way researchers spend money, since grant money comes with an expiration date; the money must be spent within some window of time. Since it's easy to justify the purchase of computing equipment, research money sometimes goes into a new, albeit small, compute cluster. Utility compute services like AWS are metered; using them doesn't require a large initial capital outlay. So it's not possible -- or at least hasn't been possible -- to park your money there. I think a similar dynamic is at work with corporate managers and directors, who are the marketing target of Amazon's push into the enterprise: the pay-as-you-go model just doesn't fit well with the budgeting practices of large institutions. Hence, ladies and gentelmen, Amazon's new pricing model.

The scary thing is I almost actually understood this.

Better Gmail is Not Better

CPU consumption by Firefox 3.0.7 with:

No pages loaded 1-3%
Old version of Gmail 1-3%
New version of Gmail 4-5%
New version of Gmail (with Better Gmail 2) 73-85%

It turns out that TestNG has built-in support for the functionality I just implemented for our JUnit tests.

At work the time to execute all of our our growing set of JUnit tests has slowly crept up. In 2005 it took 25 minutes to build and test everything. Now it takes 65 minutes, most of which is spent running tests. So we decided to run the longest-running tests (defined at test classes which take > 30 seconds to execute) only once a week. So by default when one of these long-running test is run as part of a group of tests, it doesn't execute; to cause those long-running tests to run within Ant, you'd set a property to indicate this. However, we still wanted to run those tests in, say, Eclipse, because presumably someone who tries to run the test in Eclipse really wants it to run; one way to make this work is to check whether the value of the TERM environment variable is set to "dumb."

I created an annotation called IntegrationTest, which can only be applied to types, not methods. A nice enhancement could be to apply the annotation to methods, too, so that only certain long-running test methods are excluded or included as desired.

To mark test classes as long-running and I annotated each one like so:

@IntegrationTest
@RunWith(JUnit4ClassRunner.class)

The code for JUnit4ClassRunner is (approximately):

package com.example.build;

import java.util.logging.Level;
import java.util.logging.Logger;

import org.junit.runner.Description;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;

public final class JUnit4ClassRunner extends BlockJUnit4ClassRunner {
private static final String TEST_TYPE_KEY = "test.type";

private static final Logger log =
  Logger.getLogger(JUnit4ClassRunner.class.getName());

public JUnit4ClassRunner(Class<?> klass) throws InitializationError {
  super(klass);
}

@Override
protected void runChild(FrameworkMethod method, RunNotifier notifier) {
  if (shouldRun(method)) {
      super.runChild(method, notifier);
  } else {
      log.info("Not running integration test: " + method.getMethod());
      Description desc = Description.createTestDescription(
          method.getMethod().getDeclaringClass(), "Integration test");
      notifier.fireTestIgnored(desc);
  }
}

protected boolean shouldRun(FrameworkMethod method) {
  final TestType type = getTestType();

  switch (type) {
  case UNIT_ONLY:
      return !isIntegrationTest(method);
  case INTEGRATION_ONLY:
      return isIntegrationTest(method);
  case ALL:
      return true;
  default:
      return true;
  }
}

protected TestType getTestType() {
  if (isIntegrationOnly()) {
      return TestType.INTEGRATION_ONLY;
  } else if (isEclipse() || isAllTests()) {
      return TestType.ALL;
  } else {
      return TestType.UNIT_ONLY;
  }
}

/**
     * This is a bit of a hack, but seems to work.  Eclipse sets the TERM
     * environment variable to "dumb"; it's quite unlikely that a user would
     * be running tests from the command line with their TERM set to "dumb."
     * 
     * @return true if TERM is "dumb"
     */
protected boolean isEclipse() {
  String value = System.getenv("TERM") == null ?
          "" : System.getenv("TERM");
  return value.equals("dumb");
}

protected boolean isIntegrationOnly() {
  String value = System.getProperty(TEST_TYPE_KEY, "");
  return value.equals("integration");
}

protected boolean isAllTests() {
  String value = System.getProperty(TEST_TYPE_KEY, "");
  return value.equals("all");
}

/**
     * @param method The FrameworkMethod of some JUnit test method (i.e.
     * a method annotated with @Test)
     * @return true if the class in which the method is declared is annotated
     * with @IntegrationTest
     */
protected boolean isIntegrationTest(FrameworkMethod method) {
  Class<?> klass = method.getMethod().getDeclaringClass();
  return klass.isAnnotationPresent(IntegrationTest.class);
}
}

When I was in grad school, I cooked, but it usually wasn't very good. Hamburgers, mac and cheese, spaghetti, stuff like that. Since I finished grad school a few years ago, I haven't had to scrimp, so I haven't cooked much at all. For lunch I've usually gone to the burrito joint across the street. That changed this Christmas when I got a Cuisinart and a bunch of cook books and cooking gear. I've been making my own sorbets for a few weeks now. So far I've made batches of lime-basil, honeydew melon-mint, and raspberry-anise. And for lunch I've found two recipes that taste good reheated. Here's one of them, an Orange Sizhuan Chicken, before it goes into the fridge.


To top it off, eating this way is usually cheaper than going out, although I do have to be careful when I go to Whole Foods. Change is good.

There's an article over at TPMMuckraker that got me thinking. It's a bit of a digression, but bear with me.

There seem to be two understandings of "bureaucracy," one pejorative, one objective. In the pejorative version, bureaucracy is an organization -- typically governmental, like a regulatory agency, or that pot-bellied guy who works in the zoning department -- characterized by a terrifyingly blind, sometimes petty dedication to, say, filing all of the necessary paper work for your boat license. In the objective version, which I expect is more common amongst social scientists and the like, bureaucracy is an organization characterized by a formal structure, with rules guiding how it is supposed to go about accomplishing its mission.

I prefer the objective one; it's less driven by animus and, being more generic, it lends itself to broader application. It's more useful. With that definition, a corporation is a bureaucracy, which only makes sense: it's an organization with a blind, sometimes petty dedication to profit (if, of course, you assume that corporate governance is not completely broken, which it is). Or an army: an organization with a blind, petty dedication to killing (in the national interest).

There's a stereotype that government bureaucracies are extremely inefficient, but I think that's the wrong way to look at it. They are typically just as efficient at accomplishing the goals of the organization as corporations, the military, and other institutions. They just operate under different resource constraints. A well-run military given the right resources is an efficient killing machine. A well-run technology company given the right size (e.g. IBM) or right position in the technology life cycle (e.g. the early Cisco) is an efficient profit-making machine. And a well-run regulatory agency given the right authority is an efficient regulator of whatever it's regulating.

But like it or not, that's all they do. They don't know how to handle things that conflict with their mission; when that happens, people outside the bureaucracy may have to apply force to cause it to behave correctly. Depending on the degree of asymmetry of power between the outside group and the bureaucracy, something terrible might happen before the outside group succeeds, if they succeed at all.

So, with that in mind, please, dear reader, do remember that while bureaucracies serve a useful function within their own domain, they do little else. Banks, for example, do not love you. They're financial intermediaries, run for a profit, and they want your money; they don't give a damn whether your mother just died of cancer. Actually, I take that back, they do care if your mother just died of cancer, because that may make you more vulnerable and increase the chance that they can trick you into to giving them money to which they have no rightful claim. Bastards.

Global cooling is not, mmkay?

Looks like someone deployed a new feature too soon: Google is telling me that Google may be running malicious code.

It sure would be cool to be able to say that you devised a way to sequester carbon dioxide created in the production of cement; creating something that useful is a progressive engineer's dream.

What a meaningless exercise:

Yet one man’s flexibility is another man’s wishy-washiness, and Mr. Obama’s willingness to adapt carries the risk that he will either alienate his liberal base or fail to convert Republicans whose support he hopes to win. During his transition, Mr. Obama managed to charm conservatives; he wooed them at one dinner honoring Mr. McCain, and at another at the home of the columnist George F. Will.

But just days into the Obama presidency, some conservatives sound wary.

“I thought he did very well during the transition on things like the dinner with George Will, and all the words sounded good,” said Newt Gingrich, the Republican former speaker of the House. “But I think they are right at the cusp of either sliding down into a world where their words have no meaning or having to follow up their words with real behavior.”

There was a campaign, then an election, a transition period, and an inauguration. Unsurprisingly, the new administration is new. But note the melodramatic tone: "just days into Obama presidency," "the cusp of sliding down into a world where their words have no meaning." Sigh.

The early reviews of Amazon's CloudFront are in. The question is will Akamai do business differently now? Is Amazon going to eat Akamai's business from the bottom up, much like Windows NT and Linux did (in different ways) for the UNIX workstation market? Akamai's CDN includes real-time communications like VOIP and conferencing. I wonder if Amazon wants into that space too.