I’m preparing for my first global travel for global health, but the net is paying attention to a paper that I think I’ll like, and I want to mention it briefly before I fly.

Computational Complexity and Information Asymmetry in Financial Products is 27 pages of serious TCS, but it is so obviously applicable that people outside of our particular ivory tower, and even outside of academia entirely are blogging and twittering about it, and even reading it!

Freedom to Tinker has a nice summary of this paper, if you want to know what it’s about in a hurry.

Mike Trick makes the salient observation that NP-hard doesn’t mean computers can’t do it. But the assumption that this paper is based on is not about worst-case complexity; it is, as it should be, based on an assumption about the average-case complexity of a particular optimization problem over a particular distribution.

As it turns out, this is an average-case combinatorial optimization problem that I know and love, the densest subgraph problem. My plan is to repeat the problem here, and share some Python code for generating instances of it. Then, you, me, and everyone, can have a handy instance to try optimizing. I think that this problem is pretty hard, on average, but there is a lot more chance of making progress on an algorithm for it than for cracking the P versus NP nut.

First, the Densest subgraph problem (bottom of p. 5):

Fix M; N; D; m; n; and d to be some parameters. The (average case, decision) densest subgraph problem with these parameters is to distinguish between the following two distributions P and D on (M;N;D) graphs, where R is obtained by choosing for every top vertex D random neighbors on the bottom; and P is obtained by first choosing random hidden subsets S from [N] and T from [M] with |S| = n and |T| = m, and then choosing D random neighbors for every vertex outside of T, and D-d random neighbors for every vertex in T. We then choose d random additional neighbors in S for every vertex in T.

Then the Densest subgraph assumption (middle of p. 6) is:

Let (N; M; D; n; m; d) be such that N = o(MD), , then there is no and poly-time algorithm that distinguishes between R and P with advantage .

Or, to say the same thing in Python, with a little help from networkx:

import random
from networkx import Graph

def planted_dense_subgraph(M=1000, N=1000, D=500, m=25, n=25, d=15):
    """ Generate a bipartite graph with a planted dense subgraph
    (distribution P)

    Parameters
    ----------
    M, N, D, m, n, d : int, optional
      M and N are the sizes of the bipartitions and m and n are the
      size of the planted node sets.  D is the degree of the M-vertices
      and d is the number of edges from an m-vertex to n-vertices

    Output
    ------
    G : Graph
      A bipartite graph, with vertices T_1, ..., T_M and B_1, ..., B_M
    T_hidden, B_hidden : lists
      The vertex sets of size m and n that are hidden in the T and B
      vertices
    """

    T = ['T_%d'%i for i in range(M)]
    B = ['B_%d'%i for i in range(N)]

    T_hidden = random.sample(T, m)
    B_hidden = random.sample(B, n)

    G = Graph()
    G.add_nodes_from(T)
    G.add_nodes_from(B)

    for t in T:
        if t in T_hidden:
            G.add_star([t] + random.sample(B, D-d))
            G.add_star([t] + random.sample(B_hidden, d))
        else:
            G.add_star([t] + random.sample(B, D))

    return G, T_hidden, B_hidden

def random_graph(M=1000, N=1000, D=500):
    """ Generate a bipartite graph without a planted dense subgraph
    (distribution R)

    Parameters
    ----------
    M, N, D : int, optional

      M and N are the sizes of the bipartitions and D is the degree of
      the M-vertices

    Output
    ------
    G : Graph
      A bipartite graph, with vertices T_1, ..., T_M and B_1, ..., B_M
    """

    T = ['T_%d'%i for i in range(M)]
    B = ['B_%d'%i for i in range(N)]

    G = Graph()
    G.add_nodes_from(T)
    G.add_nodes_from(B)

    for t in T:
        G.add_star([t] + random.sample(B, D))

    return G

If I give you the graph produced by of one of these functions, you can’t tell me which function I used with any more accuracy than if you flip a coin to decide.

As the authors say, this is an assumption. It could be proven false by a clever algorithm tomorrow.

Wow, where does the day go? I spent all my non-meeting time debugging something. At least I fixed it before 5 PM.

The details of the problem are boring, but the whole ordeal could have been avoided if I had just followed the two rules of optimizing software in my Generic Disease Modeling System. What are they?

First Rule of Program Optimization: Don’t do it
Second Rule of Program Optimization (for experts only!): Don’t do it yet

Maybe next week I’ll get a second to write about the good kind of optimization; my statistical physics friends have posted an article on the arxiv which I am a co-author on, about an application of bounded-depth minimum spanning trees, Clustering with Shallow Trees.

This weekend marks the submission of my first “Global Health” paper. Congratulations to me! And many, many thanks to all the people who have worked with me to make it happen. I’ll go into details sometime in the future, first let me see how things go in the refereeing process.

While I was over-working on that business, I got an interesting Call-for-Papers forwarded from global health/AI researcher Emma Brunskill. The AAAI Spring Symposium on Artificial Intelligence for Development (AI-D) is an effort to build a community of people applying computer science and artificial intelligence in less-developed settings.

TCS people, don’t let the “AI” in their title turn you off. Eric Horvitz says that this is for all of us.

From the CfP:

There has been great interest in information and communication technology for development (ICT-D) over the last several years. The work is diverse and extends from information technologies that provide infrastructure for micropayments to techniques for monitoring and enhancing the cultivation of crops. While efforts in ICT-D have been interdisciplinary, ICT-D has largely overlooked opportunities for harnessing machine learning and reasoning to create new kinds of services, and to serve a role in analyses of data that may provide insights about socioeconomic development for disadvantaged populations. The unprecedented volume of data currently being generated in the developing world on human health, movement, communication, and financial transactions provides new opportunities for applying machine learning methods to development efforts, however. Our aim is to foster the creation of a subfield of ICT-D, which we refer to as artificial intelligence for development (AI-D), to harness these opportunities.

It’s great to see Computer Science trying to address the social issues of our time.

I don’t feel like having that post about how big things are brewing in US health care reform on the top of my blog anymore, so here is a quick replacement: a ranking paper that caught my eye recently on arxiv, where computer scientists is applied to politics: On Ranking Senators By Their Votes, by my fellow CMU alum, Mugizi Rwebangira (@rweba on twitter).

Whoops, I got busy again and didn’t have time to make new pictures of TFR vs HDI for Rif and Tanja, let alone fix the Bayes factor estimation code or implement the nested sampling version (which I think will be the cool way to estimate evidence). But coming soon: How MCMC is tying my new work in Health Metrics to my education in Operations Research. That will be in two weeks, at best.

Until then, here is some light reading to get ready for a big week of US healthcare reform debate: Get Sick, Get Out, a survey conducted by lawyers interested in catastrophic medical payments and their connection to housing forclosures. It’s 40 pages long, but it’s in legal-journal format, where they have like 10 words per page if you skip the footnotes. From the abstract:

Half of all respondents (49%) indicated that their foreclosure was caused in part by a medical problem, including illness or injuries (32%), unmanageable medical bills (23%), lost work due to a medical problem (27%), or caring for sick family members (14%).

I’m excited for the next week of healthcare reform debates. When my most jaded friends are forwarding me Moveon.org videos (and I’m listening to 4 minutes of recent REM), I know something unusual is going on.

Happy labor day weekend!

(Updated 9/2/2009)

I never took a statistics class, so I only know the kind of statistics you learn on the street. But now that I’m in global health research, I’ve been doing a lot of on-the-job learning. This post is about something I’ve been reading about recently, how to decide if a simple statistical model is sufficient or if the data demands a more complicated one. To keep the matter concrete (and controversial) I’ll focus on a claim from a recent paper in Nature that my colleague, Haidong Wang, choose for our IHME journal club last week: Advances in development reverse fertility declines. The title of this short letter boldly claims a causal link between total fertility rate (an instantaneous measure of how many babies a population is making) and the human development index (a composite measure of how “developed” a country is, on a scale of 0 to 1). Exhibit A in their case is the following figure:

An astute observer of this chart might ask, “what’s up with the scales on those axes?” But this post is not about the visual display of quantitative information. It is about deciding if the data has a piecewise linear relationship that Myrskyla et al claim, and doing it in a Bayesian framework with Python and PyMC. But let’s start with a figure where the axes have a familiar linear scale!

I was able to produce this alternative plot without working much because the authors and the journal made their dataset easily available on the web as a csv file. Thanks, all! In fact, they have provided more data than 1975 and 2005. Here is everything from the years in between as well:

Since we discussed this in journal club, I can now repeat my fellow post-docs’ observations and sound smart. In fact, I already did with that bit about the axes being on a non-linear (and non-logarithmic) scale. The letter claims that there is a breakpoint around human development index .86, and the data is piecewise linear with decreasing slope below this value, and increasing above. The next observation that is not mine originally, is that the data also looks like a quadratic would fit it pretty well.

To be slightly more formal, here are two alternative models for the association between HDI and TFR:

and

How should I decide between these two models? Well, I’m open to suggestions, but I learned about one way when I read a citation classic by UW statistician Adrian Raftery and co-author about Bayes Factors. They describe the ratio . If K is more than 3.2, this constitutes “substantial” evidence that model 2 is superior.

Since I’ve titled this post in the form of a tutorial, I’m now going to go through calculating the Bayes factor with MCMC in Python, which turns out to be a slightly challenging computation (but easy to code; thanks, PyMC!).

To set up the models in the fully Bayesian way, we need some priors, which I’ve made up below in a convenient, reusable form:

class linear:
    def __init__(self, X, Y, order=1, mu_beta=None, sigma_beta=1., mu_sigma=1.):
        if mu_beta == None:
            mu_beta = zeros(order+1)

        self.beta = Normal('beta', mu_beta, sigma_beta**-2)
        self.sigma = Gamma('standard error', 1., 1./mu_sigma)

        @potential
        def data_potential(beta=self.beta, sigma=self.sigma,
                           X=X, Y=Y):
            mu = self.predict(beta, X)
            return normal_like(Y, mu, 1 / sigma**2)
        self.data_potential = data_potential

    def predict(self, beta, X):
        return polyval(beta, X)

    def logp(self):
        logp = []
        for beta_val, sigma_val in zip(self.beta.trace(), self.sigma.trace()):
            self.beta.value = beta_val
            self.sigma.value = sigma_val
            logp.append(Model(self).logp)
        return array(logp)

and

class piecewise_linear:
    def __init__(self, X, Y, breakpoint=.86,
                 mu_beta=[0., 0., 0.], sigma_beta=1., mu_sigma=1.):
        self.breakpoint=breakpoint
        self.beta = Normal('beta', mu_beta, sigma_beta**-2)
        self.sigma = Gamma('standard error', 1., 1./mu_sigma)

        @potential
        def data_potential(beta=self.beta, sigma=self.sigma,
                           X=X, Y=Y, breakpoint=self.breakpoint):
            mu = self.predict(beta, breakpoint, X)
            return normal_like(Y, mu, 1 / sigma**2)
        self.data_potential = data_potential

    def predict(self, beta, breakpoint, X):
            very_high_dev_indicator = X >= breakpoint
            mu = (beta[0] + beta[1]*(X-breakpoint)) * (1 - very_high_dev_indicator)
            mu += (beta[0] + beta[2]*(X-breakpoint)) * very_high_dev_indicator
            return mu

    def logp(self, beta_val=None, sigma_val=None, breakpoint_val=None):
        for beta_val, sigma_val in zip(self.beta.trace(), self.sigma.trace()):
            self.beta.value = beta_val
            self.sigma.value = sigma_val
            logp.append(Model(self).logp)
    return array(logp)

Then to try to calculate the Bayes factor, I can draw samples from the posterior distribution of each model, and look at the harmonic mean of their posterior liklihoods.

def bayes_factor(m1, m2, iter=1e6, burn=25000, thin=10, verbose=0):
    MCMC(m1).sample(iter, burn, thin, verbose=verbose)
    logp1 = m1.logp()

    MCMC(m2).sample(iter, burn, thin, verbose=verbose)
    logp2 = m2.logp()

    mu_logp = mean(logp2)
    K = exp(pymc.flib.logsum(-logp1) - log(len(logp1))
            - (pymc.flib.logsum(-logp2) - log(len(logp2))))

    return K

Unfortunately, it seems to take a prohibitively large number of samples to get the same average twice.

To make it all run, I’ve made a little module to load the data, but I won’t bore you with the details; it’s online here if you want to play around with it yourself.

>>> import data
>>> m1=models.linear(X=data.hdi, Y=data.tfr, order=2)
>>> m2=models.piecewise_linear(X=data.hdi, Y=data.tfr, mu_beta=[1,-10,1])
>>> model_selection.bayes_factor(m1, m2, iter=1e7)

According to the Bayes factor, the piecewise linear model is (to be filled in soon) better than the quadratic model. Or, more quantitatively, the observed data is (tba) times more likely under model 2 than model 1. Cool!

As a side-effect, this yields an alternative way to ask if the association on the “high development” piece of the piecewise model is really positive:

>>> m2 = models.piecewise_linear(X=data.hdi, Y=data.tfr, breakpoint=.86, mu_beta=[1,-8,1], sigma_beta=1., mu_sigma=.1)
>>> MCMC(m2).sample(iter=1000*1000+20000, thin=1000, burn=20000, verbose=1)
>>> m2.beta.stats()['95% HPD interval']
array([[ 1.91520875,  2.01669476],
       [-9.84036014, -9.48869047],
       [-3.85410676, -1.23869569]])
>>> m2.beta.stats()['mean']
array([ 1.96932651, -9.66349812, -2.5683748 ])

The research questions for the computer scientist in me are: did I draw enough samples to get a correct answer? and did I really need to draw that many?

The tentative answers are no and no! See the comments for leads on more efficient schemes.

(Tap… tap… tap… is this thing on? Good.)

July was vacation month, where I went on a glorious bike tour of the Oregon/California coast, and learned definitively that I don’t like biking on the side of a highway all day. Don’t worry, I escaped in Coos Bay and took trains and buses between Eugene, Santa Cruz, Berkeley, and SF for a vacation more my speed.

But now that I’m back, August is turning out to be project month. I have 3 great TCS applications to global health in the pipeline, and I have big plans to tell you about them soon. But one mixed blessing about these applications is that people actually want to see the results, like, yesterday! So first I have to deal with the results, and then I can write papers and blogs about the techniques.

Since Project Month is a little over-booked with projects, I’m going to have to triage one today. You’ve heard of the NetFlix Challenge, right? Well, github.com is running a smaller scale recommendation contest, and I was messing around with personal page rank, which seems like a fine approach for recommending code repositories to hackers. I haven’t got it working very well (best results, 15% of holdout set recovered), but I was having fun with it. Maybe someone else will take it up, let me know if you get it to work; networkx + data = good times.

    f = open('download/data.txt')
    for l in f:
        u_id, r_id = l.strip().split(':')
        G.add_edge(user(u_id), repo(r_id))

[get the code]

I’ve been spending the week at the Infectious Disease Modeling Summer School here at UW. It’s very interesting, and good for me to learn more about how people in my new field think (especially people in my new field, outside of my little institute…)

I’ve discovered a pet peeve during this week of presentations, though. I’ve seen a lot of numerical examples where the numbers work out perfectly… a little too perfectly. If you split 1000 people into an experimental and control group by choosing a random subset of 500, fine. But if you look within that group to see how many have a trait that occurs independently with probability 0.2, you do not often find exactly 100 in group A and 100 in group B. I think a little more complexity in the numbers makes the example easier to understand.

I’m sure that you, my loyal reader, can generate random numbers from a multitude of distributions, if you wanted to spend the time. But if you’re busy, busy, busy, then you can have wolfram alpha do all the work. It actually comes through for that one: “sample Binomial(500, .2)“.

I wrote two months ago about the mysterious differences in health care costs that I found so intriguing in a talk by Jonathan Skinner. (That was two months ago? Really?) Since then, the surgeon/author Atul Gawande has brought the mystery to the national stage. In a long story for the New Yorker, he gave the non-technical version of Skinner’s talk, and today he addressed some of the feedback that this article has received over the last month.

His short answer to the mystery is this:

Analysis of Medicare data by the Dartmouth Atlas project shows the difference is due to marked differences in the amount of care ordered for patients—patients in McAllen receive vastly more diagnostic tests, hospital admissions, operations, specialist visits, and home nursing care than in El Paso.

But that is not the end of the story. It only takes a sentence to explain the “proximal” cause of these cost differences, but it takes the whole article for Gawande to do justice to his theory on the underlying cause, and his is certainly not the only theory.

Since his theory of the root cause of this inequality is centered on physicians putting profit over patients, it has made some doctors uneasy. Greg Roth, a physician that I work with hadn’t had time to read the article when we last chatted, but he did attend Skinner’s talk with me two months ago. Greg told me about a detail that has emerged as doctors put Gawande’s article under their microscopes: we might be making a mountain out of molehill-sized mystery.

Look at this plot, which shows the complementary cumulative distribution function for the primary quantity in Gawande’s article, Total Medicare reimbursements per enrollee for 2006.

Investigative reporter have to get the story, and raking the muck way out in the tail of this distribution turned out to be a good bet this time. But McAllen is 6 standard deviations above the mean (not to imply that this distribution is normal… should it be?) How much impact would it have, for the whole population, if the outliers were greatly improved?

If through anti-fraud policing, better culture, and general hard work, the top 10% of hospitals reduced their cost per patient to the national average, that would reduce the average cost by 3.6%. Outliers show what is possible, but making a big change involves more than outliers.

The political situation in Iran has been in the news and on the nets a lot this week. I hope that the friends and families of all my Iranian colleagues are safe. I’m thinking of you.

I thought Obama’s statement was pretty astute, when he said,

It’s not productive, given the history of US-Iranian relations, to be seen as meddling, the US President meddling in Iranian elections.

I remember learning from some Iranian computer scientist that his impression of the popular narrative of how the Shah returned to power in the 1950s differed from my impression in some critical details.

By coincidence, this week also sees the preliminary publication of the Iran National Burden of Disease 2003 Study. This is a national-sized study of the big project that I have been working on for the last many months. I plan to write more about the global version soon, including especially the challenges for theoretical computer science that you might want to be working on.