Gray Calhoun's academic homepage

Expectations for “econometrics” PhD students

Wed, 13 Apr 2016 00:00:00 +0000

A list of expectations for students planning to do a field in Econometrics.

LaTeX CV template

Thu, 09 Jul 2015 00:00:00 +0000

I recently read Karen Kelsky’s (“Dr. Karen’s”) excellent academic CV rules and tried to follow her advice and revise my CV. It seems like other people could find it useful too, so I wrote up most of the changes as a LaTeX document class (the safecv document class) and put it on GitHub.

The repository gives you a document class that sets up formatting for section headers, etc. and also gives you two empty-ish templates to use for your own CV. One is tailored a bit towards grad students entering the academic job market for the first time, and one is more generic. I’m not a graphic designer, and you’re probably not either, so the CV uses Times New Roman. It is the default, conservative, “professional” font for those of us who predate Office 2007, which is essentially everyone who will be in a position to hire you. (Since I have no impulse control, I took this opportunity to buy the Lucida TeX fonts and could not be happier with how they look. But my wife still insists that my CV looks better in Times.)

Is it a great CV template? Nope. Is it better than most of the other CV templates I’ve seen? Yep. Almost entirely because, as Dr. Karen suggests, all of the important dates are flush left and the text is relentlessly left-aligned. Most of the other CV templates I’ve seen are much slicker looking, but they put key information in different random spots in different sections of the document. Don’t do that.

Please open an issue on GitHub if you find errors. Pull requests are welcome too. And read Dr. Karen’s advice most of all.

Information about PhD Econometrics this Fall at Iowa State

Wed, 20 May 2015 00:00:00 +0000

(This is the text of an email I sent to incoming PhD students for Iowa State. Maybe it’s useful for other people too.)

Hi everyone,

My name’s Gray Calhoun and I’m going to be teaching your Econometrics class at Iowa State this fall (Econ 671). There is a preliminary version of the syllabus on the course homepage, http://gray.clhn.org/671, but it may be changed slightly before the semester starts. This email has a little more information about our expectations and some advice on preparing for the class.

First, this class is going to introduce you to statistical programming, and we’re going to use the language R for all of the assignments. If you have not programmed in R before, you may want to start to familiarize yourself with it over the summer. Fortunately, it’s free. You can download R from http://cran.r-project.org (or using your OS’s package manager). You should also install RStudio, which can be downloaded for free from http://www.rstudio.com, and use it as the editor/IDE for R. (It is much better than using R directly, and we’ll use RStudio in class.)

Then, to actually become familiar with R, you may want to work through a short tutorial. Two that are recommended are:

Try R, http://tryr.codeschool.com (it works in your web browser)
Swirl, http://swirlstats.com (through the R console).

There is also documentation for RStudio on its homepage.

Second, if you haven’t taken a statistics or econometrics class before, or if you last took one a few years ago, you’ll benefit from reviewing some basic undergraduate textbooks. I recommend the following:

Freedman, Pisani, and Purves’s “Statistics.” It’s currently on the 4th edition, but you can buy an older edition. (Or get a copy from the library, you’re probably not going to need it as a reference.) This is a very, very basic stats 101 level treatment. E.g. it doesn’t really get more advanced than the t-test and you can probably read it in a week. But it will give you a good introduction to probability and statistics if you’ve never seen them before.
Tufte’s “Data Analysis for Politics and Policy.” This is a very simple introduction to linear regression. It doesn’t get into the math very much, but it is captures the intuition pretty well. It’s available for $2 from Tufte’s website, http://www.edwardtufte.com/tufte/ebooks. Again, this is the sort of book you can read in about a week and it will give you a decent introduction to the second half of our class.

Reading through both of those books will give you an overview of what we’re going to do in the class and shouldn’t take that long. So please try to do it if you haven’t seen this material before. Do keep in mind that these books are not representative of the mathematical level of the course.

If you’ve already seen this material, and have free time this summer that you’d like to spend preparing for the econometrics sequence, please feel free to email me and I will be happy to provide more suggestions. Working through the recommended text, Freedman’s “Statistical Models, Theory and Practice,” and doing the book’s exercises in R would be very good preparation, but I can give other recommendations too. :)

In any case, I’m very happy you’ll be attending Iowa State and I look forward to meeting you in August. Enjoy the summer!

A visual analysis of some macroeconomic series

Mon, 10 Nov 2014 00:00:00 +0000

Slides for a presentation to the Statistical Graphics Working Group in the Fall, 2014.

Short macros and utility functions for Julia

Mon, 20 Oct 2014 00:00:00 +0000

Link to a GitHub Gist with some short examples of Julia macros. Please add to it!

Preliminary notes on Julia’s macros

Mon, 13 Oct 2014 00:00:00 +0000

First things first, I’m entirely unqualified to write a review, or even an email or a tweet, discussing macros in Julia. I don’t know anything. That said, I’m scheduled to give a 50 minute talk on Wednesday about macros in Julia, so I need to do some research and organize my thoughts. And you, dear reader, are reading the result of that research and the mechanism for organizing my thoughts. So be skeptical of everything you read here: it’s just my opinion, and my opinion isn’t especially well-informed. A better resource is Julia’s documentation on metaprogramming.

What are macros? How are they different than functions?

Julia’s functions, like functions in most languages, take a collection of arguments, run some instructions on them, and return a value. Julia’s macros, on the other hand, take a collection of expressions, run some transformations on them, and return Julia code. That new code is run in place of the original macro.

Now, what does that mean? I’ll go back to my first love in Julia, Dahua Lin’s @devec macro. Let x and y be numeric vectors that have the same length and suppose that we want to do some simple math on their elements and store the results in a new vector r. Like this:

r = exp(-abs(x - y))                      # basic version

You might disagree, but this is clearly the way we want to write these operations: our intent is unambiguous. But this isn’t the way we want the computer to execute these operations. As Dahua points out, it’s slow and uses more memory than necessary, because each operation is executed sequentially: first x-y is calculated and stored in a temporary array (call it temp1); then abs(temp1) is executed and stored in an array we can call temp2; then -temp2 is executed and stored in an array temp3; and finally exp(temp3) is executed and stored in r. Each intermediate step traverses its input, which takes time, and stores its output in a new array, which takes memory.

Dahua argues that we’d like to run the code

r = similar(x)                                # version 2
for i = 1:length(x)
    r[i] = exp(-abs(x[i] - y[i]))
end

which is faster and more memory efficient. The operations still create temporary variables, but those variables are now scalars and not vectors, and we only traverse the array once. Potentially noticable gains.

I’d go even further. The first line of that code frightens me, because x can be a vector of integers but we’d still exect that r would be floats (and, indeed, when you run this code with x as an integer, it fails with an error). Second, x and y can have different lengths. The code as written implicitly checks that x[i] and y[i] are legitimate indexes, but it does it at each step of the loop. It would be faster to check before the loop starts, and turn off checking inside the loop. (There’s another macro, @inbounds, that turns it off.) So I’d prefer to use the code

@assert length(x) == length(y)                # version 3
firstval = exp(-abs(x[1] - y[1]))
if length(x) > 1
    r = similar(x, typeof(firstval))
    r[1] = firstval
    @inbounds for i = 2:length(x)
        r[i] = exp(-abs(x[i] - y[i]))
    end
else
    r = firstval
end

This code does the following:

Checks that x and y have the same length.
Calculates the first element of exp(-abs(x - y)) so that we can learn its type.
If x and y only have one element, set r equal to firstval and stop. Otherwise, allocate an array for r, store firstval as its first element, and then iterate our calculations down the rest of x and y, assuming that the indexes are valid for each i.

Running all three options on my laptop with 80,000,000 elements gives the timings listed in the table below (I wrapped the code in trivial functions for the timings; this Gist has executable code.)

Version	Elapsed time (sec)	Memory allocated (MB)
1	7.21	2,560
2	5.03	640
3	4.84	640

We obviously want to run the last version of the code. (The second version is essentially as efficient, but will throw an error if x is an integer.) But, just as obviously, the last version of the code is hideous; so much so that I felt like it needed external documentation! So we clearly want to write the first version, but execute the second version.

Baby steps towards writing a simple devectorization macro

This is my first macro. Fun!

I’ll walk through the steps I took. First, as recommended in Paul Graham’s On Lisp, we’ll write out a representative call to the macro and the expression that we actually want evaluated. And let’s start with a baby version of the expression first:

@ourdevec r = exp(-abs(x - y))     # representative call
firstval = exp(-abs(x[1] - y[1]))  # baby expression goal

So now we want to write code that converts the first expression to the second. Julia expressions are represented as Expr objects and they have a head and args field. (This is being revisted and eventually may change, though.) Here, we have

julia> :(r = exp(-abs(x - y))).head
:(=)                                     # value returned

julia> :(r = exp(-abs(x - y))).args
2-element Array{Any,1}:                  # value returned
 :r
 :(exp(-(abs(x - y))))

And looking at the type of each of the argumets gives

julia> map(typeof, :(r = exp(-abs(x - y))).args)
2-element Array{Any,1}:                  # value returned
 Symbol
 Expr

So the left side is just a Symbol (the variable type Julia uses for variable names) and the right side is another Expr. We could keep on going sub-expression by sub-expression, but fortunately there are several functions that present everything at once for us:

julia> Base.Meta.show_sexpr(:(r = exp(-abs(x - y))))
(:(=), :r, (:call, :exp, (:call, :-, (:call, :abs, (:call, :-, :x, :y)))))

which tells us that each sub-expression is a function call, until we finally get to the symbols :x and :y. It also shows us that the first element of args for a function call is the symbol that represents the function, and the rest of the elements are the expressions or symbols passed to that function as arguments.

To produce our baby expression, we want to replace :x and :y in our original expression with :(x[1]) and :(y[1]). More generally, we want to take every symbol that’s not a function, and replace it with a reference to its first element. (Important realization: what if we want to mix in scalars? Then this might not work! Oh crap! Let’s disregard that problem for now and worry about it tomorrow. This post is hard enough as it is.)

This replacement is something we can do pretty easily with recursion, since :(x[1]) is just another type of expression (a :ref). We’ll define two functions that help:

getindex(s::Symbol, i) = Expr(:ref, s, i)
getindex(e::Expr, i) = Expr(e.head, e.args[1],
              [getindex(a, i) for a in e.args[2:end]]...)

getindex(:x, 1) returns :(x[1])
getindex(:(x - y), 1) calls both getindex(:x, 1) and getindex(:y, 1); takes the result; and puts them together in the original expression.

getindex already exists and is the function that’s called when you write x[i] for arrays, so it seems reasonable to extend its methods like this.

The baby macro is dead simple now

macro our_devec_p1(e)                # Ver 1 (baby macro)
    :($(e.args[1]) = $(getindex(e.args[2], 1)))
end

and we can verify that it “does the right thing” by using macroexpand.

julia> macroexpand(:(@our_devec_p1 r = exp(-abs(x - y))))
:(_38_firstval = exp(-(abs(x[1] - y[1]))))  # Value returned

A note on what happened to firstval: since macros write out code that’s executed in-place, there’s a chance that it could clobber variables that already exist. Macros, unlike functions, don’t define their own scope. To avoid overwriting existing variables, variables in macros are given unique names. (The function that does this is gensym.) The _38_ represents the unique identifier added to firstval to make it unique. (The _ really displays as # and screws up our code highlighting; I’ve manually replaced the symbol.)

Two more tiny steps towards a devectorization macro

If we look back at the expanded code, we can see three distinct parts:

Evaluate the vectorized expression for the first element. We’ve already done this part.
Check that the vectors have the same length, and check that x has more than one element.
Loop over the elements of x and y and assign the results of each calculation to r.

This parallels our initial description, but now we’re going to implement the second and third parts in the macro.

First add the length check. Setting up the problem just like we did for the baby macro earlier, we start with the expression

r = exp(-abs(x - y))

and we need to produce

length(x) == length(y)

We can write similar functions as before (which is a sign that we might want to think about generalizing this as utility code, but nevermind that for now…)

getvecs(s::Symbol) = s
getvecs(e::Expr) = [[getvecs(a) for a in e.args[2:end]]...]

which will recurse down a nested expression until it finds the symbols at the bottom, then return a single array that lists them all. (Putting ... at the end of the Expr method flattens the results.)

It’s clumsy, but we can assemble the expression we need by hand, as the next macro shows.

macro our_devec_p2(e)
  vecs = getvecs(e.args[2])
  check_lengths = Expr(:comparison)
  check_lengths.args = Array(Any, 2 * length(vecs) - 1)
  check_lengths.args[1:2:end] = map(s -> Expr(:call, :length, s), vecs)
  check_lengths.args[2:2:end] = :(==)
  :($check_lengths)
end

(I’m sure that there’s a slick, functional way to assemble the check_lengths expression, but I don’t know it.) Using macroexpand to verify that we’re on the right track gives

julia> macroexpand(:(@our_devec_p2 r = exp(-abs(x - y))))
:(length(x) == length(y))                # value returned

And now the loop. Again, the expression we start with is

r = exp(-abs(x - y))

and now the expression we want to return is

if length(x) > 1
    r = similar(x, typeof(firstval))
    r[1] = firstval
    for i = 2:length(x)
        r[i] = exp(-abs(x[i] - y[i]))
    end
end ### Ignore the `else` part for now

The body of the loop is almost what we did before when we calculated the first element, but now we don’t want gensym to create a new variable name, we want to clobber r. Julia, forntunately, offers a mechansim for “unhygenic” macros through the function, esc, which works as follows

macro our_devec_part3(e)
    vecs = getvecs(e.args[2])
    quote
        if length($(vecs[1])) > 1
            $(esc(e.args[1])) = similar($(vecs[1]), typeof(firstval))
            $(esc(e.args[1]))[1] = firstval
            for i = 2:length($(vecs[1]))
                $(esc(e.args[1]))[i] = $(getindex(e.args[2], :i))
            end
        end
    end
end

You should expand this part on your own to verify that it works. (It does, though.)

Finishing the macro

We can put the three steps together for our final devectorization macro:

macro our_devec(e)                        # Final version
    vecs = getvecs(e.args[2])

    check_lengths = Expr(:comparison)            # Part 2
    check_lengths.args = Array(Any, 2 * length(vecs) - 1)
    check_lengths.args[1:2:end] = map(s -> Expr(:call, :length, s), vecs)
    check_lengths.args[2:2:end] = :(==)

    quote
        @assert $check_lengths
        firstval = $(getindex(e.args[2], 1))     # Part 1
        if length($(vecs[1])) > 1                # Part 3
            $(esc(e.args[1])) = similar($(vecs[1]), typeof(firstval))
            $(esc(e.args[1]))[1] = firstval
            @inbounds for i = 2:length($(vecs[1]))
                $(esc(e.args[1]))[i] = $(getindex(e.args[2], :i))
            end
        else
            $(esc(e.args[1])) = firstval   # new but easy
        end
    end
end

And one final expansion to verify that it does what we’d hoped:

julia> macroexpand(:(@our_devec r = exp(-abs(x - y))))
quote  # none, line 10:
    if length(x) == length(y)
        nothing
    else
        Base.error("assertion failed: length(x) == length(y)")
    end # line 11:
    _51_firstval = exp(-(abs(x[1] - y[1]))) # line 12:
    if length(x) > 1 # line 13:
        r = similar(x,typeof(_51_firstval)) # line 14:
        r[1] = _51_firstval # line 15:
        begin
            $(Expr(:boundscheck, false))
            begin
                for _52_i = 2:length(x) # line 16:
                    r[_52_i] = exp(-(abs(x[_52_i] - y[_52_i])))
                end
                $(Expr(:boundscheck, :(Base.pop)))
            end
        end
    else  # line 19:
        r = _51_firstval
    end
end

(I was happily surprised that the index of the for loop is taken care of automatically.)

So now the code

@our_devec r = exp(-abs(x - y))

runs as fast and with the same memory allocation as the explicitly devectorized code from before. You can download all of the code for this post here. (Obvious caveats are that the code is probably pretty fragile and you’d probably need to make it more robust if you use it for real.)

Are macros just about performance?

The devectorize macro we defined is representative of a class of macros: it takes Julia code that would run fine otherwise, and it rewrites the code to use the computer more efficiently. (The Julia manual calls these “Performance Annotations”). Some other macros like this are

@inbounds, which we’ve seen already
@parallel, which can split loops across multiple processors to be run in parallel
@devec (the real version) which does more general devectorization.
@simd, which enables some sort of CPU-level devectorization that I don’t understand but looks impressive

I’m sure there are a few others, but these are, as far as I can tell, unrepresentative. Most macros are not written just to translate blocks of Julia code into blocks of faster Julia code. Dahua’s blog post was the first thing I read that got me really excited about Julia, though, so if I’m going to talk about macros…. But the main use of macros, at least historically (and “historically” here means “in Lisp, as far as I can tell.” Julia’s not old enough for any usage to be “historic”) is extending the language by adding new features.

First, a small example of language extensions. Perl, if you haven’t used it, is a programming language that excels at text manipulation. It does other things well too, and it was probably as important to web programming during the dot-com bubble as JavaScript has been for the whole Ajax/Web 2.0 [bubble]. One of the reasons why Perl is great at text manipulations is because it has excellent Regular Expressions (“regexes”), giving it an entire mini-language for finding and replacing patterns in strings.

Perl’s regexes are built into the language—large parts of the language are built around them—so they’re fast and immediately available. Julia has regexes as well, but they’re implemented as a macro. (My understanding is that this is how Common Lisp implements them as well.) Since these macro calls are translated before any code is actually run, the regex is parsed once, in advance, and they’re fast too.

So both languages have fast regexes. But the implementation methods are very different and the difference matters when you or I want to add a feature like regexes to the language; say (hypothetically) a regression modeling language like R’s, or a different text-processing minilanguage. With Perl, new features will never have the same performance or support that the built-in features do. With Julia, your macros are on equal footing with the native ones, so these new features have exactly the same support and potential for performance as the “built in” regex.

For flashier examples, there are at least two packages that support pattern matching through the macro system. (Match.jl and PatternDispatch.jl; you’ll have to look at the packages to really understand what they enable, but these are like powerful switch statements.) And support for DocStrings is (finally) being added to Base Julia, as a macro that was originally developed in a separate package (Docile.jl.)

Alternatives to macros

Another question: could we have written @our_devec as a function? Can we write macros as functions in general? Julia functions can take expressions as arguments and can return unevaluated expressions, and Julia provides an eval function that evaluates expressions on the fly. Specifically, would

eval(our_devec(:(r = exp(-abs(x - y)))))

do the same thing as

@our_devec r = exp(-abs(x - y))

As implemented in Julia, the answer is clearly “no.” eval has weird scoping issues: it evaluates expressions in the global scope, so if it’s called inside a function it wouldn’t have access to x and y. (I’ve read several places that this is for performance.)

But, even if it’s not the way Julia implements it, we can certainly imagine a “local eval” function. And we can certainly imagine people who don’t care about performance. So, without those constraints, do macros provide anything that we can’t get from evaluating expressions returned from functions?

I don’t think so, but I could be wrong. One reason is an R-News article by Thomas Lumley (2001), “Overcoming R’s Virtues,” that describes how to implement Lisp-style macros through R’s functions. (Unlike Julia, R lets functions evaluate expressions in essentially whatever environment you want.) So, from a conceptual standpoint, macros are just one mechanism for transforming expressions and choosing where to evaluate them, and there are potentially many other conceptually equivalent ways to do that.

“Conceptual equivalence” and “practical equivalence” are very different things, of course. The hypothetical eval version of macro execution would have to parse the expression on the fly each time it’s called, which we’d expect would give it a performance penalty compared to a “native” implementation and is going to make them less appealing in real applications. One of the driving principles in Julia’s development has been the idea that user-added functionality should be just as good as built-in functionality. This was part of the reason that Julia’s numeric types are implemented in Julia, and not as specialized C code, and it’s part of the reason that macros are used to implement fundamental objects like regexes: putting them on equal footing forces the language to support all of these extensions really well.

And obviously, performance-oriented macros like @our_devec are off the table without a real macro system.

TBL talk on open-ended team activities

Fri, 03 Oct 2014 00:00:00 +0000

Some short notes I presented to the Iowa State “TBL” meeting; it went over one of the team activities I’ve done with my econometrics class. I got lots of good feedback and suggestions. I presented the contents of this outline, but as slides.

Background on the class I teach

Started TBL this semester
Teach PhD Econometrics (basically a statistics class)
- Core required class in the Econ department
- Mostly Economics PhD students (one or two students from other departments)
- 19 students this year (fairly typical class size)
3 teams with 6 to 7 students
In first lecture, I sold TBL as a way for the students to get experience with “open ended and poorly defined problems,” which will help them in their research

Example of one team activity

Some comments

There is no “right answer” but there are many, many “wrong answers”
Potential for an answer being /wrong/ seems more important for team’s internal discussion than whether or not it can be /right/
I’m having trouble getting much discussion between teams, though
- If teams all choose wildly different approaches, they don’t have a lot to say to each other
This might be addressed with better homework & preparation
- For HW, students come up with different potential approaches
- Decide on a list of allowed approaches early in the class period
- Teams are required to choose and defend one of those options

Last comments

Don’t expect students to magically figure out the “right way” to handle challenging problems
“Problem solving” needs to be taught as well
Early on, you may want to split a problem into several small steps and lead the students through it
Or, make “planning” something that the students are required to do and are graded on before “acting”

Cobbling together parallel random number generation in Julia

Tue, 08 Jul 2014 00:00:00 +0000

I’m starting to work on some computationally demanding projects, (Monte Carlo simulations of bootstraps of out-of-sample forecast comparisons) so I thought I should look at Julia some more. Unfortunately, since Julia’s so young (it’s almost at version 0.3.0 as I write this) a lot of code still needs to be written. Like Random Number Generators (RNGs) that work in parallel. So this post describes an approach that parallelizes computation using a standard RNG; for convenience I’ve put the code (a single function) is in a grotesquely ambitiously named package on GitHub: ParallelRNGs.jl. (Also see this thread on the Julia Users mailing list.)

A few quick points about RNGs and simulations. Most econometrics papers have a section that examines the performance of a few estimators in a known environment (usually the estimators proposed by the paper and a few of the best preexisting estimators). We do this by simulating data on a computer, using that data to produce estimates, and then comparing those estimate to the parameters they’re estimating. Since we’ve generated the data ourselves, we actually know the true values of those parameters, so we can make a real comparison. Do that for 5000 simulated data sets and you can get a reasonably accurate view of how the statistics might perform in real life.

For many reasons, it’s useful to be able to reproduce the exact same simulations again in the future. (Two obvious reasons: it allows other researchers to be able to reproduce your results, and it can make debugging much faster when you discover errors.) So we almost always use pseudo Random Number Generators that use a deterministic algorithm to produce a stream of numbers that behaves in important ways like a stream of independent random values. You initialize these RNGs by setting a starting value (the “pseudo” aspect of the RNGs is implicit from now on) and anyone who has that starting value can reproduce the identical sequence of numbers that you generated. A popular RNG is the “Mersenne Twister,” and “popular” is probably an understatement: it’s the default RNG in R, Matlab, and Julia. And (from what I’ve read; this isn’t my field at all) it’s well regarded for producing a sequence of random numbers for statistical simulations.

But it’s not necessarily appropriate for producing several independent sequences of random numbers. Which is vitally important because I have an 8 core workstation that needs to run lots of simulations, and I’d like to execute 1/8th of the total simulations on each of its cores.

There’s a common misconception that you can get independent random sequences just by choosing different initial values for each sequence, but that’s not guaranteed to be true. There are algorithms for choosing different starting values that are guaranteed to produce independent streams for the Mersenne Twister (see this research by one of the MT’s inventors), but they aren’t implemented in Julia yet. (Or in R, as far as I can tell; they use a different RNG for parallel applications.) And it turns out that Mersenne Twister is the only RNG that’s included in Julia so far.

So, this would be a perfect opportunity for me to step up and implement some of these advanced algorithms for the Mersenne Twister. Or to implement some of the algorithms developed by L’Ecuyer and his coauthors, which are what R uses. And there’s already C code for both options.

But I haven’t done that yet. :(

Instead, I’ve written an extremely small function that wraps Julia’s default RNG, calls it from the main process alone to generate random numbers, and then sends those random numbers to each of the other processes/cores where the rest of the simulation code runs. The function’s really simple.

function replicate(sim::Function, dgp::Function, n::Integer)
    function rvproducer()
        for i=1:n
            produce(dgp())
        end
    end
    return(pmap(sim, Task(rvproducer)))
end

That’s all. If you’re not used to Julia, you can ignore the “::Function” and the “::Integer” parts of the arguments. Those just identify the datatype of the argument and you can read it as “dgp_function” if you want (and explicitly providing the types like this is optional anyway). So, you give “replicate” two functions: “dgp” generates the random numbers and “sim” does the remaining calculations; “n” is the number of simulations to do. All of the work is done in “pmap” which parcels out the random numbers and sends them to different processors. (There’s a simplified version of the source code for “pmap” at that link.)

And that’s it. Each time a processor finishes one iteration, pmap calls “dgp()” again to generate more random numbers and passes them along. It automatically waits for “dgp()” to finish, so there are no race conditions and it produces the exact same sequence of random numbers every time. The code is shockingly concise. (It shocked me! I wrote it up assuming it would fail so I could understand pmap better and I was pretty surprised when it worked.)

A quick example might help clear up it’s usage. We’ll write a DGP for the bootstrap:

const n = 200     # Number of observations for each simulation
const nboot = 299 # Number of bootstrap replications
addprocs(7)       # Start the other 7 cores
dgp() = (randn(n), rand(1:n, (n, nboot)))

The data are iid Normal, (the “randn(n)” component) and it’s an iid nonparametric bootstrap (the “rand(1:n, (n, nboot))”, which draws independent values from 1 to n and fills them into an n by nboot matrix).

We’ll use a proxy for some complicated processing step:

@everywhere function sim(x)
    nboot = size(x[2], 2)
    bootvals = Array(Float64, nboot)
    for i=1:nboot
        bootvals[i] = mean(x[1][x[2][:,i]])
    end
    confint = quantile(bootvals, [0.05, 0.95])
    sleep(3) # not usually recommended!
    return(confint[1] < 0 < confint[2])
end

So “sim” calculates the mean of each bootstrap sample and calculates the 5th and 95th percentile of those simulated means, giving a two-sided 90% confidence interval for the true mean. Then it checks whether the interval contains the true mean (0). And it also wastes 3 seconds sleeping, which is a proxy for more complicated calculations but usually shouldn’t be in your code. The initial “@everywhere” is a Julia macro that loads this function into each of the separate processes so that it’s available for parallelization. (This is probably as good a place as any to link to Julia’s “Parallel Computing” documentation.)

Running a short Monte Carlo is simple:

julia> srand(84537423); # Initialize the default RNG!!!
julia> @time mc1 = mean(replicate(sim, dgp, 500))

elapsed time: 217.705639 seconds (508892580 bytes allocated, 0.13% gc time)
0.896 # = 448/500

So, about 3.6 minutes and the confidence intervals have coverage almost exactly 90%.

It’s also useful to compare the execution time to a purely sequential approach. We can do that by using a simple for loop:

function dosequential(nsims)
    boots = Array(Float64, nsims)
    for i=1:nsims
        boots[i] = sim(dgp())
    end
    return boots
end

And, to time it:

julia> dosequential(1); # Force compilation before timing
julia> srand(84537423); # Reinitialize the default RNG!!!
julia> @time mc2 = mean(dosequential(500))

elapsed time: 1502.038961 seconds (877739616 bytes allocated, 0.03% gc time)
0.896 # = 448/500

This takes a lot longer: over 25 minutes, 7 times longer than the parallel approach (exactly what we’d hope for, since the parallel approach runs the simulations on 7 cores). And it gives exactly the same results since we started the RNG at the same initial value.

So this approach to parallelization is great… sometimes.

This approach should work pretty well when there aren’t that many random numbers being passed to each processor, and when there aren’t that many simulations being run; i.e. when “sim” is an inherently complex calculation. Otherwise, the overhead of passing the random numbers to each process can start to matter a lot. In extreme cases, “dosequential” can be faster than “replicate” because the overhead of managing the simulations and passing around random variables dominates the other calculations. In those applications, a real parallel RNG becomes a lot more important.

If you want to play with this code yourself, I made a small package for the replicate function: ParallelRNGs.jl on GitHub. The name is misleadingly ambitious (ambitiously misleading?), but if I do add real parallel RNGs to Julia, I’ll put them there too. The code is still buggy, so use it at your own risk and let me know if you run into problems. (Filing an issue on GitHub is the best way to report bugs.)

P.S. I should mention again that Julia is an absolute joy of a language. Package development isn’t quite as nice as in Clojure, where it’s straightforward to load and unload variables from the package namespace (again, there’s lots of code that still needs to be written). But the actual language is just spectacular and I’d probably want to use it for simulations even if it were slow. Seriously: seven lines of new code to get an acceptable parallel RNG.

Slowly moving to Julia from R

Tue, 18 Feb 2014 00:00:00 +0000

Julia is a new computing language that’s gotten a lot of attention lately (e.g., this Wired piece) and that I’ve ignored until recently. But I checked it out a few days ago and, holy crap, it’s a nice language. I’m rewriting the code in my book to use Julia instead of R and I’m almost certainly going to use it instead of R in my PhD class next fall.

So, why Julia and why not R? (And, I suppose why not Python/Matlab/other languages?)

Multiple dispatch — so you can define a function

TrickyAlgorithm(aThing, bThing)

differently to depend on whether “aThing” is a matrix of real numbers and “bThing” is is a vector, or “aThing” is a vector and “bThing” is a matrix, or any other combinations of data types. And you can do this without lots of tedious, potentially slow, and confusing (to people reading and maintaining the code) argument checks and conversion within the function.

This is a form of Object Oriented Programming, but in typical OOP “TrickyAlgorithm” would need to be a method of “aThing” or “bThing”. Also note that multiple dispatch is present in R as well.

Homoiconicity — the code can be operated on by other parts of the code. Again, R kind of has this too! Kind of, because I’m unaware of a good explanation for how to use it productively, and R’s syntax and scoping rules make it tricky to pull off. But I’m still excited to see it in Julia, because I’ve heard good things about macros and I’d like to appreciate them (I’ve started playing around with Clojure and like it a lot too…). And because stuff like this is amazing:

@devec r = exp(-abs(x-y))

which devectorizes x and y (both vectors) and evaluates as

for i = 1:length(x)
    r[i] = exp(-abs(x[i]-y[i]))
end

(This example and code is from Dahua Lin’s blog post, “Fast Numeric Computation in Julia.”) Note that “evaluates as” does not mean “gives the same answer as,” it means that the code “r = exp(-abs(x-y))” is replaced with the loop by “@devec” and then the loop is what’s run.

Speed — definitely faster than well written R; I don’t have a great feel for how well it compares to highly optimized R (using inline C++, for example), but I write one-off simulation programs and don’t write highly optimized R.And the language encourages loops, which is a relief. R discourages loops and encourages “vectorized” operations that operate on entire objects at once (which are then converted to fast loops in C…). But I use loops all the time anyway, because avoiding loops in time series applications is impossible. R’s poor support for recursion doesn’t help either.And, more to the point, I teach econometrics to graduate students. Many of them haven’t programmed before. Most of them are not going to write parts of their analysis in C++.

The syntax is fine and unthreatening, which will help for teaching. It basically looks like Matlab done right. Matlab’s not a bad language because its programs look like they’re built out of Legos, it’s a bad language because of its horrendous implementation of functions, anonymous functions, objects, etc. Compared to R, Matlab and Julia look downright friendly. Compared to Clojure… I can’t even imagine asking first year PhD students (some with no programming experience at all) to work with a Lisp.

The last point that’s always mentioned in these language comparisons. What about all of the R packages? There are thousands and thousands of statistical packages coded up for R, and you’re giving that up by moving to a different language.This is apparently a big concern for a lot of people, but… have you looked at the source code for these packages? Most of them are terrible! But some are good, and it might take some time to port them to Julia. Not that much time, I think, because most high-performance popular R packages are a thin layer of interoperability over a fast implementation in C or C++, so the port is just a matter of wrapping it up for Julia. And most of the well designed packages are tools for other package developers.That’s not quite true of R’s statistical graphics, though. They’re really great and could be hard to port. And that’s more or less the only thing that I’m sure that I’ll miss in Julia. (But hopefully not for too long.)

Lastly, and this is important: the same massive quantity of packages for R is a big constraint on its future development. Breaking backwards compatibility is a big deal but avoiding it too much imposes costs.

Anyway, since I converted some R code to Julia I thought it would be fun to compare speeds. The first example is used to show the sampling distribution of an average of uniform(0,1) random variables. In R, we have

rstats <- function(rerror, nobs, nsims = 500) {
    replicate(nsims, mean(rerror(nobs)))}

which is (I think) pretty idiomatic R (and is vectorized, so it’s supposed to be fast). Calling it gives

> system.time(rstats(runif, 500))
#     user  system elapsed
#    0.341   0.002   0.377

For comparison to the Julia results, we’re going to care about the “elapsed” result of 0.377 seconds; the “system” column isn’t relevant here. Calling it for more observations and more simulations (50,000 of each) gives

> system.time(rstats(runif, 50000, 50000))
#      user  system elapsed
#   204.184   0.217 215.526

so 216 seconds overall. And, just to preempt criticism, I ran these simulations a few times each and these results are representative; and I ran a byte-compiled version that got (unexpectedly) slightly worse performance.

Equivalent Julia code is

function rmeans(dist, nobs; nsims = 500)
    means = Array(Float64,nsims)
    for i in 1:nsims
        means[i] = mean(rand(dist, nobs))
    end
    return means
end

which is pretty easy to read, but I have no idea if it’s idiomatic. This is my first code in Julia. If you like to minimize lines of code and preallocation of arrays, Julia has list comprehensions and you can write the stylish one line definition (that gave similar times)

rmeans_pretty(dist, nobs; nsims = 500) =
    [ mean(rand(dist, nobs)) for i = 1:nsims ]

We can time (after loading the Distributions packages):

julia> @elapsed rmeans(Uniform(), 500)
### 0.093662961

so 0.09 seconds, or about a quarter the time as R. But (I forgot to mention earlier), Julia uses a Just In Time compiler, so the 0.09 seconds includes compilation and execution. Running it a second time gives

julia>  @elapsed rmeans(Uniform(), 500)
### 0.004334132

which (as a reader had to point out to me) is 1/20th the time again. So we are getting into the ~100 times faster range.

Running the larger simulation, we have

julia> @elapsed rmeans_loop(Uniform(), 50000, nsims = 50000)
### 77.318591953

so the R code is a little less than three times slower here. (The compilation step doesn’t make a meaningful difference.) So, Julia isn’t hundreds of times faster, but it is noticeably faster than R, which is nice.

But speed in this sort of test isn’t the main factor. I’m really excited about multiple dispatch — it’s one of the few things in R that I really, really liked from a language standpoint. I really like what I’ve read about Julia’s support for parallelism (but need to learn more). And I like metaprogramming, even if I can’t do it myself yet. So Julia’s trying to be a fast, easy to learn, and elegantly designed language. That’s awesome. I want it to work.

ps: and it’s open source! Can’t forget that.

Quick thoughts and advice on whether to get a PhD in economics

Sat, 01 Jun 2013 00:00:00 +0000

A friend emailed me to ask about grad school; I thought I might as well share my thoughts here too. First, I should make it clear that I basically never recommend getting a PhD to anyone. Grad school pays almost nothing, is very stressful, damages most people’s relationships, exacerbates any latent depression or anxiety issues you might have, and the only payoff is that you’ll get to learn about one or two things in intense, intense detail for years. So I emphasize the negatives, reasoning that if you can hear all the downsides and still think, “screw it, sounds like fun”… then going to grad school might be a good idea. But if I can scare you away, then you never should have tried to get a PhD in the first place.

That said, here are some more thoughts, in no particular order.

If you’re going to get a PhD in the social sciences, or anything close to the social sciences, it should be in economics. You can do pretty much any of the other disciplines, but you’ll get paid considerably more and there are more academic positions, private sector and government jobs, etc available. Doing economics in a business school may be even better.
I’d only suggest getting a PhD if you want a job teaching and/or doing research as a professor or at a research institute or think-tank. I don’t think that there are that many other jobs where the training is going to help. I’m sure there are jobs where having a PhD would be a nice addition for a candidate, but probably not so much that it’s worth 5–6 years of full time school. Of course, you might know of particular positions that I don’t.
If you actually, you know, have a job, career, life, etc., that you’d be putting on hold for grad school (which is the case for the friend I mentioned above, but wasn’t for me when I applied) you should probably be pretty selective about where you’d go; so top 10 or so only. Basically, you want a setup where, even if grad school goes badly and the job market is terrible when you graduate, you can have a job “better” than what you have now on graduation. That’s probably true if you go to Harvard, MIT, Stanford, etc., but not necessarily if you go to, say, UCSD. Obviously, you want to know who you plan to work with, but people change their minds once they get to grad school all the time, so being at a great school will give you options if you change your mind.
If you’re coming straight out of college, or if you have a bad job with poor career prospects (I was working as a file clerk in the post dot-com bubble Bay Area two years after I graduated from college), you can take a bigger risk and go to a top 30 school that matches your presumed research interests. UCSD turned out to be a great fit for me, mostly because I was right when I expected to do econometrics.
The first year of grad school (in Econ) will be really tough and unpleasant if you’ve been out of school for a little while. There’s a lot of math and it may have been a little while since you sat down and worked through proofs, etc. Expect that the core sequence of classes will be tough.

Other opinions

Noah Smith is much more positive than I am
Chris Blattman is more specific
Susan Athey’s advice on applying to grad schools looks great and you should definitely read it.

Numeric stability in Clojure's statistics package, Incanter

Wed, 10 Apr 2013 00:00:00 +0000

See below for some updates; and I don’t follow Clojure’s development, so the whole post might be out of date by the time you’re reading it.

I’ve been casually interested in Lispy languages for a while; i.e. I’d like to learn one and am not going to let the fact that I only know bits of elisp after 15 years of using Emacs deter me. Clojure seems hot and I really like the talks I’ve seen by Rich Hickey, its creator. Simple Made Easy is especially good. Plus, Clojure even has a well-regarded statistics library, Incanter, so awesome.

Anyway, procrastinating tonight, I decided to check out Incanter’s source code on Github. I have a really simple method for evaluating open source statistics packages: find the linear regression function, and look for variables named xpxi or xtxi and, if they exist, basically avoid the package (for some reason, these variable names are ubiquitous.). Inverting the X’X matrix is a pretty bad idea–it is a numerically unstable way of calculating the regression coefficients that (in problems I’ve worked on) sometimes leads to a non-idempotent projection matrix X(X’X)⁻¹X’ (or, using less terminology, X’X(X’X)⁻¹ may not equal the identity matrix). Needless to say this results in pretty bad estimates of the OLS coefficients. Douglas Bates talks about performance issues in this R-news article too, but I’m much more concerned about numeric instability. I don’t necessarily have the most informed opinion about the best way to get the OLS estimates, but I’ve gotten good results from the QR decomposition.

As of today, you can probably guess, Incanter fails this test. The source code and documentation are pretty unconcerned with the actual implementation of OLS, and I can’t figure out exactly what algorithm solve uses (I’m unpersuaded by the claim that it is “equivalent to R’s solve function” and can’t really track it down any further than that part of the code).

These details are important! I mean, I appreciate the effort and the good intentions that goes into developing open source packages like this. But if you’re developing statistical software for other people to use, you really need to understand the numeric properties of the routines you’re writing and you need to transparently communicate that understanding to other people who might use your code. So I guess I’ll stick with R for a while longer.

Update on July 7, 2013

This post drew a response from an Incanter user (who wants to stay anonymous, or I’d just quote the email). In short, the email pointed out that, if you know how Clojure handles dependencies and libraries, it’s not hard to verify that Incanter’s solve uses LAPACK’s DGESV from JBLAS to invert the X’X matrix using an LU factorization, which is the exact same algorithm as R’s solve, so my suspicion there was misplaced. Great!

Obviously my first reaction to the email was astonishment that anyone’s read my blog. But I think my original point still stands. Looking for a variable named xtxi used to estimate OLS is a quick and dirty way to evaluate a statistics package, because inverting the X’X matrix is numerically unsound compared to other methods of estimating OLS. R, for example, does not use solve for OLS, it uses the QR decomposition.

Here’s some R code where the difference matters (I don’t know Clojure, but this uses the same algorithms). This isn’t quite linear regression, it’s a comparison of different methods for constructing projection matrices, P = X (X’X)⁻¹ X’ (so it’s basically identical to linear regression). Here are three different methods:

projection.LU1 <- function(x) x %*% solve(crossprod(x)) %*% t(x)

projection.LU2 <- function(x) crossprod(t(x), solve(crossprod(x), t(x)))

projection.QR <- function(x) {
  QR <- qr(x)
  tcrossprod(qr.Q(QR)[, QR$pivot[seq_len(QR$rank)], drop = FALSE])
}

The first inverts X’X using the same algorithm in Incanter; the second uses a slightly better version but is basically the same, and the third uses the QR decomposition, just like R.

From the mathematical definition, we can see that P= PP, a property called idempotence, which is an easy property to verify numerically. Here’s a set of 51 observations for 11 regressors (each column is z raised to the pth power for p = 0, 1, 2,…,10 and z between zero and one).

X <- outer(seq(0, 1, 0.02), 0:10, "^")

And now we can “verify” idempotence (up to numerical tolerance)

> all.equal(projection.LU1(X), projection.LU1(X) %*% projection.LU1(X))
[1] "Mean relative difference: 0.0002737938"

> all.equal(projection.LU2(X), projection.LU2(X) %*% projection.LU2(X))
[1] "Mean relative difference: 0.0001990939"

> all.equal(projection.QR(X), projection.QR(X) %*% projection.QR(X))
[1] TRUE

All of the code is available for download here: https://gist.github.com/grayclhn/5717763

You can see (and verify it for yourself by downloading the code) that the first two methods of calculating P, which invert X’X using the LU factorization just like Incanter, are not idempotent. The third method, which uses the QR decomposition just like R, is idempotent. So in this example, the QR decomposition works and the LU factorization doesn’t.

This example is obviously contrived, but it’s not isolated. Chapter 11 of Seber and Lee’s (2003) Linear Regression Analysis shows the same thing: that if the regressors are “badly” distributed, the QR decomposition is more reliable. (In the interest of full disclosure, I should admit, embarrassing as it is, that chapter 11 of Seber and Lee is essentially all I know about these issues, so I’m not claiming a lot of expertise.)

As one last point, let me preempt anyone who might respond, “yes, these issues matter in those particular examples, but that will never come up in real research.” Try to guess why I look to see how a stats package calculates the linear regression coefficients, and why I have this particular criterion that I care about instead of any other, and why….

It’s obvious, right? This is something I’ve personally screwed up before. An early version of my job-market paper had fantastic empirical results that turned out to be entirely an artifact of using (X’X)⁻¹ to calculate the F-test statistic instead of using the QR decomposition and the “projection.QR” function in the code example is copied directly from that project (a later version). I was lucky and paranoid enough to catch it before circulating the paper but the event definitely left an emotional impression.

An extremely subjective outline of the economics job market

Sat, 01 Aug 2009 00:00:00 +0000

I had promised that I’d consolidate the collected “job market wisdom” of the 2009 graduating class and present it as a set of notes for future classes. I put it off (after getting a job, one still needs to defend…) and I’ve forgotten a lot of the suggestions that other people had. So these notes largely reflect my own job market experience. I’ll try to include other people’s opinions when I can remember them. If you have comments or information that I missed, please let me know. Some people have and requested that they stay anonymous, but I’m happy to attribute the advice to you if you want credit.

Since these documents last longer than is originally intended, I should give a little bit of background information. I went to graduate school at UCSD where I studied econometrics for six years. I went on the job market in the 2008-2009 academic year and defended in July, 2009. My first job out of grad school is an Assistant Professorship in the Economics Department at Iowa State University.

— Gray Calhoun

Other job market guides

John Crawley’s Job Market Guide and Advice is the official guide, and you should read it before reading any further here. (First, because it’s better than my guide. Second, because I’m going to assume you’ve read it.) The JOE website links to the latest version.
William Thomson’s A Guide for the Young Economist covers the job market (in the second edition), and you should read the book anyway. Lucky you, the sample chapter that MIT Press has put online is the one covering the job market.

Brief time frame for job market

A basic timeline: what happens when; what you need to own or create by that date; and what you probably want in addition to the bare minimum.

mid-October through end of November

The deadlines for job applications largely fall in late October through mid November. About a quarter of the applications are due after Thanksgiving. At the miniumum, you need:

A generic cover letter: Writing more than one cover letter per “class” of job application is overkill; I had one cover letter for research positions (regardless of the department), one for private sector jobs, and one for teaching positions. They got tweaked a little for some of the positions, but not by more than a sentence or two (one example: I mentioned that “I worked closely with Dimitris Politis” when applying to statistics departments). The cover letter needs to include this information:

Your field (and secondary fields if they’re relevant)

The position you’re applying for (sometimes the letter goes to HR. Sometimes the department is hiring for several positions and the same person handles all of it. Etc.)

Your advisors names

The fact that you’ll attend the AEA meetings (It is not always assumed that you’re attending the AEA meetings. I was asked about this by email a few times when I omitted it in the cover letter).
A paper (your “job market paper”): This should go without saying. I was surprised by how many people read my job market paper, though. Before going through the process, I had the impression that job market papers were not actually read, but that the interview decision was made largely on letters of recommendation. I was wrong.
Three letters of recommendation.
A one to two page summary of your dissertation.
A CV: UCSD encouraged us to use its template, which is (unfortunately) very ugly. It probably doesn’t matter if it is ugly… but I didn’t use their template. Karen Kelsky has fantastic instructions for putting together a CV, and I tried to follow them the last time I rewrote my CV. A LaTeX template is available here.
A webpage. Nothing fancy, just links to your CV and your job market paper, which should be clearly labeled “job market paper” if you have more than one paper online (or even if you don’t). A picture of yourself looking relaxed but professional is good too.

Additional material you might want beyond the minimum

You may also need to put together a few things that many, but not all, schools require. You don’t need to put much effort into these elements of your application, unless you badly want a teaching job. But many schools will consider your application incomplete if it doesn’t have these items included.

A summary of teaching evaluations.
A one-page “statement of teaching philosophy.”

And, if you can pull it off, you’ll want these things too. (They are highly recommended):

Another paper: “Highly recommended” is an understatement. I have heard rumors that the department (UCSD) will re-evaluate where they’ll push a candidate on completion of a second paper.
Another letter of recommendation.
A resume: If a job asks for “a resume,” it does not mean “a CV.” A resume should have different information than a CV; for example, computer skills and non-research work experience. Obviously, if you are only applying to academic jobs, you don’t need a resume and, knowing that, you may not want to put your resume on your website (having the resume on your website signals that you’re applying to non-research jobs, which signals that you’re not confident about your ability to land a research job, which signals lower ability… I’m overthinking this, but not really kidding)

Miscellaneous other points about this stage of the job market:

If you are not a native English speaker, you absolutely need to have a classmate who is a native English speaker proofread all of your jobmarket documents.
Other academic departments (other than economics; statistics or public policy, for example) may be on a slightly earlier or later schedule than the econ job market. Government departments that don’t hire many Ph.D. economists also can operate on a different application schedule (the State Department, as a hypothetical example), so (if they hire later) they can be useful for hedging. You should be aware that these departments typically pay noticeably less than institutions that hire lots of economists.
There’s disagreement on whether you should or should not apply for jobs that are “out of reach.” I did. The cost of applying to any given job is tiny, and it seemed like getting my paper read by as many people as possible had benefits on its own. I didn’t see a downside, and don’t know what the counterargument is; but it does apparently exist.
It is often easier to apply for a job than to determine whether or not you are a good fit for that job. If you find yourself agonizing and trying to decide whether you should apply, save yourself some time and apply.
Economics Department jobs are posted on JOE (the AEA’s website), ERN, FRN, Walras.org, inomics, UK-JOE. Most business schools, government agencies, and private companies looking for Ph.D. Economists post to JOE as well. European departments sometimes post here, sometimes post elsewhere. Many smaller, less prestigious liberal arts colleges won’t post anywhere but will have faculty jobs listed on their HR website.
Other types of jobs can be posted elsewhere. For Public Policy: APPM; Health Economics: AcadamyHealth; Statistics: IMS, ASA, and the online stats job site
Be aware that this process is stressful and busy for everyone. You won’t be an exception.
About organization: I kept an Excel file where each row contained all of the information for a given job: deadline, department, contact info, JEL codes, etc. In retrospect, this was complete overkill. I made slight errors in transcribing the information, and I went back and checked the original listing before writing each cover letter. The point is, you’re only sending 200 job packets at most, so automating and computerizing the process is silly. If I were doing it over, I’d print out each job listing and keep it in a binder sorted by the application deadline.

In the first two to three weeks after Thanksgiving

Schools will contact you about interviews at the meetings. This is more stressful than earlier, but not particularly busy.

Your advisor and committee can make phone calls. Ask them to if you’re not getting interviews. This stage is the one that your advisor’s pull matters. They can get you interviews at the AEA meetings, not flyouts and certainly not jobs.
There are sales (Macy’s, after Thanksgiving, etc.) that are especially helpful for buying luggage and suits. Shopping would be a nice way to blow off steam.
Generally, people perform worse in their first few interviews, perform best in the early afternoon on the first day, and perform worse again as they get fatigued later in the day and on the second and third days. If you could schedule your interviews to reflect that pattern, it would be great. But you really can’t. You will get calls about interviews over about a two week period; suppose the first interview you get is very prestigious, and the next interview is much less. You have no way of knowing where they fall in the distribution of interviews you’re going to have. By the time you’ve scheduled enough interviews to know how you should schedule them, it’s too late.
One thing you can do is know how far it is between hotels; this is hard to do if you haven’t lived in the city, so ask someone who has. Last year (January of 2009), some people didn’t realize that SF has hills, and walking up those hills is more difficult than walking down.

First few days of January

You’ll be at the AEA meetings, have fun! The biggest thing you’ll need is to have practiced describing your job market paper and dissertation! Professional attire and comfortable shoes are also a good idea, but are much less important.

You really are running from hotel room to hotel room for three (or more) days; one of the only things I was surprised about was how much my feet hurt.
You need to practice a lot for the interviews. Practice with friends. There are lists online of potential questions specific to the Economics job market. The basic format is:

You sit down, someone says, “tell us about your research” with some amusement.

You describe your job market paper – they may interrupt with questions and they may not; it really depends on the interviewer.
You want this period to be very busy. It may or may not be stressful
If you are invited to a reception, you should go. Even though you’ll be tired and feel exhausted. I skipped some and was asked (by one of them) why I didn’t attend, so apparently people do notice.
It is hard to describe how different various people’s experience was at SF last year. Some people had a great time and enjoyed meeting with everyone. Other people had interviews that they thought were hostile and unpleasant. In general, the people who resolved beforehand that the experience would be fun usually had fun – obviously, there’s a huge amount of endogeneity in that statement, but it does mean that your experience is heavily influenced by your attitude even if (and I don’t believe this) your outcomes are not.
If you have co-authored papers (I didn’t, but this advice comes from someone who did), expect to be asked where the idea came from, who did what, etc. I assume this is even more likely if it is your job market paper.
Some research has been done on what happens in a “bad” job market. In short, the best candidates get more interviews and other candidates get fewer and worse interviews. Whether or not you believe the results, this research suggests that whenever you’re asked how the job market is going (by someone other than a close confidant) you should reply, “I’m really happy with the interviews I have and I’m busier than I expected to be.” It’s plausibly true, it’s a subtle signal that you’re a “good” candidate, and (most importantly) it’s a much more pleasant answer than, “this year sucks and I hate all of my interviews,” especially when you are asked the question during an interview.

January and February

This is when flyouts and job negotiation. Here’s what you’ll need for sure:

A reliable laptop: If you have any doubt about the reliability of your laptop, buy a new one. The tiny netbooks only cost a few hundred dollars.
Slides (well, pdfs if we’re being literal): Specifically, you’ll need to give a polished job market presentation that runs 1:15 to 1:30. Different departments have different length seminars. You should be practicing a lot. You should also be aware that the conference rooms vary a lot. [UCSD students] are used to a short and wide seminar room, others can be very long and skinny.

You usually arrive the day/night before
You’ll have dinner the night you interview or the night before. Sometimes, the “official” dinner will be the night after you interview, but people will want to take you out to an “unofficial” dinner the night before; so arrive early the day before if you can to keep that night’s dinner free.
Leave town that night or the next morning
Interviews with the faculty run about half an hour to an hour. you may talk be about your paper, maybe about the university, maybe about misc. topics (the Detroit lions losing, for example, or about the university’s retirement plan). The most fun interviews are when you talk about the other person’s research.
The dinner may or may not be an “interview,” depending on who is free that night. If it is an interview, one standard format is to ask technical questions over dessert. This can be done pleasantly or not; at ISU, I was politely asked to explain how I planned to do a few technical extensions to my job market paper that I’d discussed earlier. So, you should not drink at dinner.
An anonymous reader had the following comment: ‘It might be obvious, but the job interview is the entire time you are with potential colleagues on the fly-out – this includes dinners and happy hours and drinks and whatever. Some freshwater schools, in particular, will wait to ask hard questions at the end of a dinner including drinks, and perhaps even a happy hour before that. Be professional the whole time and be ready to be on your game (your research, the particulars of what your job market does) whenever you are around these potential colleagues. Remember two parts to the term “potential colleagues:” the “potential” means you have to know your stuff around them, and they are very interested to know if you know; the “colleague” means it pays to show that you are a human being that can be sociable, and so isn’t a drag to have around.’

Other concerns during this period

Contacting institutions about nearby interviews is okay. This caused me some stress about what “nearby” means when you’re flying out of San Diego, especially because people here have misconceptions about actual distance between cities on the east coast. I still don’t know.
After you get an offer, you can and should contact everyone else that you interviewed with — they understand how the process works and understand that you’re on a deadline. If you hadn’t heard from them since the AEA meetings, it’s almost certain that they’ll tell you, “thanks for letting us know. We’re going to hire someone else.” But it’s still important; it’s good for “networking;” and they’ll probably say something like, “I really liked your research,” which will make you feel good about yourself.
Fly direct whenever possible. There’s a temptation to try to save the department paying for your flyout a little bit of money, but you are going to be very tired and probably sick, and missing a connecting flight could be a disaster. No one ever was offered a job for saving the department $300 on a plane ticket.

Hopefully this period is busy. It is extremely stressful.

General points about the process

Here are a few things we did when I was on the job market that you might find helpful. I haven’t seen them mentioned on other job market guides.

We set up a different mailing list. People had to opt in to it, the idea being that people are sometimes shy about sharing information because they don’t want to come across as gloating or hurt someone’s feelings if the job market is going particularly badly. If people opt in to a second mailing list, you can be at least somewhat comfortable that they value the information. Almost everyone participated.
For the same reason, we set up a shared spreadsheet with everyone’s interview information. A spreadsheet format makes a little more sense for organizing this information, and getting regular email updates of everyone else’s great interviews is extremely irritating. So we set up the spreadsheet separately.
We held practice interviews. The Federal Reserve interviews on campus before any of the scheduled faculty interviews, and we wanted to practice for those interviews too. Also, there are only two interviews with faculty, and we wanted to practice a little bit more than that. I think that some of us did about 5-10 practice interviews overall.
We attended each other’s practice job talk seminars, and scheduled extras. Again, the job talk matters, so having lots of practice is helpful.

Speculative points on networking

I didn’t do many of these things, but in retrospect wish that I had. So take it for what it’s worth.

The job market is an excellent time to start “networking.” After the job market, you’re going to have a job (I am sure of this. It is a tautology). One aspect of any job is “building a professional network.” If you were smart, you started this in grad school by meeting speakers, going to conferences, following up with people, paying attention to who’s working on what, etc. But, like most of us, you probably did an incomplete job of it. This year, you are going to meet a lot of people at a lot of different institutions, and many of them are going to have very similar research interests to yours — there’s a lot of self-selection. So, approach the job market from two angles: you do want a job, but you also want to start meeting future collaborators. This means: follow up with people, write thank you notes after interviewing, try to talk to them about their research and discuss common interests, etc.
You will be very tempted to skip departmental seminars during the Fall quarter and consequently avoid meeting the speakers. Don’t. There are many obvious reasons that you should continue to go to the seminars and meet the speakers; I assume you can see them yourself. Another, not at all obvious reason is that you are likely to take a job at a department that does not have regular seminars in your field – many departments have a single, weekly, “department seminar” that hosts all outside speakers. So this could be your last year (for the foreseeable future) of working at a department that has regular seminars in your field. Take advantage of it while it lasts.
A note about thank-you notes. They’re not necessary, but they’re nice. Despite what you read in job-search books, write emails and not hand-written letters. People can and do reply to emails. Moreover, since no one checks his or her department mailbox regularly, you can’t be sure that a hand-written letter would even be received. Sending or not sending a note will make no difference on whether or not you are hired, but getting hired shouldn’t be your only goal this year.
Different advisors handle the job market differently. Some advisors are very upbeat before the market, some are very negative, some are very hands-on, some aren’t. Often candidates in consecutive years who share the same advisor will have interviews at many of the same places. So it probably makes sense to contact your advisor’s previous advisees and ask them about their job market experience and what you should expect.

First, it’s useful information on its own.

Also, you are doing similar research to your advisor’s previous advisees, so you should know them. Even if you were friends in grad school, it may have been a few years since you last talked.
For exactly the same reason, you may want to contact recent UCSD grads who work in your field (Micro Theory, for example). (Speaking from the future as someone who now gets contacted by UCSD grads, it’s great. I enjoy it and it never feels like an imposition or burden. More of you should do it.)
For private sector jobs, and probably some government jobs, the stereotypical private sector job search advice applies: talk to people inside the companies you’re considering working at, and if you don’t know someone working there, ask someone who does to introduce you. One of the job market guides (I think Stanford’s) says (in a different context) that a job at an economic consulting company is a sure thing for any of their graduates, which would suggest that trying to get those jobs is a waste of time. This may or may not be true for UCSD graduates in a typical year, but probably isn’t true in an exceptionally bad year. Moreover, it ignores an important point. There are huge differences in working at one company vs. another, in one branch vs. another, and in one city vs. another. Some of these differences are apparent to outside applicants, but most aren’t; and some are clear but there’s no reliable way for an outside applicant to control how the application is handled. The point of talking to people is to ensure that the right branch/department in the right city in the right company is handling your application and to make sure that you know what the “right” one is for all of those variables.

Closing thoughts

Do not isolate yourself from other grad students. This year is extremely stressful, and almost no one else you know can empathize with what you’re going through (it’s like being a teenager). Talk to each other. Go out to dinner at the AEA meetings. Text each other. One nice feature of the shared spreadsheet was that it removed uncertainty about how other people are doing – which is nice on its own, but also helped make it easier to talk about the job market… instead of asking Chris (for example) “how’s it going?” and worrying that he’ll respond by crying, I could say, “congratulations on that new interview” or “man, why the hell aren’t more places lining up to interview you?”
Do not isolate yourself from your family. Where you move affects your family too. Your spouse is also extremely stressed out. This can go a bit too far, though: I met a few candidate’s spouses at the AEA meetings and found it a bit weird.
You may have noticed that this outline did not mention any of the economics job-market rumor boards, wikis, etc. They don’t have much information, people on them lie, and it’s depressing and stressful to read the messages. I avoided them. They may have improved since I was on the job market, but I doubt it.