Buried in my 3000 lines of arc was this innocuous-looking 2-liner:

  (def most-recent-unread(user feed)
    (most doc-timestamp (rem [read? user _] feed-docs.feed)))

A little reorganization keeps the lists always sorted, so I can just:

  (def most-recent-unread(user feed)
    (car (rem [read? user _] feed-docs.feed)))

You can assess the level of optimization in a codebase by the size of the 'longest pole in the tent'. By that heuristic this past week is basically a testament to my incompetence. The good news with incompetence: it lets me find 10x speedups like this. That (hopefully) won't last.

Thanks to aw for hammering away at me to look at the mote in my own code before blaming my tools (http://arclanguage.org/item?id=11515). I could have saved myself a world of pain[1] if I'd just remembered these manual-profiling tools I wrote a year ago:

  (= times* (table))

  (mac deftimed(name args . body)
    `(do
       (def ,(symize (stringify name) "_core") ,args
          ,@body)
       (def ,name ,args
        (let t0 (msec)
          (ret ans ,(cons (symize (stringify name) "_core") args)
            (update-time ,(stringify name) t0))))))

  (proc update-time(name t0) ; call directly in recursive functions
    (or= times*.name (list 0 0))
    (with ((a b)  times*.name
           timing (- (msec) t0))
      (= times*.name
         (list
           (+ a timing)
           (+ b 1)))))

  (def print-times()
    (prn "gc " (current-gc-milliseconds))
    (each (name time) (tablist times*)
      (prn name " " time)))

This isn't the first time I've 'forgotten' to profile before optimizing. Hopefully it'll be the last.

[1] 2 days of experiments with ab, 2 days of looking at erlang/lfe/termite/tornado/unicorn, 1 fun day of rewriting in SBCL, and 1 day of moving to a new data structure I turned out to not need.