I like the examples stuff. The only issue that I have with it is that you are providing explicit, yet non-tested results. It makes sense to me to either provide just the code and have the documentation system generate the results, or to provide test cases instead.
Ideally, the 'results' in the documentation would still be computed dynamically, so you can always see whatever the current output would be, but also have the expected results available for separate regression testing.
Also, I think it would be great if the regression test results could be automatically generated and stored somewhere else, because I don't want to have to manually type in expected results for every case. Some mechanism to 'freeze' the current output as the accepted value for regression purposes would be cool.
Yeah, that whole side is a bit clunky. It gives readers feedback when things are broken so they aren't misled. But the job of the writer is still hard. Things would be improved by my last idea, to run examples along with other tests.
Here's a couple more issues:
a) Expected results are currently unevaluated, so there's no way to represent tables, and so on.
arc> (help counts)
[fn] (counts seq)
Returns a table with counts of each unique element in 'seq'.
Examples:
arc> (counts '(b a n a n a))
(obj b 1 a 3 n 2) <-- ERROR
b) We can't provide examples for non-deterministic functions like rand-elt.
c) We can't show stdout and stderr.
---
Last but not least, I've been thinking about the freezing idea myself in other contexts. I think there's something big there.
As for the 'freezing', I've been thinking about it in other contexts as well. I noticed that one of the companies I worked at seemed to want to do some of their regression tests that way, so it makes sense to provide a way to automate it.
In particular, I would like a way to convert an interactive console session (my normal method of testing code) into a set of unit tests. Not sure exactly what the right way to do that would be though.
It's still not a very good form of unit test, as it won't cover cases as mentioned above with randomness or side-effects. Mutation testing or fuzz testing / generators are probably better ways to generate unit tests, but they don't make very good examples.
Yeah. B can probably work just fine with dynamically evaluated examples, but stderr/stdout would require user interaction, which may not be acceptable. Maybe some way to indicate if an example should be evaluated or not?
What do you mean by 'run examples along with other tests'? Does that mean that you would include other tests in the help output? That the examples would be included when running the tests? Or something else?
(examples do
(do (prn "line 1")
(prn "line 2")
(prn "line 3"))
_)
Here's how it looks:
arc> (help do)
[mac] (do . args)
Evaluates each expression in sequence and returns the result of the
last expression.
Examples:
arc> (do (prn "line 1")
(prn "line 2")
(prn "line 3"))
It makes sense to specify the example expr / expected pairs as sexps & strings: run repl-output on exprs, compare those strings to the expected strings.
Thanks for thinking through that! Yes, I'd had a similar though not as fully-formed idea, but was stuck on distinguishing stdout from return value in the examples. But yeah, that's overthinking it -- we have to mentally parse the two at the repl anyway.
Only other quibble: a string isn't very nice to read next to code, even if it looks nice at the repl. Hmm..
Maybe we should make an auto scraper/notifier? Though iirc, the arc server doesn't really appreciate frequent queries. On closer look, it should be fine; the default throttle window should be 10s. It could query once a minute, and update an rss feed for arc forum posts and comments. Or provide webhook + email support.
I actually built one back in the day: http://akkartik.name/post/2011-05-13-05-47-29-soc. Trouble was, it kept getting banned/blocked even when I scaled back to crawl HN every five minutes. Eventually I was crawling so infrequently that I was missing stories that dropped off /newest in the meantime, so I stopped crawling.
I actually had an arc version for a brief period[1], but it didn't seem like there was as much need for it here. But perhaps the email notifications would be useful. I'll see if I can bring it back.
[1] I was certain I'd showed it here, but no I only showed it over email to a couple of folks. I really should bring it back just so y'all can look at it. Or at least locate the (rails) sources.
