Arc Forumnew | comments | leaders | submit | akkartik's commentslogin

The reason to use 'write rather than 'display is precisely so that things can be pasted into the repl. All you should have to do is 'read and 'eval the entire error string.

But there's definitely something busted about this. In anarki:

  arc> (= x "abc")
  arc> (err x)
  Error: "abc"           ; ok, looks good
  arc> (= x 'abc)
  arc> (err x)
  Error: "error: abc"    ; whaa..
There's also a long-standing issue that's bothered me before:

  arc> (= x "abc")
  arc> (err x " shouldn't be a string")
  Error: "abc \" shouldn't be a string\""
You're right, that's pretty ghastly. It should print:

  Error: "abc" shouldn't be a string
or:

  Error: "\"abc\" shouldn't be a string"
I'm still mulling what the culprit is here, but I don't think it's write vs display.

-----

4 points by rocketnia 4586 days ago | link

The culprit is that 'err is defined to be Racket's 'error. It looks like every single use case of 'error is discouraged for one reason or another in the Racket reference:

http://docs.racket-lang.org/reference/exns.html#%28def._%28%...

- (error sym) creates a message string by concatenating "error: " with the string form of sym. Use this form sparingly.

- (error msg v ...) creates a message string by concatenating msg with string versions of the vs (as produced by the current error value conversion handler; see error-value->string-handler). A space is inserted before each v. Use this form sparingly, because it does not conform well to Racket’s error message conventions; consider raise-arguments-error, instead.

- (error src frmat v ...) creates a message string equivalent to the string created by

  (format (string-append "~s: " frmat) src v ...)
When possible, use functions such as raise-argument-error, instead, which construct messages that follow Racket’s error message conventions.

-----

2 points by rocketnia 4584 days ago | link

Er, I knew it was weird for me to say "'err is defined to be Racket's 'error," but I just realized, that factoid was in the original post of this thread. :-p

-----

2 points by akkartik 4591 days ago | link | parent | on: A follow-up on the pipe operator

Done! https://github.com/arclanguage/anarki/commit/afdfdf929a

I've also extracted the _ handling into a new function called (ugh) functionize. And I've fixed zap to show how you can make other functions "_ aware".

But like I said, this isn't as valuable in arc since you already have the [.. _ ..] syntax. All it permits is more widespread ssyntax use. You can say _.x instead of [_ x], and you can chain ssyntax around obstacles.

I found the idea interesting because it asks the question: do we need [.. _ ..] syntax, or can we eliminate the need for this reader macro just with a few simple changes to the functions where we most often use it?

-----

4 points by rocketnia 4591 days ago | link

This is a horrible change. I didn't respond to the original post because of "Thumper's rule," and I couldn't rebut thaddeus in time to stop you. :(

-

== Problems with functionize in general ==

The functionize-based utilities are discontinuous about the way they detect underscores: The body can use _ three times, or two times, or one time, but as soon as it uses _ zero times, it means something completely different. Thanks to this, I can break several layers of code by making a single local edit. But will I? Yes:

The 'treemem function detects occurrences of _ without regard for quoting or local scopes. So if I use an _ to activate one functionize-based utility, then I'll accidentally activate all the other functionize-based utilities which surround that one. If I want to avoid refactoring several layers of code each time I edit, I pretty much have two options:

- I can avoid putting an _ anywhere in my code, in which case this functionize feature won't be very useful to me.

- I can make sure to activate each and every functionize utility as soon as I use it, in which case they would have been better off as 'let variants. For instance, I might settle on the idiom (zap (do '_ ...) foo), but it would be more convenient to say (zaplet orig foo ...).

-

== Problems in Arc ==

Your Anarki commit is one of those things that is "guaranteed to break all your code." Personally, I like using the pattern (zap [map [...] _] args), which now breaks since the _ activates zap's automatic function wrapper. It seems you would want me to write (zap (map [...] _) args) instead, but for compatibility with Rainbow, Jarc, etc., I think I'll define a macro (itfn ...) and write (zap (itfn:map (itfn:...) it) args). Effectively, I'll be recreating the [...] functionality in the way I like.

Meanwhile, Arc already covers a lot of the functionality of Clojure's -> and ->> operators using (aand ...). If you still miss -> or ->>, I recommend just implementing an 'aand variant that doesn't short-circuit on nil. Call it 'ado or something.

Here's how to use 'aand/'ado with the examples from that thread you linked to (http://arclanguage.org/item?id=18044):

  (-> (range) [map [* _ _] _] [filter even? _] [take 10 _] [reduce + _])
  (-> 5 [+ _ 3] [/ 2 _] [- 1 _])
  
  (ado (range) (map [* _ _] it) (filter even? it) (take 10 it) (reduce + it))
  (aand 5 (+ it 3) (/ 2 it) (- 1 it))
-

== Considerations in Wart ==

Clojure's -> and ->> operators are not infix, and that might be good for their indentation. I'm not really sure.

  (-> (...)
    (...)
    (...
      ...)
    (...))
  
  (...) ->
    (...) ->
    (...
      ...) ->
    (...)
  
  (...)
    -> (...)
    -> (...
         ...)
    -> (...)
On the other hand, I like that you can potentially use infix to combine things like -> and ->> into one chain:

  (...)
    -> (...)
    ->> (...
          ...)
    -> (...)

-----

1 point by akkartik 4591 days ago | link

Just spotted your comment while I'm working on something else. Haven't digested it all, but judging from the first five words -- feel free to revert! It was definitely intended as an experiment, and I'm not attached to it. I may well do so myself later today if you don't get to it first.

-----

1 point by akkartik 4591 days ago | link

Ok, done reading now, and you're right, I'll revert it.

I can only defend myself against the wart section :) In wart the pipe operator can only take two args and is intended to be used in infix. I use a non-infix transform for more args, and for prefix mode in general: https://github.com/akkartik/wart/commit/ec0f9a38b8

My weak defense for the rest: functionize and the _ syntax was only intended for tiny expressions.

-----

2 points by rocketnia 4590 days ago | link

"I can only defend myself against the wart section :) In wart the pipe operator can only take two args and is intended to be used in infix. I use a non-infix transform for more args, and prefix mode in general"

Oh, so you're pursuing both options at once. I look forward to you figuring out what kind of indentation you prefer here. :) My "considerations about wart" section was only wishy-washy anyway.

---

"For the rest, my weak defense is that functionize and the _ syntax is only intended for tiny expressions."

In Penknife, when I used the a'b operator as sugar for (b a), I found I ended up with a few really long lines of a.b.c'd'e.f, so it kinda suffered from its own success. ^_^ My a'b is the same as your (a -> b._), and it exactly corresponds to your no-underscore special case, (a -> b), so I expect you to have the same issue.

I suspect these syntaxes actually have a special tendency to let sugar accumulate, driving them away from the ideal "tiny expressions" case. Specifically, they make it possible to inject new code without breaking apart the surrounding sugar first:

  a.b.f.c.d                  # before refactoring
  a.b."foo".c.d              # illegal
  a.b -> (_ "foo") -> _.c.d  # legal? (not quite the example you gave)
  a.b'[itfn:it s.foo].c.d    # Penknife code of similar generality
Fortunately for wart, its infix operators allow whitespace in between, which possibly means you can write these long expressions on multiple lines. (That wasn't the case in Penknife.)

-----

1 point by akkartik 4590 days ago | link

"I look forward to you figuring out what kind of indentation you prefer here. :)"

Here's my current thinking: https://github.com/akkartik/wart/blob/aec0af2676/082macex.wa...

"these syntaxes actually have a special tendency to let sugar accumulate, driving them away from the ideal "tiny expressions" case."

Definitely. The wart example you gave is indeed legal, but the actual _ expression is still simple, even though the chain is long.

It seems nested expressions often work fine because each macro only wraps its arg and leaves the rest to the inside.

  (zap (map car._ _) (list list.1 list.2)
  => (zap (fn(_) (map car._ _)) ..)
  => (zap (fn(_) (map (fn(_) car._) _)) ..)
But that's less readable than when you have the [] delimiting scopes.

Some other concrete examples I came up with of the problems you pointed out:

  (zap (cons? & (fn(_) ..)) ..)  # broken by nested scope
  (zap (cons? & (fn(x) (+ '_ car.x)) ..)  # broken by quoting

-----

1 point by akkartik 4590 days ago | link

It's out, and I'm not putting it back or anything, but FWIW:

  (mac functionize (expr)
    (if (and (treemem '_ ssexpand.expr)
             (~caris expr 'make-br-fn)
             (~caris expr 'fn))
      `(fn(_) ,expr)
      expr))

  arc> (= x '((1) (2) (3)))
  arc> (zap [map [car _] _] x)
  arc> x
  (1 2 3)
This idea was in the wart version from the start (https://github.com/akkartik/wart/commit/7e5d620b28#diff-1) but I didn't mention it for simplicity, and then totally forgot to carry it over into anarki.

It's still pretty hacky though, and has more in common with C's macros than with lisp's.

-----

1 point by akkartik 4593 days ago | link | parent | on: Why should zero be truthy?

Interesting. One quibble with this idea: it doesn't matter as much that map et al work on nil if nil isn't at the end of each list.

So perhaps the reason for empty list to be special is that so many list algorithms are recursive in nature, and it's nice to be able to say "if x recurse" rather than *if !empty.x recurse". Hmm, the empty array or empty string isn't included in every array/string respectively, so perhaps it's worth distinguishing from nil in some situations..

-----

2 points by akkartik 4583 days ago | link

I just ran into a case where I wished the empty list wasn't the same as the false value. When implementing infix in wart (http://arclanguage.org/item?id=16775) I said: "Range comparisons are convenient as long as they return the last arg on success (because of left-associativity) and pass nils through."

  (a < b < c)
  => (< (< a b) c)   ; watch out if b is nil

  (< nil x)          ; should always return nil
But there's one situation where you want (< nil x) to not be nil: when x is a list and you want to perform lexicographic ordering (https://en.wikipedia.org/wiki/Lexicographical_order; http://rosettacode.org/wiki/Order_two_numerical_lists).

This problem wouldn't occur if there was a first-class boolean type; then I could use False or something for chaining comparisons.

-----

2 points by akkartik 4583 days ago | link

Ok, I'm now experimenting with a new keyword in wart called false.

a) There's still no boolean type. The type of false is symbol. (The type of nil has always been nil; maybe I'll now make it list.)

b) if treats both nil and false as false-y values.

c) nil and false are not equal.

d) Comparison operators now short-circuit on false BUT NOT nil.

I can mostly use either in place of the other. But I'm trying to be disciplined about returning false from predicates and nil from functions returning lists.

Wart now has four hard-coded symbols: nil, object, caller_scope and false.[1]

Thoughts? It was surprisingly painless to make all my tests pass. Can anybody think of bugs with this kinda-unconventional framework? If you want to try it out:

  $ git clone http://github.com/akkartik/wart
  # Optionally "git checkout 0ff47b6bce" if I later revert this experiment.
  $ cd wart
  $ ./wart
  ready! type in an expression, then hit enter twice. ctrl-d exits.
[1] fn is just a sym with a value:

  let foo fn (foo () 34)
  => (object function {sig, body})

-----

2 points by fallintothis 4582 days ago | link

Technically, my first thought was that something was broken. Hitting C-d as soon as I got the prompt:

  $ time ./wart
  ready! type in an expression, then hit enter twice. ctrl-d exits.
  => nil

  real    0m29.200s
  user    0m27.602s
  sys     0m0.000s
Anyway, I was going to test to see if you had Arc's t; but it doesn't look like it:

  (if t 'hi 'bye)
  020lookup.cc:28 no binding for t
  => bye
Note that it's trivial to add:

  (<- t 't)
  => t
  (if t 'hi 'bye)
  => hi
The reason I thought to try this was because I initially balked at maintaining false and nil at the same time with the same truth values. Then I thought of t, and suddenly the pieces clicked together: at least in part, it seems like you just want a Python-like system anyway.

Once I got the landscape laid out in my head, I started objecting to it less, because I could make sense of it. You're most of the way there:

- false is a separate, canonical false value.

- t (if you chose to have it) is a separate, canonical truth value.

- nil is an empty list, but empty lists are false.

Compare to Python's True, False, and []. The major differences being:

1. No first class boolean type. In wart, this produces more of a disconnect between t and false. t (i.e., 't) is just a normal symbol whose truth value is incidental. But false is a special, unassignable keyword.

  (<- false 'hi)
  => hi
  false
  => false
Python lacks symbols (you can't just say True = 'True), so this disconnect between symbolic value and keyword doesn't exist. There is still, however, a different sort of disconnect in Python because the "first class" boolean type gets contaminated by the int type:

  >>> False < 10
  True
  >>> 867 + True
  868
  >>> isinstance(True, int)
  True
  >>> isinstance(0, bool)
  False
  >>> True, False = 0, 1
  >>> if True: print "hi"
  ...
  >>> if False: print "hi"
  ...
  hi
2. You don't take Python's next logical leap. Since you already make the empty list false, other values become fair game, such as the thread's original idea (make 0 false), the empty string, other empty data structures, etc. But like I said before, I make do in such systems. Keeping nil falsy is really just your prerogative, if you want to avoid calls to empty? that much. ;)

-----

2 points by akkartik 4582 days ago | link

Thanks for trying it out, and for the comments! Yeah it's gotten slow :(

I hadn't realized how close to python I've gotten. Seems right given how the whitespace and keyword args are inspired by it. On rosetta code I found a cheap way to get syntax highlighting was to tag my wart snippets with lang python :)

I've been using 1 as the default truth value, and it's not assignable either. I was trying to avoid an extra hard-coded symbol, but now that I've added false perhaps I should also add true.. I'm not averse to going whole-hog on a boolean type, I'd just like to see a concrete use case that would benefit from them. pos seems a reasonable case for keeping 0 truth-y, and the fact that lists include the empty list seems a reasonable case so far to keep nil false-y. But you're right, I might yet make empty strings and tables false-y.

(True, False = 0, 1 :( That's the ugliest thing I've ever seen python allow. At least throw a warning, python! Better no booleans than this monstrosity.)

-----

2 points by rocketnia 4582 days ago | link

"pos seems a reasonable case for keeping 0 truth-y"

While I personally like 0 being truthy, I don't see this as a convincing reason.

I'd treat 'pos exactly the same way as 'find. They're even conceptually similar, one finding the key and the other finding the value. For 'find, the value we find might be falsy, so truthiness isn't enough to distinguish success from failure. The same might as well be true for 'pos.

---

"But you're right, I might yet make empty strings and tables false-y."

What if the table is mutable? That's an interesting can of worms. :)

JavaScript has 7 falsy values, all of which are immutable. If we know something's always falsy, we also know it encodes a maximum of ~2.8 bits of information--and usually much less than that. It takes unusual effort to design a program that uses all 7 of those values as distinct cases of a single variable.

This means if we have a variant of Arc's (and ...) or (all ...) that short-circuits when it finds a truthy value, we don't usually have to worry about skipping over valuable information in the falsy values.

If every mutable table is falsy as long as it's empty, then a falsy value can encode some valuable information that a practical program would care about, namely the reference to a particular mutable table.

---

"(True, False = 0, 1 :( That's the ugliest thing I've ever seen python allow. At least throw a warning, python! Better no booleans than this monstrosity.)"

There's some rationale here:

http://www.python.org/dev/peps/pep-0285/

http://docs.python.org/2/whatsnew/2.3.html

http://www.python.org/download/releases/2.2.1/NEWS

The PEP describes the design and rationale of introducing booleans to Python this way. Version 2.3 implements this. Version 2.2.1 preemptively implements bool(), True, and False to simplify backporting from 2.3.

Notably, the variable names "True" and "False" were chosen to be similar to the variable name "None", and all three of these are just variables, not reserved words.

Later, version 2.4 made it an error to assign to None:

http://docs.python.org/2/whatsnew/2.4.html

From what fallintothis says, apparently the same change hasn't been made for True and False.

-----

2 points by akkartik 4582 days ago | link

Hmm, so how is one expected to check for list membership in arc? Ah, this would seem to be the canonical idiom:

  (aif (mem f seq)
    <operate on car.it>)

-----

2 points by rocketnia 4581 days ago | link

Oh, very nice! ^_^ I remember using this a few times, but yours looks much better:

  (aif (pos f seq)
    <operate on seq.it>)

-----

1 point by akkartik 4581 days ago | link

Thanks a bunch for the python links, especially the last one. They were most illuminating.

I think the error is that None, False and True were ever 'constants' rather than literals.

Update: ah, this is fixed in python 3.

-----

2 points by akkartik 4579 days ago | link

So I emailed Guido van Rossum with this question and he was nice enough to respond :)

http://python-history.blogspot.com/2013/11/story-of-none-tru...

Couldn't have done it without you, rocketnia.

-----

2 points by rocketnia 4582 days ago | link

"I was trying to avoid an extra hard-coded symbol"

Speaking of which, why are you making false and nil count as symbols at all?

I suppose it gives them an external representation without coining a new syntax like #f.

-----

1 point by akkartik 4582 days ago | link

Yeah, I'm just minimizing how much I need to change, picking my poison between hard-coded symbols and extra cell types.

-----

1 point by akkartik 4574 days ago | link

I've added some messages to at least set expectations on how slow it is:

  $ wart
  g++ -O3 -Wall -Wextra -fno-strict-aliasing boot.cc -o wart_bin        # (takes ~15 seconds)
  starting up...       (takes ~15 seconds)
  ready! type in an expression, then hit enter twice. ctrl-d exits.

-----

2 points by akkartik 4597 days ago | link | parent | on: Why should zero be truthy?

Ah, I thought of one reason:

  (if (pos 1 '(1 2 3))
    ..)
pos can return 0 on success. We need to be able to distinguish that from nil for failure.

So it turns out that empty string or empty table actually have a stronger case for being false than 0 does.

-----

2 points by rocketnia 4597 days ago | link

If you're talking about treating 0 as falsy, then why not redesign 'pos in the process? I would happily use a 'pos that worked like this:

  > (pos 1 '(1 2 3))
  (0)

-----

2 points by rocketnia 4597 days ago | link

Oh, there's also the issue with 'find. It's already cumbersome to search a list of booleans for false and a list of lists for an empty list, and now it would be difficult to search a list of numbers for zero.

Of course, we could just do the same thing as 'pos:

  > (find 0 '(1 2 0 3))
  (0)

-----

2 points by akkartik 4601 days ago | link | parent | on: mzscheme dies

Oh, are you trying arcueid now because mzscheme wasn't working?

I've never seen this. Did you consider just trying racket from the same package?

What arc are you using? Does it have the mutable-pair bugfix from anarki? http://arclanguage.org/item?id=13616

If these questions don't help, one last idea is to try Pauan's https://github.com/arclanguage/arc-nu, which seems to be a better-quality implementation of the arc core. I haven't played with it much, but reports of what's broken might get me to do so, knock it into shape.

Another option to consider is pointing us at your code. If nothing else, you'll get information about whether we're able to reproduce your problem on various combinations of platform+arc. As you might have noticed before, providing a concrete failing scenario often yields better answers. (And narrowing it down to just a little code is even better.)

-----

2 points by lark 4601 days ago | link

Yes, I was trying arcueid becauze mzscheme wasn't working. The stack trace above is from arcueid, not mzscheme. I didn't consider using racket but I'm trying it now.

I'm using Arc 3.1. I don't know if it has the mutable-pair bugfix. I wish there were updated instructions somewhere explaining what's recommended or stable.

I'll try arc-nu too if racket crashes too. Thank you for these questions, they helped.

-----

2 points by thaddeus 4601 days ago | link

I suggest starting with Anarki: https://github.com/arclanguage/anarki if you haven't already. Noting the readme instructions also point to racket rather than mzscheme.

Arc 3.1 with either mzscheme 372+ or any version of racket will address the mutable pairs issue (http://arclanguage.org/item?id=10254).

-----

2 points by lark 4600 days ago | link

I tried Anarki before but I recall at the time it wasn't backwards compatible with Arc. There was something about saving and reading tables that required specifying their datatype that differed from Arc. Has this changed?

I would like to use a version that also supports file uploads. Anarki is the only version I know of that supports them. Do other Arc implementations support file uploads?

-----

2 points by thaddeus 4600 days ago | link

Ahh, I think you're referring to the template issue: http://arclanguage.org/item?id=16171.

I don't see how that issue would stop you from using Anarki. And it appears news.arc was modified to accommodate the change.

I have to say, I don't like that change in making templates into first class objects. It would have been better to add something like 'inst-with' to extend inst.

If that change is a show stopper for you, then I would just fork anarki and eliminate the template changes, or fork Arc 3.1 and add the file upload change.

Also, can you provide some insight as to what you're doing? Are you just creating a HNish clone with file uploads for added content? As I mentioned in the other post, I wouldn't use that stuff for data storage anyway. So maybe consider making more dramatic changes and use a db?

-----

2 points by lark 4600 days ago | link

Yes, it was the template issue. It wouldn't stop me from using Anarki but it would stop be from using some other Arc implementation if it became necessary, because I would have to modify the application.

Forking Anarki also means maintaining the fork, which means one more thing I have to worry about and probably break. I'd rather only add what I need: the ability to have file uploads to Arc 3.1. Uploading files through HTTP is needed regardless of the storage behind the application (filesystem vs database).

On a database, is there a sample Arc program that writes to Postgres?

-----

2 points by thaddeus 4599 days ago | link

Can't say I've seen a sample program for Postgres, but I would expect it to be fairly straight forward to use any database supporting HTTP as a protocol. And, really, almost any DB would be an upgrade compared to storing each record in it's own flat file - correct?

MongoDB: http://docs.mongodb.org/ecosystem/tools/http-interfaces/

Datomic: http://blog.datomic.com/2012/09/rest-api.html

OrientDB: https://github.com/orientechnologies/orientdb/wiki/OrientDB-...

MySQL http://code.nytimes.com/projects/dbslayer

Riak: http://docs.basho.com/riak/latest/dev/references/http/

And so on.

Of course Anarki also has HTTP Get/Post functionality where Arc 3.1 does not:

https://github.com/arclanguage/anarki/blob/master/lib/http.a...

You would also need to parse the HTTP results.

If you're dead-set on Postgres you could do something like this: http://rny.io/nginx/postgresql/2013/07/26/simple-api-with-ng... Though this seems a little a hackish to me.

Personally I'd use something like FleetDB (http://blackstag.com/blog.posting?id=3) over TCP, before I'd go back to that 1 record = 1 file nonsense.

-----

2 points by lark 4599 days ago | link

Thank you for looking up this work. A database makes a lot of things simpler, like indexes, and particularly joins. Thank you also for pointing out Anarki has HTTP Get/Post, I didn't know this.

Could I ask why you wouldn't go back to saving data in files?

It's a clean, fast solution if one doesn't need joins. Using SQL requires building up strings, or putting together an ORM. Better not go down that path if you don't need them.

https://news.ycombinator.com/item?id=6580834

-----

3 points by thaddeus 4599 days ago | link

> wouldn't go back to saving data in files.

It's not the file structure itself that's the issue, it's mechanisms built around accessing the file(s). For a SMALL HN clone it's probably OK, but if the requirements change even a little you're in for some trouble.

Food for thought: Once your app data can't fit into memory you're going to need to move to a real database or spend big bucks on more hardware. You could do what HN does and purge older data (reloading only when required), but still if your app design requires regular access/writes to older data and it doesn't all fit into memory then your hardware is going to start to thrash (HN had these type of problems from what I read). And with HN, since each record is stored in it's own file, it's going to be exponentially bad.

Also, it's pretty useful to have your dbs and applications running on different servers. Doing this provides other applications running elsewhere to access the data too. So once again it all boils down to your design and expectations, which is why I originally asked if you are just running a HN clone.

FYI I just posted a FleetDB v2 link on the Arc forum if you want to check that option out.

Added: Looks like there's another HTTP library https://github.com/arclanguage/anarki/blob/master/lib/web.ar..., not sure which one is better, but this looks newer.

-----

2 points by lark 4597 days ago | link

Thank you for FleetDB. The ability to execute transactions atomically is useful.

Access to older data is equally a problem for the filesystem and a database. Databases don't get away with it. Once the app data can't fit into memory databases thrash on disk too.

The only argument I see is that save-table saves an entire record in a single file. Wanting access to a specific field of that field means loading the entire record. A lot of databases function this way too, but in general they don't have to. They could bring in memory individual fields. You could claim that as the reason for filesystem access to be slow.

But even then, save-table could be modified to save each field in a separate file.

If the only thing missing from a filesystem is atomicity of multiple transactions, then I'd rather implement just that, in Arc, rather than write an entire database.

-----

2 points by thaddeus 4597 days ago | link

> Thank you for FleetDB.

No problem. Note though that FleetDB is all in memory too. The advantage is that you can:

  1. Put the database on its' own server with dedicated RAM. 
  2. Compose and re-use queries.
  3. Have index support, built in, without code writing.
  4. Have robust/high performance concurrency semantics, not just atomic file writes.
> Databases don't get away with it. Once the app data can't fit into memory databases thrash on disk too.

You're correct in that they too will do seeks to disk, but nowhere close to the same amount when compared to handling each record in it's own file. Just try loading a million files vs 1 file with a million records.

> But even then, save-table could be modified to save each field in a separate file.

Which will exponentially increase the number of disk seeks.

As I stated before it really depends upon your design. Ask yourself: Are you going to have say a million records where you have to run a query againts all records? Would indexes help with that query on that data? Do you really want to code each index into a table then change all your application code to use them? Or would you rather instruct your database to add an index and discover the existing queries already take advantage of them? Note that the last time I checked, HN off loaded that kind of heavy lifting to a third party service[1] that uses[2], you guessed it, a database!

I am not a database expert and I don't want to convince you to use tool that you don't need to use. I'm just saying I see enough advantages in using a database that I don't plan on using file writes.

[1] https://www.hnsearch.com [2] http://www.thriftdb.com/documentation

-----

2 points by akkartik 4597 days ago | link

"I have to say, I don't like that change in making templates into first class objects."

Can you elaborate on why? Since I was the one to make that change, I'm happy to be overruled or to discuss this further to address people's concerns.

-----

2 points by thaddeus 4597 days ago | link

Sure.

It seems to me, the original problem was this: One defines a template with a non-nil value as the default for a fields argument, then upon instantiating the template one wants to pass in a nil value to eliminate that fields default non-nil value from being part of the resulting table.

By making templates first class objects you have permitted these over-rides to occur, but at the same time you have introduced an additional layer of complexity for every user of templates (and in some cases even tables), when they don't care about that specific circumstance.

So at first glance I can't help wonder why not just re-write templates in a manner that eliminates poor handling non-nil values. Then we could have maintained the inst function while also allowing for other methods to be created, such as an 'inst-with' that allows nil values to over-ride defaults (or just have inst allow for over-rides).

I'm not sure how templates are written, but I imagine they make heavy use of tables, which is where the nil pruning is probably occurring. So templates would instead need to use association lists up until the output stage where data then gets pushed into a table.

Or, if instead, you want tables to support nil values, then that's also a change worth considering, but I see the introduction of a whole new type as a means to solve a certain circumstance as overkill with downside effects.

Now I haven't looked at any code to generate my thinking. I only have this to go off: http://arclanguage.org/item?id=15664

-----

1 point by akkartik 4597 days ago | link

You're absolutely right in the historical particulars. Templates started out as light macros atop tables, and their serialization was indistinguishable from that of tables. But they always seemed like a separate type given all the names we've devoted to manipulating them separately from tables. I thought the layer of complexity already existed, and was trying to simplify things so people didn't need to know that templates are built out of tables.

The problem with not supporting nil values in templates -- and therefore in the data models of any webapp -- is that you can't save booleans as booleans. That seems ugly.

Adding new names to paper over problems with existing names seems ugly as well.

Your idea of questioning the design of tables themselves is interesting. I hadn't considered that. A persistent data structure seemed like a separate beast, distinct from a table. But perhaps all tables should be easy to persist. Hmm.

---

My attitude towards anarki has been that it's trying to be the core for a hundred-year language, so it should get the core right without regard to how things were before. So we should avoid unseemly hacks for things that seem simple -- like storing a user pref that's a boolean -- without regard to how commonly they happen, or how easy workarounds are to find.

But I can absolutely imagine that others might have a different perspective on anarki. Let me ask you this: how would you design templates if you were doing so from scratch? I'm trying to get at how much of your argument hinges on compatibility. Perhaps there's a more general question here that we should be discussing. Perhaps we should pay more attention to lark's use case: not wanting to think about arc's internals.

-----

2 points by thaddeus 4597 days ago | link

> is that you can't save booleans as booleans.

Hmm. Well I see that as a separate issue. I think nil and boolean false are very different things. nil means no value or false while false means a false value. In fact I was recently going to suggest that arc should support proper booleans and that arc tables should also store both boolean values, while maintaining it's nil pruning feature. Which really stems from wanting to easily transform to/from json or edn like formats. Currently one has to fudge arc's write-json by passing around symbols for 'true / 'false.

> how would you design templates if you were doing so from scratch?

Well that's interesting that you ask, because I did just that when I implemented templates in Clojure. Of course Clojure supports true boolean values and maps can hold nil values, so it was trivial and very useful.

My Clojure templates are on steroids though. Not only do they match most of the features in arc, but they are cumulative, and accept anonymous functions as field values. The values can also refer to other fields inputs or results.

So for example I just did this one:

  => (deftem :article
      :id        #(UUID) 
      :published true       
      :text      nil 
      :msec      #(msec)
      :seconds   #(seconds (% :msec))
      :date      #(format-date (date (% :msec)) "yyyy/MM/dd HH:mm:ss"))
  
  #result 
  
  => (invoke :article :text "cool" :published nil)
  
    {:id #uuid "08692e96-de87-49aa-9ad1-e33bcd61e712", 
     :text "cool", 
     :msec 1382644173685, 
     :seconds 1382644173, 
     :date "2013/10/24 15:49:33"}
Notice how :seconds and :date use :msec as an input argument? ... Well that's how I would do it. :)

-----

1 point by akkartik 4597 days ago | link

Wow, I'd love to see that code!

Thanks also for the connection to json. There's a lot to consider here.

-----

3 points by thaddeus 4597 days ago | link

The code is

  1. In bad shape (wrote it early on) 
  2. Includes partial features not fully implemented.
  3. Has references to functions I can not split out without  
     creating a bunch of work.
  4. Includes features you would not care for (datomicish).
  5. Has oddities that make you wonder "why like that",
     until you realize you can pass in say a map of args   
     instead. 
With the above reasons, I was going to say I'll pass on releasing the code, but at long as you're ok just getting the scaffolding that will not run for you then here you go:

  (def index* (ref (hash-map)))
  (def templates* (ref (hash-map)))
  (def mutes* (ref (hash-map)))
  (def selfs* (ref (hash-map)))

  (defmacro deftem [name & fields]
    `(let [tem#   (quote ~name)
	   order# (evens (list ~@fields))
	   fmaps# (apply hash-map (list ~@fields))]
	(dosync (alter templates* assoc tem# fmaps#)
	        (alter index* assoc tem# order#)
	    fmaps#)))

  (defmacro defmute [name & fields]
    `(let [tem#   (quote ~name)
	  items# (list ~@fields)]
      (dosync (alter mutes* assoc tem# items#)
        items#)))

  (defmacro defself [name & fields]
   `(let [tem#   (quote ~name)
	  items# (list ~@fields)]
      (dosync (alter selfs* assoc tem# items#)
        items#)))

  (defn invoke-fields
   ([tem base fields allowables]
     (invoke-fields tem base fields allowables nil))
   ([tem base fields allowables testfn]
    (let [fks   (keys fields)
          selfs (@selfs* tem)]
      (reduce
       (fn [m k]
          (assoc m k
	        (if (detect? k fks); must use 'detect' opposed to 'find' for nil vals must be inserted.
		          (aifn (fields k)
	                  (try (it m)
	                    (catch Exception e (it)))
	                  (let [bfn (base k)]
                      (if (and (detect? k selfs)(fn? bfn))
                          (try (bfn (merge m {k it}))
	                          (catch Exception e (bfn)))
                           it)))
              (aifn (base k)
                    (try (it m)
                         (catch Exception e (it)))
                     it))))
		 (hash-map) allowables))))


  (defn invoke [tem & fields]
    (let [temx  (split-name tem)
          tem1  (first-identity temx)
	  atem? (is (last temx) "+")
	  xfn   (type-fn tem)
          temk  (xfn tem1)
	  base  (@templates* temk)
	  prox  (@mutes* temk)
	  fval  (first fields)
	  fmap  (cond (map? fval) fval ; for file loading map of saved records
	              (coll? fval) (apply hash-map fval)
	           :else (apply hash-map fields))
	  imap  (invoke-fields temk base fmap (@index* temk))]
       (reduce
	   (fn [m [k v]]
		   (if (or (missing? k base)(nil? v)(detect? k prox))
	         (dissoc m k)
		       (assoc m (if atem? (nsify temk k) k) v)))
	   	(hash-map) imap)))

-----

2 points by thaddeus 4600 days ago | link

I'm not aware of a save table issue. I just ran this and had no problems:

  arc> (save-table (obj a 10 b 20) "tfile")
  nil

  arc> (load-tables "tfile")
  (#hash((b . 20) (a . 10)))
edit: IMHO I would avoid using save-table/load-table as data storage mechanism. It may work for certain application designs[1], but otherwise you would be better off writing to a real database.

1. http://arclanguage.org/item?id=15419

-----

1 point by akkartik 4597 days ago | link

Why are you concerned about compatibility? Did you have production data with the old version?

The reason I changed it was that the old way was buggy; there were scenarios where a round-trip to disk would corrupt data.

-----

2 points by lark 4597 days ago | link

Yes, I have production data with Arc 3.1.

It's scary to hear saving corrupts data.

-----

2 points by akkartik 4596 days ago | link

Eek. I can see how that would be scary. Though to me it's very good to hear that somebody is doing stuff with arc :)

If you haven't made any changes to templates (created your own, modified the ones in news.arc) you should be safe.

If you decide to migrate to anarki at some point, I'd be happy to help. Make a copy of your data, install it on a new server, and go over things to make sure everything looks good.

-----

1 point by lark 4588 days ago | link

Does this mean Anarki is not backward compatible with Arc?

-----

4 points by rocketnia 4588 days ago | link

In order to really define "backward compatible," you'd have to define Arc in a way that's implementation-independent. In Arc, the code is the spec, so as soon as the code changes, compatibility becomes subjective.

For instance, suppose Anarki defines a new utility and uses it to simplify the implementation of 10 other utilities. (It does this in a few places.) Now suppose my Arc 3.1 code has defined a utility with exactly the same name, and running this code on Anarki causes those other 10 utilities to misbehave, thus wrecking my program. This is a case where Anarki isn't compatible with Arc 3.1, but since it's so easy for me to choose a different name for my utility, it's hardly even worth mentioning. Pretty much any substantial update to Arc would break it in exactly the same way.

There's only one difference between Arc 3.1 and Anarki that's ever gotten in my way, and that's the way Anarki has revamped the [...] syntax to support multi-argument functions. When I say [do t] or [do `(eval ',_)], Anarki treats these as 0-arity functions, and when I say [let (a . b) _ ...], Anarki chokes when trying to search the dotted list for any underscored variables. Once again, this is the kind of change that's pretty easy to work around, and I can't really say Anarki is worse for having this extra functionality.

I'd say Arc platforms are not really portable with each other, in the sense that not all code that works on one platform will work on another. However, I've found it pretty easy to develop my code so it'll work on multiple Arc platforms at the same time.

-----

3 points by akkartik 4588 days ago | link

Yeah we've discussed this before: http://arclanguage.org/item?id=16178

-----

2 points by akkartik 4601 days ago | link

You're welcome. Yeah sorry I've been trying to get the word out about the right versions, and it feels like a failure that even a veteran member of this group like you is not aware of https://sites.google.com/site/arclanguagewiki, specifically https://sites.google.com/site/arclanguagewiki/arc-3_1/known-... :/

-----

2 points by lark 4600 days ago | link

On the following link:

https://sites.google.com/site/arclanguagewiki/

This link has been very helpful to me in the past and I'm glad it exists. But I do wish it contained only one sentence. The sentence that is the most important one.

-----

3 points by akkartik 4600 days ago | link

What's the most important sentence, something like "don't use arc 3.1, it has a bug, use this stable branch instead"? It's a wiki, please feel free to make changes to it. It might be easier to discuss this using actual changes.

Update: Ok, I've taken a stab at making the biggest gotcha more clear.

-----

2 points by akkartik 4600 days ago | link

Then again, perhaps we should rethink if that page is the best option. I work for Google, but Google sites is a pretty insane beast a lot of the time. For example, there seems to be no way to change the title of the site as a whole; the directions at https://support.google.com/sites/answer/90453 are out-of-date.

Some of us started a site at http://arclanguage.github.io a while ago, and it's now the canonical reference for arc functions. Perhaps we should focus on that. But it's a little bit harder to make changes to; you have to git clone, install jekyll, make changes, commit, git push. On a wiki you can stay within your browser. Do people have preferences?

Either way, we should have one place, and we should decide what the first sentence (in bold) should be.

-----

3 points by thaddeus 4600 days ago | link

Yeah, it's really confusing. I have no idea why Anarki is listed as a library. Anyone looking for a starting point would completely overlook it.

-----


Most interesting. I wonder how the environment knows when to draw a circle. The visualization piece lives and dies by how much work it is to setup for more abstract code.

The probing and tracing demos are impeccable.

-----

1 point by akkartik 4614 days ago | link | parent | on: Deeper into the guts of macroexpansion

Thanks rocketnia! I'm still trying to get mac to generate the comma-backquote expressions, but I'll report back.

(Thanks for finding the typo in foo!)

-----

1 point by akkartik 4607 days ago | link

I've been playing with this for a few days, and I'm still stuck on how to modify this decl:

  (def foo (a) `(list ,a))
so that:

  arc> (foo 3)
  (list 3)
instead returns (list ,3). How can you generate a literal unquote? Perhaps this is a bug in wart's core? Perhaps it's impossible to do if commas don't expand to unquote?

-----

2 points by rocketnia 4605 days ago | link

Sorry, that toy example is too small for me. I don't understand why this macro redefinition approach would lead to that snag (although it does look like a plausible snag to end up in ^_^ ). Could you give some more context?

---

"Perhaps it's impossible to do if commas don't expand to unquote ?"

From what you're saying, it sounds like you're using some custom data structures, but that you just haven't given yourself all the primitives you need to manipulate them.

  > foo <- (make-unquote 12)
  ,12
  
  > (unquote? foo)
  t
  
  > (unquote-get foo)
  12
  
  > foo <- (make-quasiquote 12)
  `12
  
  > (quasiquote? foo)
  t
  
  > (quasiquote-get foo)
  12
Pardon the pseudo-Wart, pseudo-Arc, pseudo-Scheme. XD

-----

1 point by akkartik 4605 days ago | link

Ah, yes I can manipulate commas like that. I was wondering if there's a quasiquote/unquote idiom to handle them :)

-----

1 point by akkartik 4600 days ago | link

Ok, I think I've found a testcase proving that putting the if within backquotes won't work.

  # Simplified expansion of:
  #   mac (foo x)
  #     `(+ ,x 1)
  #
  #   mac (foo x) :case nil
  #     `(list ,car.x)

  mac (foo x)
    `(if nil
       ,`(list ,(car x))
       ,`(+ ,x 1))

  (foo 3)
The problem here is that both branches of the if will be macroexpanded, throwing an unnecessary error for the call to car.

So it seems we do need the if to be outside backquotes, and my subsequent hacks :/ Do you see any holes in this argument?

-----

2 points by rocketnia 4600 days ago | link

First of all, I'm noticing your expansion is suspicious in a couple of ways.

The case itself should be a piece of generated code:

  -#   mac (foo x) :case nil
  +#   mac (foo x) :case `nil
  
  -  `(if nil
  +  `(if ,`nil
I made this same change back in http://arclanguage.org/item?id=18050 where I said "What you really needed was this." It might have looked like I was only talking about the expansion result, but the usage syntax changes too.

Also, this seems to be unrelated to the issue you're having, but I just realized you need to be careful about variable scope. What if the original definition said "mac (foo x)" but the extension said "mac (foo y)"? I recommend putting in a form that binds the right variables:

  -  `(if ,`nil
  +  `(if ,(with (x x)
  +          `nil)
  -    ,`(list ,(car x))
  +    ,(with (x x)
  +       `(list ,(car x)))
       ,`(+ ,x 1))
(Careful though, because this (with ...) form will add a layer of indirection to caller_scope.)

---

To address the issue you're encountering, yes, every case will expand at every call site, even if there's no intuitive need for it to expand there. It can't branch on a decision that hasn't been made until run time, unless you're willing to use the full power of fexprs. (I assume you're trying to maintain a semblance of stage separation, or you wouldn't distinguish macros from functions.)

Keeping this stuff in mind, you can write your code so that it doesn't give you that error:

  # Simplified expansion of:
  #   mac (foo x)
  #     `(+ ,x 1)
  #
  #   mac (foo x) :case `nil
  #     if acons.x
  #       `(list ,car.x)
  #       `(err "I hope this code will never be reached.")

  mac (foo x)
    `(if ,(with (x x)
            `nil)
       ,(with (x x)
          (if (acons x)
           `(list ,(car x))
           `(err "I hope this code will never be reached.")))
       ,`(+ ,x 1))

-----

2 points by rocketnia 4599 days ago | link

"What if the original definition said "mac (foo x)" but the extension said "mac (foo y)"? I recommend putting in a form that binds the right variables"

Whoops, that (with ...) stuff isn't ideal because it leaves the outer variables in scope. That is, if the original definition has a parameter called "list", an extension will see that binding even if it uses a different name for the same parameter, so it might try to call the global 'list and be surprised.

I suppose I recommend doing this instead, where g123-args is a gensym:

  mac (foo . g123-args)
    `(if ,(apply (fn (x)
                   `nil)
                 g123-args)
       ,(apply (fn (x)
                 (if (acons x)
                   `(list ,(car x))
                   `(err "I hope this code will never be reached.")))
               g123-args)
       ,(apply (fn (x)
                 `(+ ,x 1))
               g123-args))
Er, and there's one more change I should probably make here so you can actually use this for what you're trying to do.

  let g1-old foo
    (mac (foo . g2-args)
      `(if ,(apply (fn (x)
                     `nil)
                   g2-args)
         ,(apply (fn (x)
                   (if (acons x)
                     `(list ,(car x))
                     `(err "I hope this code will never be reached.")))
                 g2-args)
         
         (,g1-old ,@g2-args)
         ; or in Arc, ",(apply rep.g1-old g2-args)"
         
         )

-----

2 points by akkartik 4616 days ago | link | parent | on: Deeper into the guts of macroexpansion

Thanks for that idea! No I don't have a second use case that requires dispatching on the value of macro args. I'll keep an eye out and report back.

I'm a little leery of using pointer-based semantics. Probably because of my C background :p Your analysis was very thorough, and I like that push doesn't silently modify the values of other variables besides the one requested. I hadn't noticed that subtlety before. Yet another reason to avoid scar and co.

-----

2 points by fallintothis 4616 days ago | link

I'm a little leery of using pointer-based semantics. Probably because of my C background :p

Yeah, it's an interesting problem. One that I should probably understand better... I mean, I "get" it: the RAM machine model is simple. But I've not programmed enough C (or similar) to understand how far the semantics would permeate day-to-day programs.

There's the part of me that's like "oh, just have primitives to reference & dereference addresses; easy". I seem to recall Lisps that use something like (ref x) and (deref x), and the explicitness pleases me, coming from "modern" languages where you have to remember odd rules about whether/where things are passed as pointers. But then I read C code and see all these pointers-to-pointers all over the place---in spots of code where I normally wouldn't think of pointers at all. Then again, that might be endemic to C, where you're probably just being destructive by default.

I've always wondered how ML works out in this respect (http://homepages.inf.ed.ac.uk/stg/NOTES/node83.html). Its system seems elegant on first glance, but I have no experience in it. I seem to keep finding myself regretting this lately...I should pick up a project to scratch my ML-learning itch.

-----


"if it's really hacky my apologies, but took me 10 mins just to re-learn how to write even really basic expressions"

I thought the execution was fine! I only objected to the direction :p Only simplification I could see was to turn:

  (if args 
      `(,f ,@(join args (list x)))
      `(,f ,x)))
into:

  `(,f ,@args ,x)

-----

2 points by thaddeus 4616 days ago | link

yikes.. lol.

Also, and I'm not sure, but I have a sneaking suspicion I might need to w/uniq those input args. I just didn't get that far into arcs' internals last night.

-----


Interesting! Clearly my knowledge of the clojure primitives was extremely vague; I wasn't even aware that they operated on forms rather than function values.

Arc could eliminate the ->/->> dichotomy in another way entirely, using the existing syntax for single-arg fns. The examples at http://clojuredocs.org/clojure_core/clojure.core/-%3E%3E would become (arc-ish pseudocode):

  (-> (range) [map [* _ _] _] [filter even? _] [take 10 _] [reduce + _])
  (-> 5 [+ _ 3] [/ 2 _] [- 1 _])
At the same time, the examples at http://clojuredocs.org/clojure_core/clojure.core/-%3E become:

  (-> "a b c d" upcase [replace _ "A" "X"] [split _ " "] first)
What's more, you could mix first and last because all the arguments are actual values rather than hacky s-exprs. I already used that in the second example above, if you look closely.

---

The one missing use case is this:

  (-> animals 'dog 'thaddeus 'age)
Arc just has a different, retro-email solution for that:

  animals!dog!thaddeus!age
Though it all unravels if one of the keys is a string. Ok, we could use some help here. But it feels like a separate mechanism; I would avoid mixing `(,form ,x) and `(,x ,form) in different paths of a single macro.

---

Hmm, even this isn't too bad:

  (-> animals [_ 'dog] [_ 'thaddeus] [_ 'age])
You don't save typing compared to:

  (((animals 'dog) 'thaddeus) 'age)
But the matching parens are closer together and so easier to read.

-----

2 points by thaddeus 4616 days ago | link

I agree, mixing `(,form ,x) and `(,x ,form) is really ugly and I do like

  animals!dog!thaddeus!age
much better - I forgot arc could do that lol.

Also, if anyone were really caught up on the strings as keys hiccup, the idiomatic approach for arc would likely be this: http://arclanguage.org/item?id=13006. If one could only find a pleasant & available symbol.

-----

More