"I am not saying that these three designs of mine represent the last word on the issues -- "100%" is really a bit of a misnomer, since no design is ever truly 100%. I would prefer to think of them as sufficiently good that they at least present low-water marks -- future systems, I'd hope, can at least build upon these designs."
But it's not clear that an 85% solution or 95% solution helps to solve the problem he identified earlier in the post:
"[The 80%] socket interface isn't general. It just covers the bits this particular hacker needed for his applications. So the next guy that comes along and needs a socket interface can't use this one. Not only does it lack coverage, but the deep structure wasn't thought out well enough to allow for quality extension. So he does his own 80% implementation. Five hackers later, five different, incompatible, ungeneral implementations had been built. No one can use each others code."
At best the number of different, incompatible systems will be lower over time. But there's no reason to believe that dissatisfaction with prior solutions will be more likely to build on them.
I'm curious if Conrad Barski was aware of Olin Shivers's regular expression library when he built http://www.lisperati.com/arc/regex.html and if so, if he built on the design. That would be a strong counter-example to my hypothesis.
"At best the number of different, incompatible systems will be lower over time. But there's no reason to believe that dissatisfaction with prior solutions will be more likely to build on them."
Could you clarify this? I think there might be a typo in here, but I don't know where.
Say a 95% solution leaves 1 in 20 hackers dissatisfied, where an 80% solution leaves 1 in 5 hackers dissatisfied. The number of dissatisfied hackers goes down. But won't they still continue to react to their dissatisfaction by creating new libraries that don't build on prior attempts?
"The number of dissatisfied hackers goes down. But won't they still continue to react to their dissatisfaction by creating new libraries that don't build on prior attempts?"
Who's saying they won't? If you're willing to believe that 100% solutions will lead to fewer dissatisfied hackers and less duplication of effort, I can't tell what other claims you're trying to challenge here.
---
To bring in my personal goals, I'm interested in using programming to improve the expressiveness of communication, so that we have less severity in our petty misunderstandings, our terminology barriers, etc.
While I'm interested in reducing duplicated effort or saving people from dissatisfaction, I'm mainly interested in these things because they go hand-in-hand with establishing better communication. If hackers are more productive, they can communicate more; and if they communicate more, they can find satisfactory tools and avoid duplicated effort.
---
I think it'll be more interesting to look the "80% solution" idea with a scenario that combines more than one solution, so that it's not a simple feedback loop anymore.
Suppose we have two completely unrelated projects A and B, with A being a 95% solution and B being an 80% solution.
Now let's say some hackers have goals in the A+B domain, so that people might suggest for them to use project A and/or B as part of the solution.
Only 76% are actually satisfied with A and satisfied with B at the same time. About 4/5 of the remaining hackers can still build on top of A, but they have to reinvent the functionality they hoped to get from B.[1]
Unfortunately, sometimes it's very difficult to combine the two dependencies A and B in a single program. If A and B involve different operating systems, different programming languages, etc., then the task of combining them may be even more difficult than reinventing the wheel. In these cases, the 76% quadrant of happy hackers must redistribute itself among the other quadrants.
How does it get distributed? Well, the user community itself is a feature of the system, so I'm assuming a 95% solution tends to have a more helpful community than the 80% solution by definition. We also haven't considered any reason that a helpful hacker would make a different technology choice than an unhelpful one, so I'll make the simplifying assumption that all communities have the same population-to-helpfulness ratio.[2] So I'd guess the 76% "+A+B" quadrant is necessarily redistributed to the 19% "+A-B" and 4% "-A+B" quadrants while roughly preserving that 19:4 ratio.
+A -A
+B 0% 17%
-B 82% 1%
This fuzzy reasoning suggests that if an 80% solution B is incompatible with a 95% solution A, approximately 83% of hackers won't use B to achieve their A+B goals. So we see, an 80% solution can leave the vast majority of hackers dissatisfied.
But 80% solutions must be acceptable in our software, or else all our software will be monolithic projects that take lots of resources to develop and maintain, and we won't be communicating very effectively.
Personally, one thing I take away from this is that the world could really use a near-100% approach to modularity, so that almost any two well-designed projects A and B can be used together. I've been working toward this.
(However, I also think this could backfire to some degree. Reusable software that doesn't bit-rot is in a way immortal, and certain immortal software may get in the way of or add noise to our communication.)
---
[1] While this 4/5 may look like it corresponds to the 80% solution, that's just a coincidence. This 4/5 roughly comes from the remaining 20% and 5% that each solution doesn't cover. We'd get a similar 4/5 result if we compared a 96% solution with a 99% solution.
[2] Some reasons this might be wrong: A big community invites people to use it as a symbol for inclusion-exclusion politics, even if that's non-sequitur with the original purpose of the community. Sometimes technology has lots of users not because its community is helpful or political, but because few people have the kind of expertise it would take to reinvent this wheel at all (not to mention how many people don't even know they have a choice).
Hmm, my interpretation of the socket complaint (that I quoted above) was that it was a qualitative rather than a quantitative argument.
Ah, I think I've found a way to reframe what Olin Shivers is saying that helps me make sense of it. The whole 80% thing is confusing. The real problem is solutions that don't work for the very first problem you try them out at. That invariably sucks. Whereas a solution that is almost certain to support the initial use case of pretty much anyone is going to sucker in users, so that later when they run into its limitations it makes more sense to fix or enhance it rather than to shop around and then NIH ("not invented here", or reinvent) a new solution for themselves. In this way it hopefully continues to receive enhancements, thereby guaranteeing that still more users are inclined to switch to it.
:) I'm using some critical-sounding language there ("sucker") just to help highlight the core dynamic. No value judgement implied.
---
I think both of us are starting with the same goal of helping hackers communicate better, share code better. But we're choosing different approaches. Your approach of fixing modularity is compatible with the Olin Shivers model, but I think both of you assume that receiving enhancements is always a good thing, and things monotonically improve. I don't buy that more and more hackers adding code and enhancements always improves things in the current world. There's a sweet spot beyond which it starts to hurt and turns away users who go off trying to NIH their own thing all over again.
My (more speculative) idea instead is to ensure more people understand the internals so that the escape hatch isn't NIH'ing something from scratch but forking from a version that is reasonable to them and also more likely to be understandable to others. By 'softening' the interface I'm hoping to make it more sticky over really long periods, more than a couple of generations. I don't think modularity will work at such long timescales.
It's not packaged up, all the internals are hanging out, some assembly is required. But it enumerates the scenarios I considered so that others can understand precisely what I was trying to achieve and hopefully build on it if their needs are concordant.
Thanks for the answer, especially the thoughts on self-hosted compilers. I'd always found myself vaguely unsatisfied with them as an outsider trying to understand them, so it's very useful to hear similar sentiments from someone who actually has experience building them.
BTW, I followed the instructions to run it and was pleasantly surprised by how painless it was.
I actually worried that it wasn't painless enough, and that I should have waited for the web repl to be done before submitting.
These long strings contain the compiled macros. They are only there for debugging. They belong in comments instead of strings, but I didn't think of it at that time.
1. Bounds checking errors are certainly in keeping with the spirit of scheme, no arguments there. If you care about distinguishing () from [], require if to have an 'else' branch, and consider it important to distinguish between #f and nil, then such substring behavior will fit right into that aesthetic milieu.
2. Incredibly, arc has never had a substring function, so I can't be sure what pg would do. My immediate reaction is that exceeding bounds would be permitted without error, but then sbcl seems to throw an error as well.
I looked at my toy language, which is more in the arc mould and more relaxed about the set of inputs acceptable to low-level primitives:
("abc" 0 10)
024string.cc:21 no such end-index in string: "abc" 10
=> nil
My unit tests show that I never considered out-of-bounds indices. Now that I consider them I'm tempted to permit them silently.
3. I am not persuaded by the reasons in the post. "Do one thing" is subjective about what "one thing" means. The unix creators have spoken out against all the flags that standard commands have creepingly added: http://harmful.cat-v.org/cat-v, recently https://twitter.com/rob_pike/status/410830396444532736. This disagreement with the community at large suggests that the rule of thumb leaves some room for interpretation, making it useless to resolve arguments. The phrase "referentially transparent" is grotesquely misused. The list of useful facts didn't feel compelling. The connection to recent security holes in other languages feels tenuous; you could make the same argument for not permitting lists to contain elements of multiple types if you were a java programmer, and comparing bounds checking to implicit type coercion is just a strawman. The two possible interfaces for substring are just different, there's no magic in either. I'm not convinced that the author made a good-faith attempt to build the "simple" version atop the more complex one.
There's clearly a difference of opinion here, but escalating a simple question to such diverse subjects is just an invitation for flames. Just pointing at the other ways that scheme has a similar policy toward strict and immediate errors suffices to keep aesthetics from devolving into religion.
4. The flame war itself feels, with hindsight and just from reading the archives, unnecessary and a poor advertisement for this community. Responses like http://lists.nongnu.org/archive/html/chicken-hackers/2013-02... and http://lists.nongnu.org/archive/html/chicken-hackers/2013-02... made me wince. Perhaps that's partly a culture gap; scanning the archives I found that the noob is still around, so perhaps no major offense was caused. Regardless, this wasn't a flame war so much as the insiders getting together to flame a noob. I looked around for possible reasons why it might have happened, but the mailing list seems to consistently stay around 25-50 participants per year, not high enough volume to justify such frustration. With such a small inflow every newcomer is precious, and every bad experience is hugely significant. The subject of substring doesn't seem to have come up before either, to make them so impatient.
Arc's attitude, championed by aw, is to avoid generalities and look for concrete code examples where an alternative makes for greater concision. That's our aesthetic and it has served well to avoid friction without avoiding debate altogether.
"Arc's attitude, championed by aw, is to avoid generalities and look for concrete code examples where an alternative makes for greater concision."
Yeah, this can be a pretty nice metric sometimes. :) This is probably one of the leading reasons to use weakly typed (or otherwise sloppy) versions of utilities.
Are you reading the tutorial or anarki or something else? Arc started out defining iso just for lists, but the name ('isomorphic') was evocative enough that we ended up extending it to arbitrary aggregates, including tables.
arc> (iso (obj 'a 'b) (obj 'a 'b))
t ; nil in arc 3.1
Hopefully we're still following the original vision, and it was just an oversight that iso doesn't support tables in arc 3.1. The interpretation is that is is for primitives when you care about fast comparison, and iso is for aggregates and arbitrary user-defined types. Since arc's tagged types are just lists, supporting lists and tables ensures iso will continue to support any user-defined types.
Over time my personal opinion (and that of some others here, but I'll let them speak for themselves) has diverged from the tutorial. I think it's useful to have one strong default operator that usually "does what you want", which is to make equal-looking things equal. So I advocate using iso everywhere unless you explicitly want to compare pointers.
No offense, but I'd like to give your comment an awful lot of errata:
Those objects you're creating will have the cons list (quote a) as a key. Non-atomic keys are full of gotchas because Arc's tables use use Racket's 'equal? for key comparison, rather than using 'is or 'iso. This means tables care whether a list ends in the symbol 'nil or the Racket empty list (), which is a detail that's otherwise invisible in Arc.
(If you want the symbol 'a as a key, the code is (obj a 'b).)
Arc's tagged types are not lists, and 'iso can support lists without supporting tagged types. (Personally, I wouldn't want 'iso to dig around in my tagged types anyway. I like pretending the data inside is encapsulated, except where my code absolutely needs to access it.)
Finally, does 'iso actually support tables in Anarki? I can't try it out right now, but I don't see any code for this when I search GitHub.
(defextend iso (x y) (isa x 'table)
(and (isa x 'table)
(isa y 'table)
(is (len keys.x) (len keys.y))
(all
(fn ((k v))
(iso y.k v))
tablist.x)))
; default impl for tagged types
(defextend iso (a b) ($.vector? a)
(iso ($.vector->list a)
($.vector->list b)))
Does the 'default impl' trigger your gag reflex? :)
Oops, I assumed Github's search would show me every occurrence in every file, like a recursive grep. No such luck, I guess. :-p So you got me.
While that's a reasonably useful default behavior for 'iso on tagged types, I'm not particularly happy to see it.
Arc provides very few helper utilities for working with tagged types--basically just the axioms 'annotate, 'type, and 'rep, plus 'isa and 'coerce--and the documented purpose of tagged types is for user-defined types. I came to Arc from a more OO-like background and saw this as a good way to control access to the implementation details of my types. Unless you explicitly called 'rep, you didn't get more coupling than you bargained for. This gave 'rep almost an "unsafe" reputation for me.
If 'iso gets to access the rep at will, then I may have been programming in Arc less robustly than I expected: Either each of my types should have come with an 'iso extension to reinforce my encapsulation, or I should have considered 'iso "unsafe" and used it less often. Not that I could have known this at the time! :)
But it's not a big deal. I do deep comparison operations sparingly enough as it is, and I don't know if my soft encapsulation technique actually matters because I rarely saw anyone else following similar guidelines. (I've rarely seen anyone using 'annotate at all, actually, and I think it's been a common complaint that 'annotate is basically 'cons with fewer utilities built up around it. That's a plus for me, I guess, lol.)
A good example of me considering an operation "unsafe" is earlier in this thread, where I discouraged the use of compound values as table keys. It just so happens Arc's table key comparison breaks the encapsulation of tagged types as well. :)
I agree -- while it would be sad if pg stopped using arc, that itself wouldn't be the end of the world.
The problem I see is that if pg publicly stops using arc, the already-slow trickle of newbies would slow even further.
Honestly, I'm starting to tire of this community being so small. I've been avoiding Clojure -- mostly because I really like Arc, partly because I've had bad experiences with some of the people in Clojure's community -- but Arc isn't getting to be much easier to get things done. And I'm not sure I'm going to be able to put it off much longer.
The problem with Arc as opposed to Clojure is that it is incomplete. It is missing some of the most basic features such as a a module system, a good debugger, and useful error messages. The reason none of us has come up with a solution to any of these deficiencies is that they are hard problems solve. We are all just waiting for pg to try to fix these problems instead of going out and solving them ourselves. If someone can solve these issues, pg or not, Arc should gain traction. I hate to say it, but if no one is going to do anything, Arc might as well already be dead.
I'm certainly not waiting for arc to become perfect. My side project may well end up needing me to reinvent a computing stack from scratch: http://akkartik.name/about. But I fully expect it to take 20 years, so don't hold your breath :)
Are you fully sure there is no other way to do what you want to do without it taking 20 years?
That's a question worth pondering, not just because big things tend to fail or good design is simple, but also because life has a way of changing what people choose to do over time.
Yeah, I'm actually constantly looking for shortcuts, ways to not need to reinvent the stack, ways to build things that will help existing projects, or allow people to use existing languages and tools.
When I said 20 years I wasn't saying I have a plan that's going to take 20 years. I meant that I'm building small projects that -- when I step back -- seem to be headed in that general direction over time. Projects like wart[1], tracing tests[2] and layers[3]. I'm building these little projects out in the open, so hopefully I won't fall victim to the "hacker comes out of basement after 20 years with a compiler and OS that nobody cares about" failure mode. No, my goal is to fail in some more creative way :)
I'm less worried about my goals changing. On the one hand, if they do they do, one can't be anything but fatalistic about it. On the other hand the goals I outlined above seem to have hooked into my motivation at a deep level. I may well end up switching solutions I explore, but it seems like the goal is unlikely to change.
I haven't really started building anything for the past couple of months, but I've been thinking lately about three potential little projects: a) a simple multitasking OS without virtual memory or processes, maybe even without preemptive scheduling, b) a better language for teaching programming by separating scope from the notion of functions, and c) an assembly language with structured programming and lisp macros somehow. Something 'easy' to build without needing an entire C compiler. Rephrased: "I'm willing to do my own register allocation. What can you do for me, lisp?" :)
Mostly I just get up every morning and work on whatever I want. And these are the things I've been ending up working on.
I've had bad experiences with some of the people in Clojure's community
When arc advances enough to appease you, it will also be appealing to others; then when a larger community develops around arc you will start to have bad experiences. That's the nature of mass appeal.
Even pg noticed the quality of HN's content and community degraded with popularity, which is why he's adding the pending change discussed (amongst others).
And let's all be honest, if arc solved half the problems Clojure handles, I doubt you would notice much of a difference between the languages. After all they are already very similar.
If you were to draw a line with liberal being at one end and conservative on the other, then plot various languages on that line you'll find arc and clojure would be close together as opposed to far apart.
Even Yegge's post would seem[1] to confirm this with his own data:
1. I assume arc to be like lisp. Many of the things that need a new interpreter atop scheme (lisp-1, quasiquote, unhygienic macros) are borrowings from lisp. The only major scheme-ism arc retains is call/cc. Am I missing anything?
2. I actually think arc is more batshit liberal than traditional lisp. Perhaps I'm reading too much into the absence of a module system, but I consider it a deliberate omission.
3. You're assuming that there's only one way to build out the missing infrastructure, and that is to be more like clojure. I'm skeptical of that.
Yeah, I can see how one could draw a little larger of a distinction between Arc and Clojure using the conservative/liberal categories, but even so I'm not convinced it even stands up as a good overall measure for similarity/differences (even if it's a good one). So setting that one criteria aside for a moment, I look at Arc and Clojure and here's what I also see:
- They are both Lisps with dynamic type handling, prefix notation, strings as sequences, heavy use of hash-maps, a large number of functions and macros with similar names doing the same things.
- The syntax's are so similar you could pretty much swap out color scheme templates (in fact I initially used the arc textmate bundle for Clojure).
- They both depend upon pre-existing language for compilation. Clojure drops into java, Arc drops into scheme.
- The online comparisons between them are more frequent than arc and any other language.
I could go on, but I think you get the point.
And while I know you could create a list to show differences I'm simply saying that overall they are very similar which was my initial claim.
As for "having to be more like Clojure" to advance, well no, but I'm pretty sure that it's more likely than any other unstated options. Here's what I do see to support this:
Ok, I'm convinced :) By pg's comments in particular.
(There's a lot of red herrings in your comment, though. That they look the most similar syntactically isn't too surprising, and it's irrelevant that erlang can't compete, or that they build on an existing language, etc. You're measuring position to compare trajectories.)
In any case, it feels quite delusional to compare arc to clojure at the moment :)
Let me ask you this: if you have clojure why do you care about arc? Genuinely curious.
> “There's a lot of red herrings in your comment, though. That they look the most similar syntactically isn't too surprising, and it's irrelevant that erlang can't compete, or that they build on an existing language…”
I realize many of my arguments have holes in them when viewed independently, but when put together those “soft arguments” contribute to painting an overall picture that supports my claim better than had I not given them. Albeit the "they build on an existing language" is pretty weak overall.
> “You're measuring position to compare trajectories”
Position is not something you can measure it's something you can map/use. That aside, I don’t believe what it is I think you intended to say is true. I used position to substantiate the claim of similarity, I used trends to substantiate the trajectory (i.e. the last half of my reply, what people are saying and what they are doing are trends).
> "if you have Clojure why do you care about arc?"
I like both languages. I don’t use Arc because its unfinished and unusable for the things I need to do. I often get the sense that when I promote Clojure on the Arc forum that people get defensive and wonder why I'm even around, but remember I am only saying what pg himself is:
> Would you recommend Arc to modern startups in general?
PG: No, I don't think so, not in its current state.
> Why not?
PG: It's still missing some things that most people take for granted.
> Are there any Lisps you would recommend.. ?
PG: Clojure is probably the best bet...
I like the arc language and I like the community. I view arc as unfinished and look forward to giving it a shot when it is, but until then I am going to continue to be realistic about the current state and promote Clojure as an option.
"I used trends to substantiate the trajectory, i.e. the last half of my reply, what people are saying and what they are doing are trends."
If you think arc is whatever pg says it is, then what we are saying and doing here is irrelevant :)
If you think arc is whatever we're saying and doing here, the things we take from clojure seem a tiny minority compared to the sum total of conversations. I'd say arc starts out kinda like clojure, but is looking to steal ideas from all sorts of languages including erlang.
---
I tried to phrase my question very carefully to avoid seeming defensive about something I don't care about. So let me just come out and say it: I have no problem with you talking about clojure all you want here. If you did, maybe I'd get to talk to you more! :) Arc is indeed absolutely unfinished, no disagreement there either.
Maybe what I'm actually defensive about is the prospect of any two languages becoming more and more similar. That just seems bad, nothing good can come of it. I'd rather see arc and clojure evolve in different directions and give me more ideas and more data about how good those ideas are.
Copying ideas and creating hybrids is totally fine, that's what we are good at. But then the hybrid starts at the intersection of its influences and sets off on a whole different trajectory.
So let me rephrase my question: is there something you miss from arc in clojure that has you wishing for a superset?
(The answer may take time to emerge from the subconscious. At least it has for me in similar questions.)
My comments are a targeted response to statements made in a specific comment (http://arclanguage.org/item?id=18421). I think you're treating the conversation in this thread as you treat this forum; as some place to mingle languages and abstract all ideas. That's fine if you want to do that, but I'll suggest you'll be more effective if you follow the thread and consider context when responding.
And I'm suggesting more care be taken in this regard, because it leads to you being offended (i.e. "If you think arc is whatever pg says it is, then what we are saying and doing here is irrelevant...") on statements that should be considered only in relation to the thread. It also leads to unintentional attacks such as "In any case, it feels quite delusional to compare arc to clojure ...". I don't think you realize you suggested I'm being delusional by adding that comment... Arc and Clojure are both modern lisps that have eliminated the overuse of brackets, so really is it delusional to suggest these two languages are more similar than not? Am I creating some injustice by telling someone, that's already leaning towards Clojure, that these languages are very similar anyway?
I'm going to end this thread here as I think it has already gone off the rails.
Sorry I'm causing offense. It was indeed utterly unintentional, and fwiw, I actually was never in the slightest conscious of feeling offended by anything in this thread. I'm entirely the offender and not an offended party.
I didn't mean to trigger associations like "injustice". When I said "delusional to compare arc to clojure" I was trying to be self-deprecating. Arc is just a toy, clojure is real. You're absolutely right in your defense of clojure.
Not to worry, I've known you long enough now to know you're not mean spirited or intending offence. Hopefully I didn't get too grumpy, but I needed to put and end the thread because I could see we had different ideas on what the thread was even about.
I can relate to the frustration. It's hard to predict when the next great stack will come along. I try not to make irreversible choices so I can jump to it as long as possible.
No danger of it happening here. Arc forum is running some ancient version from what I can tell. They probably learned to stop messing with it after the voting ring detector went haywire. I wonder how many of us remember that :)
