Arc Forumnew | comments | leaders | submit | shader's commentslogin
3 points by shader 13 days ago | link | parent | on: Advanced search for news.arc

This isn't actually related to the work you are doing, but I was curious why you chose to use notabug.org over, say, gitlab.com. I hadn't actually seen the service before you linked to it on this forum.

reply

3 points by hjek 13 days ago | link

I think Gogs/Gitea is interesting in that it's super lightweight and shipped a self-contained binary, that's up and running in seconds. (Gitlab recommends at least 4GB of free memory[0])

Perhaps that is not relevant when someone else is doing the hosting, but I've gotten used to Gogs/Gitea now.

[0]: https://about.gitlab.com/installation/

reply

3 points by shader 12 days ago | link

Interesting. That does sound somewhat attractive. I do occasionally try to run a git repository myself, and while I appreciate that Gitlab is open source, it was a bit cumbersome to set up.

I note that notabug said it was powered by a liberated version of gogs. What are they referring to there? Was something not sufficiently free about the original?

reply

2 points by hjek 10 days ago | link

Good question. I'm not sure, but I think it may have to do with removing Gravatar support?

reply

3 points by i4cu 13 days ago | link

Under knarks "Why the fork?" section hjek links to the 'ethical repository'[1]. Since notabug.com is only for open source projects then the repo's will likely be considered 'free software' which I believe makes it grade A in 'ethical repository' terms.

Personally I find the term 'ethical repository' offensive. It insinuates that non-free software is unethical when the majority of non-free software has no nefarious intent or code. Not exactly the greatest sales pitch in my book.

1. https://www.gnu.org/software/repo-criteria-evaluation.html

reply

3 points by shader 12 days ago | link

Yes, that's what I assumed as well; I just thought it would be an interesting discussion, and was wondering if there might not be other reasons for choosing that particular repository.

I suppose for me, the distinction between an 'open source' repository (Gitlab) and an 'ethical(?)' repository wasn't a very important one, so I was curious for the motivation behind it.

reply

1 point by hjek 9 days ago | link

I think free (or "open source") and ethical mean the same in most cases.

Exceptions might include something like Facebook, which is technically somehow usable w/o non-free JS when using their basic mobile web page, but where the company is still engaging in other unethical activities, like selling user data to sway elections.

Or something like Amazon, where you might possibly be able to buy something w/o non-free JS (haven't checked), but where the treatment of their employees is unacceptable.

But, I think, when we're talking git hosting sites, there's no difference?

But FSF considers Gitlab ethical enough for hosting GNU packages[0].

[0]: https://www.fsf.org/news/gnu-releases-ethical-evaluations-of...

reply

3 points by i4cu 8 days ago | link

As I understand it - the 'Open Source' movement concerns itself with improving the software by making the code openly accessible, where as the 'Free Software' movement concerns itself with a fighting for users rights (i.e. having the freedom to access, modify and distribute the code in a manner that empowers the user).

And so, an 'Open Source' repository holds code that is openly accessible for the purpose of improving the software. Where as an 'Ethical Repository' holds code that is graded by its' ability to guarantee users rights according to a specific set of morals (established by free software foundation). It so happens that open source repos tend to align well the ethics associated with free-software, but they should not be mistaken for each other. As an example to illustrate: If a repo SaaS were built for open source code, but restricted users from a certain country it wouldn't rank high in ethical repository grading. This is because while having the code openly accessible leans towards a Grade A rating (excellent), the restricting some users part puts it at a Grade F rating (unacceptable).

-- additional info --

https://en.wikipedia.org/wiki/Open-source_software

"Despite initially accepting it,[31] Richard Stallman of the FSF now flatly opposes the term "Open Source" being applied to what they refer to as "free software". Although he agrees that the two terms describe "almost the same category of software", Stallman considers equating the terms incorrect and misleading.[32] Stallman also opposes the professed pragmatism of the Open Source Initiative, as he fears that the free software ideals of freedom and community are threatened by compromising on the FSF's idealistic standards for software freedom.[33] The FSF considers free software to be a subset of open-source software, and Richard Stallman explained that DRM software, for example, can be developed as open source, despite that it does not give its users freedom (it restricts them), and thus doesn't qualify as free software.[34]"

reply

3 points by hjek 13 days ago | link

> It insinuates that non-free software is unethical when the majority of non-free software has no nefarious intent or code.

The term can also be used by people who consider it unethical to even give programmers the possibility to hide nefarious code from users, regardless whether they actually do or not.

reply

3 points by i4cu 13 days ago | link

> The term can also be used by people who consider it unethical to even give programmers the possibility to hide nefarious code from users, regardless whether they actually do or not.

But that's not what's happening here. They are categorically demonizing innocent people.

reply

3 points by hjek 13 days ago | link

> They are categorically demonizing innocent people.

I'm sorry; that was not my intention.

Perhaps I can make a comparison to clarify? As an example, some people think that guns are unethical because they may be seen as an unjust instrument of violence. Even if a particular gun hasn't killed anyone (yet), or even if most guns happened not to be used to kill, then surely it can still be legitimate for people to object to the passive presence of guns, because it gives gun owners the power to kill, and that power may be considered unjust by principle.

Similarly, some people think that non-free software is unethical because it gives programmers the power to do bad stuff, regardless of whether some particular non-free program is actually malware (yet).

(Sorry in advance if I've derailed this discussion into a more controversial subject.)

reply

3 points by shader 12 days ago | link

(continuing the discussion for clarity... no emotional connotation is intended)

I realize this is a comparison for clarification, but isn't it still just 'categorically demonizing innocent people'?

You picked a more controversial topic where more people are likely to agree with the demonization I suppose, but your assertion that the power to kill "may be considered unjust by principle" is not well supported by vague assertions that "some think" guns "may be seen as" unjust instruments of violence. I fully support everyone's right to object to something they see as dangerous; opinion does not constitute principle, however.

To me, something is just or unjust based on whether or not it aligns with or infringes anyone's rights. So, I suppose I might actually agree that a power could be "unjust in principle" if it could be shown that the power could not be used justly - that is, without infringing on anyone else's rights. For some powers, mostly political ones, this is the case. In this case I think guns may be a poor comparison, because they actually can be used in ways which are just (defense, etc.), even if you believe that those cases are unlikely and so desire strict gun control, etc.

In contrast, it may be that producing nonfree software is always 'wrong' (in that it infringes on the supposed rights of the users to understand and modify the program they are running) and therefore having or providing the power to do so would be 'unjust in principle'. If the concern is merely that some may produce malware, and there are actually legitimate reasons for producing nonfree software, then it is not unjust in principle to do so, or to provide someone with said power.

I hope I've understood all that correctly, and restated it well. I'm not sure that I agree with the idea that nonfree software is always bad, but I am open to it. Perhaps what I'm missing is a clear understanding of the specific rights that nonfree software violates.

reply

3 points by i4cu 13 days ago | link

No worries, I know you (as well as the authors) are simply trying to apply implicit safety measures to counter bad actors. And I'm certainly not offended by you adopting the program. It's my feeling, however, that their approach is horribly wrong and bordering on corruption. I simply don't believe they will have any success when trampling over the good actors in their process of trying to better the world. IMO; If they really wanted to make a dent, they should push for a regulation requiring that browsers provide functionality that enforces a free-software configuration OPTION. Then allow society to decide for themselves (this is a free world after all). I'd even be ok if the default setting was on. But as it sits right now they will get nowhere really fast.

edit: oh and as for the gun analogy... I'm from Canada and fully support gun control (we have it), but I'm not going around and implying that every gun owner is unethical in the process of asking for gun control. That would be shooting myself in the foot!

reply

2 points by hjek 10 days ago | link

> IMO; If they really wanted to make a dent, they should push for a regulation requiring that browsers provide functionality that enforces a free-software configuration OPTION.

Sounds interesting. Apart from the regulation part, it sounds a bit like LibreJS[0].

[0]: https://www.gnu.org/software/librejs/

reply

2 points by i4cu 9 days ago | link

Actually, I got the notion from Stallman's original post 'The Javascript Trap' [1].

"Finally, we need to change free browsers to detect and block nontrivial nonfree JavaScript in web pages. The program LibreJS detects nonfree, nontrivial JavaScript in pages you visit, and blocks it. LibreJS is included in IceCat, and available as an add-on for Firefox."

However I am opposed to that call for action given it's an all-or-none implementation. I feel it's the role of each country to regulate, which is why I expressly suggested it as a configuration option (ideally it could be enforced at the browser level country by country and if not then user by user).

1. https://www.gnu.org/philosophy/javascript-trap.html

reply

1 point by krapp 9 days ago | link

It seems like the thesis here is that whether or not "non-trivial" Javascript (which is just about all Javascript in the wild) should be trusted depends on the presence of an explicit GPL license. If so, that doesn't seem like a reliable heuristic for a script blocker to me.

reply

2 points by i4cu 9 days ago | link

I'm pretty sure it would be similar to ad-blockers. The initial implementations are trivial and easily circumvented, but as they evolve they become more useful overall.

Plus note that I was just suggesting that it would be more effective than a social movement with 'ethical repositories'. Just imagine if the ad-blocker devs tried the same strategy...

reply

3 points by i4cu 13 days ago | link

eh, hem, hem!

I've made an assumption as to why hjek selected notabug. I should know better than to make assumptions... apologies.

reply

3 points by hjek 13 days ago | link

Well, correct assumption!

reply

3 points by shader 16 days ago | link | parent | on: Installation on windows?

I think if we fork the community site to run on anarki. which I think is more likely than being given control over the Arc Forum, we should consider ways to archive and bring forward all of the stuff on the existing arc forum. It shouldn't be too hard to crawl the forum, though I think there might be some DoS prevention that would slow it down.

reply

4 points by shader 17 days ago | link | parent | on: Self-hosting the Anarki community

For anarchic service management, what do you think about trying to set it up so that the service is automatically updated if the latest commit passes some form of CI testing? The CI tests themselves could be updated if the new version passes the latest working version of the site.

It would still be possible for someone to break things in two passes of "delete the tests" then "delete the server" but regular automated backups should mitigate the risks somewhat. And we haven't had many problems with spammers anyway...

reply

5 points by i4cu 16 days ago | link

Given the open nature of the anarki repo, it's likely that news will break. And when it does we wouldn't be able to discuss it.

So unless these tests could prove that the forum would work (which is highly unlikely) then my vote would be not to do this. It's akin to putting the services issue logging/tracking system under the same service [1]. It's a bad idea IMO.

1. http://arclanguage.org/item?id=20573

reply

4 points by shader 16 days ago | link

Or maybe the immediate and painful result of breaking the forum would motivate us to be more careful and fix the issues more quickly. It probably won't happen that often anyway.

I was about to say that an outage might risk killing the community, which would be bad, but 1) we still have this forum and GitHub (as krapp points out), and 2) if the community is really so weak that it can't revert a commit in order to get the forum running again, it's probably not worth hosting a separate site anyway.

The idea is growing on me, just because of how audacious it is. (^^)

I do agree with some of your point though; it would be good to have some separate logging and bootstrap systems in place so that we can detect and repair faults more easily, without the intervention of a specific admin. For one thing, the software that pulls the changes will probably not be arc-based, so it should still be running even if the forum goes down. Secondly, we could try to set it up so that it always pulls hotfix patches immediately when the logging / monitor system indicates failure.

Also, marking a particular branch (probably not master) as a more 'stable' version might be good.

reply

2 points by i4cu 15 days ago | link

Another option is to ensure the service has a robust failover procedure towards a secondary free service as a temporary measure. And maybe someway to safely automate an intentional failover.

edit: my original comment was in consideration of the arc forum potentially going away. Honestly I'm not sure I would move over if that wasn't the case. I'd have to see :)

reply

3 points by shader 13 days ago | link

Yeah, that's part of why I've never seriously considered it before.

The only reasons for thinking about it now are that 1) we want to add some features to the forum, and there's no way to test them here, and 2) it isn't actually a bad idea to have a community site for Anarki. The risk of weakening the community has deterred me from the idea of forking the arc forum, but if we still treat this as the 'official' arc community, and make a separate site more focused around anarki, I don't think that would be too bad.

It might actually help some, since separating more could allow us to really focus on and develop our unique points of experimental language hacking.

reply

2 points by i4cu 13 days ago | link

Don't let my comments stop you. Your thinking is quite valid. I'm just trying to contribute my opinion in hopes of helping you shape whatever you decide.

> It might actually help some, since separating more could allow us to really focus on and develop our unique points of experimental language hacking.

As far as I can tell, pretty much everyone has moved over to anarki, so I don't understand your comment. How does creating a separate forum for anarki help to "focus on and develop our unique points of experimental language hacking"?

reply

3 points by shader 12 days ago | link

Thank you for your opinions; it really does help me clarify my own thoughts. That and I rather dislike talking to myself for more than a few minutes at a time...

Currently, this is an Arc forum. People are drawn here from pg's posts about Arc, and their attraction to the simplicity and beauty of the language. Also probably dreams of silver bullets... That won't stop just because we make a separate community, and I really don't think there's anything wrong with it.

However, I think we're also torn a bit between maintaining basically a bugfix version of arc, or going on to develop it further.

The question then is what "further" means - which direction would it go? One answer is that it can't really be predicted; creating such a community would be a way to find out.

On the other hand I can speculate, based on what I've seen of this community so far.

People come, entranced by pg's vision of arc, and then find out that the code base itself is really small and approachable. In general, lisp variants are used by people who like to make the language fit their needs, but where most dialects would add to the language via macros, with arc (or anarki) people find it just as easy to hack on the core of the language itself. Eventually this leads them to forking it, or building their own implementation in javascript or something.

I suspect this is partly because arc is so small, and does not have many standard libraries or a package system. You can read it all in a fairly short amount of time. So the fact that most of the code is aimed at developing arc means that the easiest thing to develop with it is... arc.

This trend of exploration and extension will undoubtedly continue, and I think it will be much more free to develop into something significant if we simply look at anarki with the slightly different perspective that having a separate community site we could actually upgrade might offer. Instead of being weighed down by arc as the 'community bugfix edition', it could become a 'language based on arc' with a solid foundation, but room to grow.

The core paradigms and strengths of arc appear to be "exploratory programming", and "language hacking". I think it would be cool if we could develop the former beyond the latter, but who knows how it will turn out?

Of course, I may be entirely off-base here myself...

reply

3 points by i4cu 12 days ago | link

Ahh, I see. That make sense now.

Hmm, you know, a new anarki forum could have the benefit of adding a tags feature for posts. A few good tags could be 'lang-design', 'arc', 'anarki', 'help', etc., etc. Not only would this allow our subtypes of members to zero in on their content of interest, but would make searching for meaningful info much easier. Now that would probably make me jump over.

reply

3 points by shader 12 days ago | link

Yep, improving the forum would help a lot, in a lot of ways.

I've often thought that the structure of the news.arc forum is rather unhelpful for the arc community, especially now that we're so small and lethargic. Conversations can go extended periods of time without comment, so they get locked. Or they fall off the front page, and hard to find again. Neither is conducive to long-term development and improvement; we probably lose a lot of valuable work and ideas that way.

One thing I'm considering as part of developing a new community site is collecting and archiving all of the arclanguage.org content, so we can actually access it. And preserve it, if the site goes down.

But I end up wasting all my time on discussion, instead of actually making progress on that...

reply

3 points by shader 10 days ago | link

Somehow i managed to trip the DoS prevention, and now my home IP is blocked. I almost thought the site actually did go down...

reply

3 points by shader 3 days ago | link

Anyone know how long an IP stays banned? Or what I can do about it? Rather inconvenient not to be able to check the forum from home...

I suppose I could set up a proxy or something. And I was planning on scraping everything to a new community site anyway, so maybe I should take this as the incentive to do so.

reply

2 points by hjek 2 days ago | link

Amazing you managed to get your IP banned!

Hacker News has an IP unpanning procedure[0] but I don't think Arc Forum has one.

In the Arc 3.1 code there is a function `set-ip-ban` for unbanning users, but no `unban` op.

(Someone should add that to Anarki, actually.)

[0]: https://news.ycombinator.com/item?id=4761102

reply

3 points by i4cu 2 days ago | link

Looks as though it ranks how bad you are and always keeps the baddest of the bad-asses in cache, while never deleting any from disk. In a low volume site like this I doubt you'll get out of it without contacting them.

reply

2 points by akkartik 2 days ago | link

Hmm, I wonder if it stays banned until they restart the server. I'd ping hn@ycombinator.com.

reply

2 points by hjek 2 days ago | link

Looks like banned IPs are written to the disk even:

    (def set-ip-ban (user ip yesno (o info))
      (= (banned-ips* ip) (and yesno (list user (seconds) info)))
      (todisk banned-ips*))

reply

2 points by akkartik 12 days ago | link

I did ping the HN admins about the lock period a year or two ago, and they were kind enough to extend it for us. It's now 90 days, if I recall correctly.

reply

2 points by akkartik 13 days ago | link

We still get newcomers with questions about Arc 3.1.

reply

2 points by i4cu 13 days ago | link

Sure, but from what I can see they soon quickly discover they need to use anarki and move over.

And impacting newbies does not appear to be considered in "focus on and develop our unique points of experimental language hacking". So...

edit: maybe I'm wrong, but it seems to me what he really wants is a language design group. And I'm fine with that, but I think it's wrong to conflate anarki users with language designers. They are not one in the same.

reply

3 points by shader 12 days ago | link

I was expecting arclanguage.org to stay mostly as it is. Support for newcomers to arc would be included in that.

You may not be entirely wrong, but I probably shouldn't have tried to compress a description of a language used and developed by a loosely federated group of unique individuals into a single phrase.

I can try to unpack it a bit...

"Experimental (language) hacking" -> Exploratory programming is supposedly a primary paradigm of arc

"Experimental language hacking" -> Arc isn't exactly production ready; it's a very experimental language, and that makes it fun (and sometimes frustrating) to use, and easier to explore new directions and possibilities. You're less likely to reverently assume that the way it is is the way it must be.

"Experimental language hacking" -> And yes, we hack on arc itself. So I am thinking a bit of a language design community, I guess. In an anarchic language community though, the lines between 'users' and 'designers' become rather vague...

reply

2 points by shader 12 days ago | link

Yep. We should definitely keep answering them too.

reply

3 points by krapp 16 days ago | link

If that's an argument against us using Anarki then it also seems like an argument against anyone using Anarki, or at least against anyone blindly committing it to production. Breakage can happen with any open source project, but given the generally slow nature of the community, even if news is likely to break, it isn't likely to break often.

Also, we already have Github to check and discuss it, and I think there is a more appropriate venue than here for those issues.

reply

2 points by shader 19 days ago | link | parent | on: Hook

The idea behind a 'hook' is that it is a place where code can be inserted without having to modify the original code. They're pretty common in emacs, for instance, where lots of packages provide hooks for easy extension in your .emacs file.

Arc hooks are apparently pretty lightweight:

  (= hooks* (table))

  (def hook (name . args)
    (aif (hooks* name) (apply it args)))

  (mac defhook (name . rest)
    `(= (hooks* ',name) (fn ,@rest)))
(from https://github.com/arclanguage/anarki/blob/7a1fba03b6faaa06f...)

So, if there's a function bound to that name in the hooks table, it is run with the arguments. You only call 'defhook when you want to bind to that point in the code.

This is a rather interesting implementation to me, because I'm used to the emacs-lisp concept, where a hook is actually a list of functions that all get called on the arguments, so it can be added to multiple times. The corresponding names in elisp are 'run-hooks and 'add-hook. https://www.gnu.org/software/emacs/manual/html_node/elisp/Ho...

reply

2 points by hjek 19 days ago | link

Ok, makes sense. Thanks for the explanation!

So I guess it's on purpose that `(hook 'somewhere)` doesn't do anything until you define a hook for that place yourself, e.g.:

    (defhook somewhere () (pr "Hello from somewhere"))

reply


True, Guile and Picolisp do not make self-contained binaries, but they're pretty lightweight and fast, and seem like they still might be half-decent choices. I guess I should probably remove Guile from the list of candidates, since it's really for plugin extensibility in other programs; I may even have been slightly overlapping it with Chicken in my mind.

Arc would have to be "polished" a bit to make it work, but that's what we're here for, isn't it?

reply

2 points by akkartik 19 days ago | link

I'm not sure system programming is a good domain for Arc. Why not just Racket directly? The interpreter makes Arc quite slow.

reply

2 points by shader 19 days ago | link

Yeah, I probably wouldn't use it for that either, at least as much because of the relative instability and hackishness of arc. It was fun for a few minutes to imagine modifying it to be used that way though.

reply

2 points by rocketnia 18 days ago | link

Interpreter? Are you talking about Racket's bytecode interpreter?

reply

1 point by akkartik 17 days ago | link

No, I was forgetting that ac is considered a compiler :) But it compiles the Arc codebase every single time Arc starts up. Maybe we should start memoizing its outputs to disk somehow, see if that makes it noticeably faster. My suspicion is egregious runtime processing like ar-denil-last and ar-apply-args will cause it to not make a difference.

Given the pervasiveness with which Arc has to make such transforms at runtime, I've gotten into the habit of thinking of it effectively as an interpreter.

reply


Sounds pretty interesting.

I have this constant problem of dissatisfaction with existing tools and paradigms, to the point that I often work my way down to the bottom of the stack considering building a new OS...

My most recent descent has started with a dissatisfaction of container orchestration tools, then discovering that it's not really easy to manage containers with unprivileged code under Linux, and now wondering about building a microkernel OS / language for distributed actor-model development from the ground up, and thinking it would be nice to have an "OS development framework" so it is easier to experiment with novel architectures without having to deal with a lot of driver development grunt work... I originally asked this question somewhere near the beginning of all that.

reply

2 points by akkartik 19 days ago | link

You're speaking my language :)

Speaking of building a new OS, check out https://gitlab.com/giomasce/asmc which uses some of the bootstrappable infrastructure to build an OS. It uses a stack-based (Forth-like) intermediate language called G to implement C.

reply


Well, that was more of a self-directed comment that if I'm using a custom language for making my projects, the odds that anyone else will want to contribute to it are rather low.

reply

2 points by akkartik 19 days ago | link

The reason I asked: I actually don't think open source projects are designed for others to contribute to :)

https://www.reddit.com/r/ProgrammingLanguages/comments/8i33h...

reply

2 points by shader 19 days ago | link

Interesting that you mention the sandboxing problem, since that's something I'm currently thinking about...

My current thought for solving the sandbox problem is to have a thoroughly first-class and recursive system for creating subsets of resources and providing them to children. Fortunately, I don't think we ever have to worry about running _unexpected_ code; just code that does unexpected things. So if we have the tools to take part of the resources available for the current "process" and pass them on to any child interpreters (tabs, etc.), then it wouldn't matter how many layers you decide to add to the system.

That said, a lot of complexity with sandboxing comes from the desire to deliberately break isolation and provide access to shared resources... I think that temptation should be avoided as much as possible, and solved with easy to use communication primitives. While that does impose an overhead, we're already quite used to isolation overheads, and it should be constant. And if the abstractions around resources are good enough, the difference shouldn't be that noticeable.

And now you know why I'm thinking about an actor-model OS, where everything (including files, etc.) are represented as isolated actors that communicate via messaging. Then it doesn't make a difference whether you "own" the 'filehandle' or a different actor does - it's just an address that you send authorized messages to anyway. Fits pretty well with microkernel and distributed OS design too.

reply

2 points by akkartik 19 days ago | link

"My current thought for solving the sandbox problem is to have a thoroughly first-class and recursive system for creating subsets of resources and providing them to children."

I hadn't been thinking about sandboxing, but my interest in testable interfaces lately has me asking how I can design an interface for memory allocation in a testable way. And that requires precisely such a recursive way for one allocator to be able to do book-keeping on memory allocated by a parent allocator. This would be the start of the capability to run the tests of an OS while running on itself.

The only way to have an address tagged as used in one allocator and free in a child allocator: metadata must be somewhere else. I'm imagining something like this, where 'the 'allocator' blocks are book-keeping metadata on used vs free areas:

    +---------------------------------------+
    |                                       |
    |              Allocator                |
    |                                       |
    +---------------------------------------+
    |                                       |
    |              Block 1                  |
    |                                       |
    +---------------------------------------+
    |              Block 2                  |
    +---------------------------------------+
    |              Block 3                  |
    |                                       |
    +---------------------------------------+
    | Block 4  +--------------------------+ |
    |          |        Allocator         | |
    |          +--------------------------+ |
    |          |        Block 1           | |
    |          +--------------------------+ |
    |          |        Block 2           | |
    |          +--------------------------+ |
    |          |        Unused            | |
    |          +~~~~~~~~~~~~~~~~~~~~~~~~~~+ |
    +---------------------------------------+
    |              Block 5                  |
    +---------------------------------------+
    |              Unused                   |
    |                                       |
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
Anyway, if I have a bootstrappable stack to start with, the timeline for this allocator just got accelerated!

reply

3 points by shader 19 days ago | link

Yeah; I guess I didn't state it that clearly, but the sandboxing reference was from the reddit thread you linked

"Sandboxing is challenging partly because our boundaries for what is trusted shift imperceptibly over time. Browsers came out in a world where the desktop was considered the crown jewels, while browsers were considered untrusted. Over time local disk has disappeared (e.g. Chromebooks) while our lives are ruled by the browser. This has made the original design of browser security (shielding the local file system; cookies; single origin) if not obsolete then at least dangerously incomplete. So any sandboxing solution has to think about how to fight this gradual creep. No matter how you organize your sandboxes using Qubes or whatnot, something inside a sandbox is liable to become more important than other stuff inside that sandbox. And even outside that sandbox."

The idea of running tests against a live application, particularly an OS, is intriguing. It shouldn't be too hard though, since the code and its state don't have to be in the same place. So you could run the live code against separate testing data in a different sandbox or something...

Your specific example of memory management makes me think that vague "recursive subsets of resources" probably won't cut it for a lot of things. After all, if you make memory access too indirect, it would be very very slow; but if you try to use the actual CPU management interfaces, it may not allow the kind of control that we want...

How much do you really need the bootstrappable stack? There shouldn't be anything stopping you from using something that's already been bootstrapped, instead of raw hex code. E.g. existing C tools, etc. There's also the picolisp-based PilOS project; basically picolisp (which is mostly bootstrapped from an assembler they wrote anyway) running directly on the metal as a kernel. (https://picolisp.com/wiki/?PilOS)

reply

1 point by akkartik 19 days ago | link

Oh sorry, I did understand your sandboxing reference. Just meant I wasn't concerned with it when thinking about the memory allocator. Should have said, "I haven't been thinking about sandboxing.."

I'm less concerned about overhead because I'm primarily concerned about tests, which by definition don't run in production.

But I think in the case of memory allocation you can do several levels of sandboxing before you start hitting performance bottlenecks. You'd have no extra indirections when using memory, because its just a pointer and you just have an address. You'd only hit extra levels of indirection when you try to allocate and the allocator is out of memory. Then you may need to request more from your parent, who needs more from its parent, and so on, all the way down to the OS.

---

"How much do you really need the bootstrappable stack?"

My basic idea is that code is easier to quickly understand if you can run it and see it in operation in a variety of tiny slices. I think it would be easy to quickly modify codebases if you had access to a curriculum of logs of the subsystems you care about, click on any line to jump to the code emitting it, and so on. The curriculum would be created automatically while running tests (because tests are selected manually to capture interesting scenarios).

This idea requires being able to drill down arbitrarily deep into the traces, because we can't anticipate what a reader is trying to change. I may be trying to investigate why a certain sequence of operations causes my laptop to lock up. That may require drilling down inside a save operation to see the data structures inside the OS that perform buffering of disk writes. And so on.

Compilers are part of this infrastructure I want to drill down into. If I want to understand how the if keyword works, it does me no good if it terminates in some opaque binary that wasn't built on my computer. It's also only mildly nicer to find out that some code in this compiler I'm reading requires understanding the whole compiler. Metacircularity is a cute trick, but it's hell on comprehension. Any sort of coiling is bad. What noobs need is to see things laid out linearly.

I've looked at PicoLisp before. I even modeled Wart's memory allocator on PicoLisp. So it's great in many ways. But like all software today it's not really intended for end-users to look at the source code. Its code is intended for insiders who have spent months and years building up an intimate understanding of its architecture. That assumption affects how it's written and managed.

reply

2 points by shader 18 days ago | link

"its just a pointer and you just have an address"

That could be true; I'm not really familiar with CPU facilities for memory isolation, but this is probably one of the most solved of the isolation challenges, since it does require CPU support.

"You'd only hit extra levels of indirection when you try to allocate..."

Good point. I wonder if there's any way to improve that, or if the worst case is rare enough that it's acceptable?

---

"This idea requires being able to drill down arbitrarily deep into the traces..."

That's a cool idea, and fits very well with the GNU objective of fully open and transparent source code. I'm not sure that bootstrapping is what is required to achieve that goal, however. What you really need is transparent source all the way down, which can be satisfied with self-hosted code, even if it's not bootstrapped. In fact, I'd argue that multiple layers of bootstrapping would make the drill-down process very challenging, because you'd have to cross very sharp API boundaries — not just between functions or libraries, but between languages and runtime environments. Making a single debug tool that handles all of that would be impressive, let alone expecting your users to understand it.

"Metacircularity is a cute trick, but it's hell on comprehension"

Metacircularity applies to interpreters built on interpreters, not compilers. I can agree that it makes things opaque though, because it reuses the existing platform rather than fully implementing it. A self-hosted compiler, even if written in its own language, could be fully comprehensible, however (depending on the quality of the code...). It's just a program that takes syntax and produces binary.

Interestingly, while writing this, I ran across the following paper, which may be of some interest: Avoiding confusion in metacircularity: The meta-helix (Chiba et al.) (https://pdfs.semanticscholar.org/4319/37e467eb9a516628d47888...) I'll read more of it tomorrow, and possibly make a separate post for it.

I do think it would be really cool and possibly also useful if every compiler was self-hosted, and could be drilled into as part of the debugging process. It would mean that you could read the code, since it would be written in the language you are currently using, and the same debugger should be able to handle it.

---

"But like all software today it's not really intended for end-users to look at the source code"

I haven't actually looked at picolisp's source much myself, but what would it take to satisfy your desires for "end-user" readable code?

reply

2 points by akkartik 18 days ago | link

Very interesting. I'm going to read that paper as well. And think about what you said about the difference between interpreting and compiling.

> what would it take to satisfy your desires for "end-user" readable code?

See code running. My big rant is that while it's possible to read static code on a screen and imagine it running, it's a very painful process to build up that muscle. And you have to build it from scratch for every new program and problem domain. And even after you build this muscle you screw up every once in a while and fail to imagine some scenario, thereby causing regressions. So to help people understand its inner workings, code should be easy to see running. In increasing order of difficulty:

a) It should be utterly trivial to build from source. Dependencies kill you here. If you try to install library version x and the system has library version y -- boom! If you want to use a library, really the only way to absolutely guarantee it won't cause problems is to bundle it with your program, suitably isolated and namespaced so it doesn't collide with a system installation of the same library.

b) It should be obvious how to run it. Provide example commands.

c) There should be lots and lots of example runs for subsets of the codebase. Automated tests are great for this. My approach to Literate Programming also helps, by making it easy to run subsets of a program without advanced features: http://akkartik.name/post/wart-layers. https://travis-ci.org/akkartik/mu runs not just a single binary, but dozens of binaries, gradually adding features from nothing in a pseudo-autobiographical fashion, recapitulating some approximation of its git history. In this way I guarantee that my readers (both of them) can run any part of the program by building just the parts it needs. That then lets them experiment with it more rigorously without including unnecessary variables and moving parts.

More reading:

* http://akkartik.name/post/readable-bad

* http://akkartik.name/about

reply

2 points by shader 19 days ago | link

I think the problem is probably a combination of lack of motivation and barriers to contribution.

On the one hand, most projects are not very well documented, advertised, supported by tests, or have communities ready to help people get started in development. So the difficulty of getting started helping is probably high.

On the other hand, most projects are the unique vision of a single developer. Everyone else probably has different ideas, and other projects they'd rather work on. It takes a certain kind of charisma, influence, productivity, etc. to get past the point where people feel like contributing to _your_ project is worth _their_ time. It would be interesting to figure out exactly what it takes to make that happen, so I could use that power for my own nefarious ends...

reply

4 points by shader 20 days ago | link | parent | on: Anarki Reference Documentation

Nice work!

To perhaps start a discussion instead of just leaving a comment: where should we go next with the self-documenting / exploratory code ideas? What's missing or frustrating about the current help system?

reply

5 points by shader 221 days ago | link | parent | on: Yet another post regarding Arc & Heroku

I've recently discovered the Caddy server (https://caddyserver.com/), which makes SSL and application proxy deployments super easy. Like, 2 lines of config:

  domain.com               #uses the domain to automatically set up SSL with Let's Encrypt
  proxy / localhost:8080   #redirect everything to Arc on 8080
I will say that Arc runs a bit resource intensive, and the slightly slow boot times mean you don't want it to have to re-launch because of infrequent requests. I don't know how well it would work on Heroku.

Also, some VPS services like vultr.com offer $5/mo nodes that have more resources than what you get from Heroku at $7/mo anyway.

-----

4 points by shader 220 days ago | link

I should mention that if what you want from Heroku is their deployment process, you can actually replicate some of it pretty easily with Caddy (though I have not done so yet myself; I plan to soon...)

Specifically, they have support for automatically fetching code from git and running a command in the repo, either periodically or triggered by a webhook: https://caddyserver.com/docs/http.git

That doesn't get you the app ecosystem that heroku offers, but you can get a lot of that pretty easily via docker and docker-compose now.

-----

4 points by shader 231 days ago | link | parent | on: Next steps

> Unvote (like HN)

That would be nice I suppose, but we can't actually modify the arclanguage forum, so it would be of limited benefit. I guess a lot of people come to arc because they want to run an HN clone though, so it might be worth pursuing.

> File upload

Yeah, I think enhancing some of the fundamental web-service functionality in arc would be good. Beyond file upload, better support for OAuth, ssl, etc. would be good too.

> Ecommerce

Good idea. Not sure how to make that better myself, but demystifying and enabling easy ecommerce would be cool.

I think that also has the same motivation for the previous category; we need better support for web application development, because that's what most people want to do.

-----

3 points by hjek 229 days ago | link

It's true that unvote wouldn't make it to Arc Forum as it's never updated, but the same is true for any new feature, sadly.

Regarding SSL, I've been working on an example Nginx + LetsEncrypt secure reverse proxy configuration for Arc, https://github.com/arclanguage/anarki/blob/master/extras/new...

It's just insane that Arc Forum doesn't encrypt HTTP but, well, as you write, we can't change Arc Forum. Maybe they'll update it some day, or someone will set up an unofficial one that's more up to date?

Regarding web application development, there's already an interesting web based app development interface, https://github.com/arclanguage/anarki/blob/master/lib/prompt...

It looks quite humble but it has both a repl and an interface for saving/running various web apps.

I've set it to be enabled by default in Anarki. And it's only available for admins, because it can run system commands. I wonder if it would be possible to allow any user to develop apps securely, e.g. by disabling unsafe commands or sandboxing it somehow? Perhaps there is a simple way of doing that? It could open Arc web app development up to any user of an Anarki driven page.

I think those features mentioned above still need better documentation, too.

Other HN features that Anarki News is lacking:

* favourites

* hide item

* past front pages

* show all stories from same site by clicking on domain name

-----

More