Arc Forumnew | comments | leaders | submitlogin
1 point by rocketnia 4154 days ago | link | parent

Hmm, actually, my definition of "dynamic" scope might be "not generally possible to associate a usage site with a particular definition site at compile time." I've referred to mutability as a kind of dynamic scope before.

This definition connects with the "not at compile time" meaning of the word "dynamic," and I wonder if it was one of the main complaints lexical scope advocates had against the status quo. I seem to remember reading about that history somewhere, but I don't have a citation, so I could just be making up rumors. :-p



3 points by Pauan 4154 days ago | link

* [1]: Because I have had a hard time using the Arc Forum lately, akkartik, myself, and rocketnia had an e-mail conversation. I compiled this conversation into an HTML page where it can be read:

http://akkartik.name/2012-11-11-lexical-global-scope.html

Thanks to akkartik for hosting it.

---

rocketnia: "Hmm, actually, my definition of "dynamic" scope might be "not generally possible to associate a usage site with a particular definition site at compile time." I've referred to mutability as a kind of dynamic scope before."

I'm fine with that definition.

---

rocketia: "We're arguing semantics here, but the way I see it, "lexical" means "local to a chunk of code," and "dynamic" means "local to a run time stack frame and its children." "Global," on the other hand, means "local to nothing.""

I'm not a fan of those definitions, in large part because the way I'm implementing Nulan is based on first-class environments, and thus "dynamic" means "evaluated in the environment where the function is run" and "static/lexical" means "evaluated in the environment where the function is defined".

I think that's more useful conceptually, rather than bringing in things like stack frames and a distinction between global/local: Nulan treats global environments exactly the same as local environments.

---

rocketnia: "Racket modules don't provide that feature by default, but the person defining the module does have the option to provide it. This makes it easy for the author of an existing module to introduce mutability without breaking backward compatibility. Since you like immutability by default anyway, I'm surprised you take the opposite stance for module-level variables."

Yes and I don't like having to go through those extra hoops. I also dislike the inconsistency. Which is why Nulan uses purely static environments everywhere, and it lets you assign to variables defined in another module. I think this is simpler to reason about and is more powerful too.

As for immutability... that's not correct. I don't value immutability by default. I value immutability for practical reasons. All of Nulan's datatypes (cons, object, number, environment, etc.) are immutable because I've found that to be remarkably useful. But I've come to the conclusion that making variables mutable by default (like Racket and Arc do) is perfectly fine, from a practical perspective.

And even if I did decide to make variables immutable by default, I'd still need to provide some way to easily create mutable variables in the situations where they're useful. And I'd like to allow for modules to modify these mutable variables even if they're defined in a different module.

You might be thinking, "but what if I want to write a module that doesn't allow for mutation?!" My solution is very simple:

  (var foo 5)

  ... do stuff with foo ...

  (const foo foo)
The "var" function creates a mutable variable and the "const" function creates an immutable variable. So, the above first creates a mutable variable, which can be mutated by the module. At the end of the module, it replaces it with an immutable variable, and due to the static global scope, this works out perfectly fine.

In fact, you could make this even stricter and delete the variable entirely, effectively making it private to your module:

  (var foo 5)

  ... do stuff with foo ...

  (unvar foo)
And now you have created a global mutable variable "foo" which is private to your module. And once again, this works just fine because the global variable is static. And if you would prefer to not put "unvar" at the end of the module, you could use something like "exclude":

  (exclude {foo}
    (var foo 5)

    ... do stuff with foo ...)
I think this is a better tactic than Racket's "you have to jump through some extra hoops to mutate things".

---

akkartik: "I don't understand how a global binding can have a lexical scope if it's not attached to a block. Do you want it attached to a file/module/namespace by default?"

Sorta. In this thread[1] I already explained my views, but to summarize, I think functions should only be able to refer to variables that exist at the time the function is defined. In other words, no late binding of global variables.

Alternatively, you could go with Racket's system which uses lexical scope at the module level, and uses modules for everything by default. That works just fine too. The important thing isn't which particular lexical scope system your language uses, the important thing is that it doesn't use dynamic scope by default.

-----

2 points by rocketnia 4154 days ago | link

This is an extremely clarifying post. I think it would have headed off a few of the misunderstandings we had over email. XD

Now to reply to the later parts of that email exchange...

---

Me: "Pauan, when you show code examples that demonstrate how you'd deal with certain library dependency corner cases in Nulan, are you considering that the application developer can't go in and change the code of the existing libraries?"

Pauan: "Yes, definitely."

Then I'll respond to your examples individually.

Regarding "two libraries [...] use the same namespace and have name collisons," we've already started to talk about this case, so I'll skip it here and address it below.

Regarding "two libraries, each of which use different, incompatible versions of some third library," the code you posted doesn't comprise a solution. How do you keep library1 and library2 from using the same version of library3?

Regarding "they'd like the new library to be able to occupy both orignal spaces," your code is again incomplete. If the two originals are library1 and library2, an intermediate library library3 depends on library1, and the new one is library4, then how do you get library3 to start using library4 instead of library1? Keep in mind you can't modify any of the libraries.

Regarding the licensing example with "they'd like it to be a drop-in replacement [...] An application I'm working on uses two libraries, one of which requires the replacement library [...] The other [...] uses the original basically by default": I think they're asking for a mechanism that lets the application choose whether the "other" library uses the original or the replacement. I don't see how your code makes this possible, and your "this could instead import replacement" comment suggests that you'd actually modify library2.

Regarding "importing a package should only introduce the names that library explicitly exports," I agree you've got the important parts of this covered.

Regarding "conrol the names you import [...] such that [...] a new export to an existing library will not affect the meaning," it looks like you do have a good response to this. Personally, I wouldn't want there to be any imports that could possibly break that way, but I guess I'm more picky than they are. :)

---

Pauan: "That's the point of lexical scoping: they can use the same names and not collide with eachother."

You missed my point. If the two libraries really use the same namespace, your code for importing both of them will turn into something like this:

  (import library1)
  (import library1)
  ...
Obviously this extrapolation of your code is a degenerate case, and I bet you wouldn't really suggest it. But the challenge itself is degenerate in exactly this way.

I often consider what would happen if two versions of a single library were both useful in a single application. If the library author didn't anticipate and avoid self-conflict, the two versions would host a train wreck of collisons on every single name, even the library name. I think that's the kind of challenge Casey McMann has in mind here.

-----

1 point by Pauan 4153 days ago | link

rocketnia: "How do you keep library1 and library2 from using the same version of library3?"

I'm assuming the challenge was something like this:

  # library1
  (import library3-version1)
  
  # library2
  (import library3-version2)
  
  # application
  (import library1)
  (import library2)
I fail to see the issue. In library1's lexical scope it will use library3-version1, and in library2's lexical scope, it will use library3-version2. I have no clue what issue you're thinking of.

---

rocketnia: "Regarding "they'd like the new library to be able to occupy both orignal spaces," your code is again incomplete. If the two originals are library1 and library2, an intermediate library library3 depends on library1, and the new one is library4, then how do you get library3 to start using library4 instead of library1? Keep in mind you can't modify any of the libraries."

The challenge was the following:

  # library1
  (def foo ...)
  
  # library2
  (def bar ...)
  
  # library3 = library1 + library2
  (def foo ...)
  (def bar ...)


  # application1
  (import library1)
  
  # application2
  (import library2)
  
  # application3
  (import library3)
In other words, you have two libraries. Then they create a third library which merges the two libraries together. Existing code that relies only on library1 will continue to work. Existing code that relies only on library2 will continue to work. New code that wishes to use either library1 or library2 can instead use library3, which combines both library1 and library2.

There's no need to change application1 and application2: they can just keep using library1 and library2. You might say that's inefficient and you'd rather them use library3, but that's up to them whether they want to switch or not. I'm also imagining the situation that library3 actually does cause some backwards compat issues, in which case it would be desirable to keep using library1/library2. The application chooses which libraries to import.

You seem to be suggesting some mechanism in which a module can actually modify what a different module imports and exports. I fail to see the benefit in that: that module's code is written under the assumption that its imports won't be changing. Could you provide an example where that would be useful?

---

rocketnia: "Regarding the licensing example with "they'd like it to be a drop-in replacement [...] An application I'm working on uses two libraries, one of which requires the replacement library [...] The other [...] uses the original basically by default": I think they're asking for a mechanism that lets the application choose whether the "other" library uses the original or the replacement. I don't see how your code makes this possible, and your "this could instead import replacement" comment suggests that you'd actually modify library2."

No I do not think that's what they're asking. I think what they're asking is: can the original library coexist at the same time as the replacement, even though they have name collisions. Notice they used the word "reconcile". This suggests reconciling the name collisions. There's no need to "reconcile" the other library to use the replacement: it can just keep using the original.

But if I'm wrong and they were in fact not talking about name collisions at all but about the orthogonal issue of replacing another module's imports... well, no, Nulan doesn't let you do that, because I don't see any useful benefit from doing so.

In any case, that's a completely different issue from avoiding name collisions, and seems to me like it'd be a part of some sort of build system or dependency resolution system, like RubyGems. That might be useful, but I don't see it as being a part of the namespace system per se.

---

rocketnia: "Regarding "conrol the names you import [...] such that [...] a new export to an existing library will not affect the meaning," it looks like you do have a good response to this. Personally, I wouldn't want there to be any imports that could possibly break that way, but I guess I'm more picky than they are. :)"

Yup, I try to be pretty laid back (at least about my code). If you're anal about it, you can use "include", but I think it's nice to have the option of just doing unqualified imports. It's up to the library/application author, in my opinion.

---

rocketnia: "You missed my point. If the two libraries really use the same namespace, your code for importing both of them will turn into something like this:"

Nope, I don't think namespace == module. You are importing the same module twice, but it's in the same namespace. And if you did this:

  (import library1)
  (import library2)
You're importing two different modules, but they're in the same namespace. In other words, "library2" can see the stuff exported by "library1", even though it didn't import it.

You might say this is undesirable. I agree it can be undesirable in certain situations, and I think the solution is to import each library in a separate namespace. But hyper-static scope can be implemented with a single namespace, so whether it's single-namespace or multi-namespace is an orthogonal issue.

---

rocketnia: "I often consider what would happen if two versions of a single library were both useful in a single application. If the library author didn't anticipate and avoid self-conflict, the two versions would host a train wreck of collisons on every single name, even the library name. I think that's the kind of challenge Casey McMann has in mind here."

In that case whichever one is defined last "wins", and you can use "include" and "exclude" to control which ones you want. I could also provide a way to import a library into a separate namespace to solve that case:

  (import-as library1 lib1)
  (import-as library2 lib2)

  (lib1 'bar)
  (lib2 'bar)
I think that's useful, but hyper-static can be implemented without it. My point is that even if a language doesn't provide include/exclude/import-as, it'd still be better off using hyper-static rather than nothing at all.

-----

1 point by rocketnia 4153 days ago | link

"In library1's lexical scope it will use library3-version1, and in library2's lexical scope, it will use library3-version2."

I'm assuming the example is supposed to look like this instead:

  ; library1 (coded with library3's version 1 in mind)
  (import library3)
  
  ; library2 (coded with library3's version 2 in mind)
  (import library3)
Each of these libraries would work with a certain library3, but library1 might not work with the newest version, or (in my dream world where everything is perfectly forward-compatible) maybe library1 is just considerably more performant with the old version.

---

"You seem to be suggesting some mechanism in which a module can actually modify what a different module imports and exports. I fail to see the benefit in that: that module's code is written under the assumption that its imports won't be changing. Could you provide an example where that would be useful?"

Here's Casey McCann's full description of the challenge again, just for clarity's sake:

Casey McCann: "At some point, two popular libraries with related functionality get tired of duplicating effort, so they collaborate to produce a third library providing the best of the shared features. To avoid unnecessarily breaking client code with the new versions, they'd like the new library to be able to occupy both original namespaces, or otherwise be easily substituted in. How would the namespace system handle this?"

I had it wrong before, when I made the sweeping statement that only the application's code should have to change in these challenges. This challenge is a case where the library developers are trying to change their code without causing disruption for anyone else.

Ideally, the only thing an application developer should have to do to upgrade is to replace the existing two libraries with the new one. This is even something that an end user might handle using a package manager.

---

"I think what they're asking is: can the original library coexist at the same time as the replacement, even though they have name collisions. Notice they used the word "reconcile". This suggests reconciling the name collisions. There's no need to "reconcile" the other library to use the replacement: it can just keep using the original."

I think the point is that the library has been upgraded, and the library developer would like the upgrade to propagate to all code that uses the library, even though a few library users might actually prefer otherwise. This is a conflict of preferences, and the language might give tools to certain participants to "reconcile" this conflict.

I prefer to view this as a last resort. The module system should give a complete(-feeling) range of freedom to every participant, and the ranges should not overlap. I would allow upstream developers to propagate their changes only as far as downstream developers have opted in. This turns into a design for package versioning.

---

"Nope, I don't think namespace == module."

I agree. The challenges are supposed to be realistic cases where name collisions would actually occur and cause havoc, so naturally they mostly involve programs made up of parts that have been written by multiple people independently. So we should expect to consider modules here, but yeah, the modules don't have to correspond one-to-one with namespaces.

However, I think Casey McCann is assuming a namespace system built on implicit name prefixes, which may or may not incorporate a way to take a name prefix and find a module that addresses it. While this may be a fine way to think of namespaces Java, Racket, and Haskell, the challenges which arise from this assumption may not apply to Nulan. No worries. :)

That said, if you change Nulan's design in order to accommodate some of these other challenges, you may come back to this one and find you've broken it. That is to say, this is still a meaningful test case even if it passes at the moment.

---

"My point is that even if a language doesn't provide include/exclude/import-as, it'd still be better off using hyper-static rather than nothing at all."

By "nothing at all," do you mean a late-bound global scope like Arc's? I think I agree with you.

-----

1 point by Pauan 4152 days ago | link

"I'm assuming the example is supposed to look like this instead"

Well, that either won't work (it would load the same "library3" in both cases), or it would work with a sufficiently smart lookup algorithm.

Basically, when importing a file, I think it should look for it in this order: the cwd, the folder (and subfolders) where the importee is defined, then lastly the folder where Nulan itself is defined.

In that case, assuming library1 and library2 are in different folders and have different versions of library3 in their path, it would work.

This implies bundling the libraries you use with your app, which you pretty much have to do anyways.

In that case, this has nothing to do with name resolution, it has to do with file resolution: finding which file to import when given a name.

You could argue that's a good case for a package manager like RubyGem, and that may be true, but I consider such a system to be in a layer above the namespace system.

---

"This challenge is a case where the library developers are trying to change their code without causing disruption for anyone else."

And Nulan's namespace system handles that just fine.

---

"I think the point is that the library has been upgraded, and the library developer would like the upgrade to propagate to all code that uses the library, even though a few library users might actually prefer otherwise. This is a conflict of preferences, and the language might give tools to certain participants to "reconcile" this conflict."

I still don't get the impression they were talking about some sort of package management or version dependency resolution system. It's certainly okay to discuss such things, but I think they're outside the scope of a namespace system. That thread (and post) seemed to me to be talking specifically about resolving naming conflicts.

I think the talk about "multiple libraries" and "multiple versions" was just to create a situation where naming conflicts occur. I don't think it was meant to imply that the goal is to be able to easily change library versions on the fly, like you were discussing.

If I wanted some sort of system like that in Nulan, it wouldn't be particularly hard to do... you'd create a new file, like say... "dependencies.nu" and you'd then put a bunch of imports into it:

  (import foo)
  (import bar)
  (import qux)
That's all it would be: a list of imports. Then all the dependencies are in one place, rather than scattered around in various files. I personally don't see much benefit to that, but you could do it if you wanted to.

---

"However, I think Casey McCann is assuming a namespace system built on implicit name prefixes, which may or may not incorporate a way to take a name prefix and find a module that addresses it."

That is quite possible.

---

"By "nothing at all," do you mean a late-bound global scope like Arc's?"

Yes. And JavaScript. Basically any language with dynamic scope at the global level by default.

-----

1 point by rocketnia 4152 days ago | link

Me: "This challenge is a case where the library developers are trying to change their code without causing disruption for anyone else."

You: "And Nulan's namespace system handles that just fine."

What I meant was that the library developer wants our applications to use their new library version without any involvement of the application developers. I seriously doubt that's what you're claiming to support.

---

"You could argue that's a good case for a package manager like RubyGem, and that may be true, but I consider such a system to be in a layer above the namespace system."

I think the only place we really disagree now is that you're content to neglect the "RubyGem" parts of these challenges for the purposes this discussion, while I consider them the hardest parts. I think if you do try to design a package manager, you may very well get most of a namespace system out of the deal, or even a whole programming language.

Unfortunately, package managers and filesystems aren't quite a full layer above the languages they're used with. Program units tend to link together using hardcoded filenames or package names, a practice that ties some of the filesystem or package manager complexity into the complexity of the language proper.

The module system I've been working on recently (ever since [1]) is meant to help resolve that, by making the non-language-specific complexity as simple and universal as possible. It's based on two notions: A program is a multiset of independently maintained extensions, and it exists in the context of an expansive historical record containing all knowledge ever discovered (or at least all knowledge we have on hand).

The extensions and units of background knowledge may ultimately be encoded as files and managed at a "layer above" using traditional package managers and filesystems. However, they may also be manipulated as first-class values in a host program, or even as visual elements in a UI.

[1] http://rocketnia.wordpress.com/2012/06/16/meaning-preserving...

-----

1 point by Pauan 4152 days ago | link

"In that case, this has nothing to do with name resolution, it has to do with file resolution: finding which file to import when given a name."

As an argument in favor of this viewpoint, I'll quote the original challenge:

"Now I'm writing a piece of code that uses two libraries, each of which use different, incompatible versions of some third library, causing namespace collisions between the two versions. How does the namespace system handle this?"

The challenge seems to imply that the two incompatible libraries were loaded successfully: it's NOT talking about file resolution. It's assuming the file resolution proceeded successfully, and thus the only issue remaining is resolving the name collisions.

You keep talking about things like dependency resolution systems, swapping in different libraries, etc. That's fine, but that does not seem to be what the challenges are talking about. I'd rather see some challenges specifically geared toward dependency resolution and file loading rather than trying to hijack namespace challenges, since I see these two issues as being orthogonal, with maybe a little overlap.

-----

1 point by rocketnia 4152 days ago | link

"The challenge seems to imply that the two incompatible libraries were loaded successfully: it's NOT talking about file resolution. It's assuming the file resolution proceeded successfully, and thus the only issue remaining is resolving the name collisions."

From what you quoted, technically that could be true.

---

"You keep talking about things like dependency resolution systems, swapping in different libraries, etc. That's fine, but that does not seem to be what the challenges are talking about."

Objection! :)

The challenge we're talking about puts the words "different, incompatible versions" in italics. If direct substitution were unimportant, the challenge would not even need to use the word "version"; it could just say there were two separate libraries with name collisions. And yet that's precisely the scenario of the previous challenge.

-----

1 point by Pauan 4151 days ago | link

"What I meant was that the library developer wants our applications to use their new library version without any involvement of the application developers. I seriously doubt that's what you're claiming to support."

I don't get the impression they're saying, "we release this new library and all the users of the existing libraries automatically switch to it". I think what they're saying is, "this library can be used by existing applications, but they still need to choose to use it". That's how I interpreted the "drop-in replacement" phrase.

Technically speaking Nulan can support that... you would change library1 and library2 to just import library3:

  # library1
  (import library3)
  
  # library2
  (import library3)
  
  # library3
  (def foo ...)
  (def bar ...)
Then the applications don't need to change at all. That's assuming of course that the developers of library3 can change library1 and library2. If they can't, then oh well, my solution is for the applications to manually update to use library3 if they so wish. In that semi-contrived example it doesn't matter because library3 is a superset of library1 and library2, so applications can just keep using library1/2 just fine.

---

"I think the only place we really disagree now is that you're content to neglect the "RubyGem" parts of these challenges for the purposes this discussion, while I consider them the hardest parts. I think if you do try to design a package manager, you may very well get most of a namespace system out of the deal, or even a whole programming language."

They are the hardest parts, I just don't think it has to do with namespace support, which that entire topic was devoted to. Nor am I particularly interested in designing a package manager.

So my suggestion still stands: if you feel this strongly about package managers, make up your own semi-contrived challenges that try to demonstrate what a package manager should be capable of doing. I'm sure it will include all kinds of stuff that was left out of the namespace challenges, because package managers are much more complex and involved than namespaces.

Then you can point to those challenges and I can say, "Nulan doesn't support that stuff" rather than trying to insist that Nulan is failing the namespace challenges because you're interpreting them far larger than what they actually said. I think it's useful to have "package manager challenges", but I think it's also useful to have "namespace challenges". I don't see why we need to throw out the namespace challenges just because you view it as "not as hard" as package managers.

---

"The challenge we're talking about puts the words "different, incompatible versions" in italics. If direct substitution were unimportant, the challenge would not even need to use the word "version"; it could just say there were two separate libraries with name collisions. And yet that's precisely the scenario of the previous challenge."

Like I said, I don't get the impression they're talking about package managers. I think they used the phrase "different incompatible versions" because that's a situation that can actually occur in real life. They're using it as an example, that's all.

It's obvious that you want to think in terms of package managers, and that's fine, but I really do not get the impression that's what they're talking about. Ultimately, it seems the only way to resolve this is to actually ask them what they meant.

-----

2 points by rocketnia 4151 days ago | link

"They're using it as an example , that's all."

Oh, I can understand that point of view. Just because there were multiple examples doesn't mean each one introduced new requirements.

---

"It's obvious that you want to think in terms of package managers, and that's fine, but I really do not get the impression that's what they're talking about."

I just wanted to help you elaborate on the claims you're making about Nulan's namespace system. You originally quoted Casey McCann's examples, but I perceived them differently than you did, and I had trouble understanding your perception from your terse example code.

Now this seems to be resolved. We may not agree in our interpretations, but now I understand your claims relative to both of them. Thanks for your patience with me. ^_^

---

"Ultimately, it seems the only way to resolve this is to actually ask them what they meant."

I don't think that's necessary, but it would be interesting to know.

-----

2 points by rocketnia 4154 days ago | link

And here are some replies to the "meta" part of that email exchange:

---

akkartik: "Thanks! I find myself wishing we could put Ross's long response in the 'namespace' section of the page rather than the 'meta' section."

I was conscious of that when I sent that reply. I'm not sure how to reply to a particular "subthread" when it's email. >_<

The only time I've ever seen email become a threaded tree is in mailing list archives, and I kinda assumed that had to do with automatic detection of quotations or something.

---

akkartik: "(I'm not actually expecting you to do this, just thinking aloud about how someone could maintain hygiene when managing a forum.)"

The difficulty here isn't managing one forum, but managing a conversation that spans multiple communication media. Unfortunately, I don't expect there to be a nice bridge that eliminates more complexity than it introduces. That kind of synergy tends to rely on foresight on the part of both systems.

-----

3 points by akkartik 4154 days ago | link

Actually, email supports threaded trees using the References: header. Every message adds the ids of all messages it's under, and it turns out that is enough information to recreate the tree structure. Text-mode email clients like pine and mutt use this.

Given my history with those programs, I'm anal about hitting reply on the precise message in a thread that I'm replying to :) That helps list archives get the tree structure right even though it's invisible over gmail. So in your place I'd have split my message into parts and attached them to the right messages in the thread :) I know, it's utterly ridiculous and irrational.

-----