← Back to context

Comment by amluto

12 days ago

It’s a closure oddity. The closure captures a reference to the variable, not the value of the variable at the time of its creation. Given that, the results are entirely predictable.

11 comments

amluto

Reply
abecedarius  12 days ago

If we're really going to argue, I'd argue for a 'for' oddity. The for statement could've been defined with a fresh binding per iteration instead of a mutation.

  • tln  12 days ago

    How does that work in practice? Python stores local variables in the stack frame with a compiler-calculated index. Do you have a growable stack frame? Add stack frames for every loop iteration?

    I don't think the Python compiler is to know if a loop variable is closed over and then adjust the code generation. So this change would likely introduce a lot of complexity, or be a pessimisation in the normal case.

    I think the simple, although less helpful, semantics are clearly better here. FWIW I don't think this comes up in practice very much. Maybe because callbacks in Python APIs aren't common (and its not due to this).

    • abecedarius  11 days ago

      See my other reply about how [x for x in y] now works. It'd be a variation on that.

      I agree the current semantics are better by the measure of performance of the simplest possible implementation. Though that's not Python's main design goal.

    • gpderetta  12 days ago

      it comes up often enough for me to be annoying. The defaulted argument works, but it is an ugly hack. It is not obvious to me that the surprising semantics are clearly better, but python has a long history to preferring implementation simplicity to user ergonomics.

  • b3orn  12 days ago

    It's unrelated to the for loop, the for only changes the variable, there are other ways you can do that and you'll run into the same "oddity"

        def test():
        n = 0
        def a():
            return n
        n = 1
        def b():
            return n
        return a, b

    a, b = test()
    a() == 1
    b() == 1

    • abecedarius  11 days ago

      > for changes the variable

      That's a design decision, not a law of nature. JS 'for let' lacks this problem; probably the committee would have named it just 'for' if not for legacy.

    • hcmarker  12 days ago

      I get why it behaves like this, but I find it odd that Python claims to have "lexical scoping". This is not lexical (as you read it). This is based on somewhat hidden reference lookups.

  • Izkata  12 days ago

    That would make it work differently than the entire rest of the language. Python is function-scoped, not block-scoped.

    • abecedarius  11 days ago

      That ship already sailed when Python 3 made comprehension expressions (like list comprehensions) introduce a nested scope. They're even a kind of 'for' too. (Previously, [x for x in y] would clobber any x in the same function. Fixing this meant implicit nested scopes.)

      It's true that this would've been a further change (every loop entering the implicit nest, rather than entering one nest which then mutates per loop). But this design was a decision, not a necessity. JS's "for let" shows this by example.

  • nbadg  12 days ago

    Not really -- in practice that would imply that ``for`` needs to freeze the value of ``locals`` during every iteration, which doesn't make sense to me.

    This is definitely a python gotcha. In fact, it's literally one of the canonical ones [1]. But it's purely the interaction between the way that names/variables work in python, and late-binding closures.

    More explicitly: it's best to think of "variables" in python as "names", because in code, you refer to names, but the underlying variable might change. This is also why python is best described neither as call-by-reference nor call-by-value, but rather "call-by-assignment" or "call-by-object": you're passing a reference to a variable bound to a particular name. Unless you're writing C extensions, you never actually have a reference to the variable itself -- instead, to the name.

    Meanwhile, closures in python are always late-binding: they are themselves "live", mutable objects, in the sense that they're just a reference to the namespace they're closing, and *not* a copy. So if you mutate that namespace between establishing the closure and making use of it, you'll see the mutated version.

    [1] https://docs.python-guide.org/writing/gotchas/#late-binding-...

    • gpderetta  12 days ago

      Yes, the best way to understand python closures is that scopes are first class objects in python (accessible via locals()) and closures close over the containing scope itself, not the single variables.

      Also worth keeping in mind that, AFAIK, there is one scope per function, which differs from other languages that can have nested scopes inside a function. I think the list comprehension is the only exception in python3, which was changed so that names no longer leak out of it, which was frankly insane in py2.