This issue tracker has been migrated to GitHub , and is currently read-only .
For more information, see the GitHub FAQs in the Python's Developer Guide.

Created on 2017-12-09 12:10 by Camion , last changed 2022-04-11 14:58 by admin . This issue is now closed .

Files File name Uploaded Description issue-32259-iterable-unpackable.patch Camion , 2017-12-10 23:32 sample patch for the changes locations. The actual message wording might have to be adapted according with discussions.

I'm new with Python and I've been blocked for day on a "TypeError: 'Fraction' object is not iterable" error message, while the problem turned out to be completely different.
I don't even know to what programming case this message would have been the right interpretation, but in this situation, it was completely wrong and I believe it can cause much loss of time for inexperienced python programmer (at least, I believe it should document alternative causes)
Here is a sample code to show how misleading it can be (It's a program to compute pi with fractions)
from fractions import Fraction
def order(x):
    r, old_r, n, old_n = 2, 1, 1, 0
    while (x>=r):
        r, old_r, n, old_n = r*r, r, 2*n, n
    return order(x >> old_n) + old_n if old_n > 0 else 0
def terms(m, n, i):
    return Fraction(4 * m, n**(2*i+1) * (2*i+1))
def terms_generator(exp_prec):
    ws = [ [terms(parm[1], parm[2], 0), 0] + list(parm)
          for parm in ((1, 44, 57),
                       (1, 7, 239),
                       (-1, 12, 682),
                       (1, 24, 12943))]
    digits = 0
    while digits<exp_prec:
        curws = max(ws, key=lambda col: col[0])
        digits = int(0.30103 *
                     (order(curws[0].denominator))
                      - order(curws[0].numerator))
        yield curws[2] * curws[0] #, digits
        curws[2] = -curws[2]                
        curws[1] += 1
        curws[0] = terms(curws[3], curws[4], curws[1])
expected_precision = 100
pi = 0
for term, dgts in terms_generator(expected_precision):
    pi += term
print("{} digits".format(dgts))
print("pi = 3.{}".format(int((pi-3)*10**expected_precision)))
a, b = 3
You'll get the error "TypeError: 'int' object is not iterable". That's because Python sees 2 items to the left of the assignment, so it needs to extract 2 items from the right side. To get 2 values from the right side, it iterates over the right side. In this case, 3 cannot be iterated over, thus the error message.
Now consider:
a, b = 3, 4
Again, to get 2 items from the right side, Python iterates over it. In this case the thing on the right side is a 2 element tuple. It's a little confusing that it's a tuple because it doesn't have parentheses, but the comma between 3 and 4 makes it a tuple. In this case, Python can iterate over the tuple, and the tuple has exactly 2 elements, so the assignment to a and b succeeds with a = 3 and b = 4.
I hope that clears it up.

Ok, but the other explanation is valid as well. That's why I suggest to modify the error message like this : 
TypeError: '[TYPE]' object is not iterable
- OR -
ValueError: not enough values to unpack (expected [N], got 1)

I'm not talking about changing the type of the exception, Serhiy. but only to make the text message more explicit by adding a lead to a secondary possible cause. I do not understand how this addition would be misleading - more than the case I presented ? (Did you check my example ?)
I agree that this message might possibly make my mistake obvious to an senior _python_ programmer, but doesn't the constraint of being _experienced in python_, not go in contradiction with the mantra "Explicit is better than implicit" ?

I agree with Camion that the error message is misleading, and not just for beginners. It threw me for a loop too, when I first read it.
Serhiy is right, the exception type cannot and should not be changed, but we can change the error message. I'm re-opening the ticket as an enhancement to improve the message.
Here's my suggestion:
TypeError: cannot unpack object ('Fraction' is not iterable)
There is no stability guarantees on error messages, so we can include it in 3.6 (and possibly older if desired).
By the way Camion, as interesting as your program to calculate the digits of pi is, in general it is better to cut the code down to the simplest version that demonstrates the problem. Something like this would demonstrate it equally as well:
def gen():
    yield 1
for x, y in gen():
    
I wrote it like that on purpose, Steven : My goal was not to show the message itself which (I believe) was sufficiently described in the explanation, but to show how hard it might be to understand the mistake in regard with the error message, even here, with a real life but not outrageously complicated code. :-)
    
By the way, I guess if the problem arises like that, it's because it might be hard to distinguish both situations at the interpreter level, but if it was possible, it would be the best solution to have a different error message (even if I understand we should be very careful with the idea of changing the exception type, even in this case).
I mean : Eric shown an equivalent error, but both are only equivalent at the present discrimination level of the interpreter & language grammar.  I understand that distinguishing both those semantically different situation might be opening the Pandora box and it might be a bad idea... or not.
The point is that, if ever both those situations were in some way distinguishable, then... well... we certainly have no idea of how much code confronted with this specific situation AND treated it with exception handling, BUT, we might make the assumption that a frequent confrontation with this case would then already have been raised before, and consequently, it might be relevant, at least to keep it in mind for a future evolution (4.0) of the language.
    
My point is that the current error message is correct and is not misleading. And this is a standard error message raised in many other cases. There is no a bug.
*Maybe* it can be improved in some specific cases. Do you want to provide a patch Steven? Without concrete patch the following discussion doesn't make sense. We even don't know if such patch is possible.
    
Serhiy, I think I got a better understanding of what is happening. It is well described by the following example :
>>> a, b = 1,
Traceback (most recent call last):
  File "<pyshell#40>", line 1, in <module>
    a, b = 1,
ValueError: need more than 1 value to unpack
>>> a, b = 1
Traceback (most recent call last):
  File "<pyshell#41>", line 1, in <module>
    a, b = 1
TypeError: 'int' object is not iterable
Again, the message might be correct, but in this case, the combination between the message and the ambiguous syntax, makes it lack from explicitness.
Understanding that, I suggest to simply add "(expected 'tuple')" at the end of the message.
ex : TypeError: 'int' object is not iterable (expected 'tuple')
    
On Sun, Dec 10, 2017 at 09:15:16AM +0000, Serhiy Storchaka wrote:
> My point is that the current error message is correct and is not misleading. 
With respect Serhiy, in this bug report you have TWO PEOPLE who have 
said that it is misleading in this context -- the person who raised the 
bug report, and me. It might not be misleading to you, but it mislead 
us. I've been using Python for over a decade and a half, and when I read 
the exception it confused me too: why is the for loop trying to iterate 
over a Fraction? Did the generator accidentally call return instead of 
yield?
I was especially confused because I was expecting a ValueError too 
many/too few values to unpack.
Of course, with a bit more thought I realised that the error isn't 
generated by the for loop, but the unpacking. 15 years of reading 
exceptions does count for something :-)
If you prefer "lacking sufficient information" over misleading, I won't 
disagree. Putting aside pointless quibbles over the exact wording of 
*why* the current error message is suboptimal, I hope that we can all 
agree that if the error occurs during an unpacking operation, the error 
message should preferably say that the object cannot be unpacked. Is 
there any advantage to not giving that information if we can?
> And this is a standard error message raised in many other cases. There 
> is no a bug.
I agreed that it isn't a bug, and changed this to an enhancement. Better 
error messages should be welcomed, not rejected because they aren't 
fixing a bug.
> *Maybe* it can be improved in some specific cases. Do you want to 
> provide a patch Steven? Without concrete patch the following 
> discussion doesn't make sense. We even don't know if such patch is 
> possible.
Someone who knows C and the interpreter internals better than me will 
have to provide a patch. I wouldn't even know where to start looking. 
But my guess would be the UNPACK_SEQUENCE op-code.
    
On Sun, Dec 10, 2017 at 10:00:27AM +0000, Camion wrote:
> Understanding that, I suggest to simply add "(expected 'tuple')" at the end of the message.
> ex : TypeError: 'int' object is not iterable (expected 'tuple')
That is incorrect: a tuple is not expected. Any iterable (sequence, 
iterator, tuple, list, set, frozenset, dict, etc) will do. One thing we 
should not do is give a misleading, overly-specific message that implies 
only a tuple will work.
We could say 
TypeError: 'int' object is not iterable (expected iterable)
but that's redundants, since the comment in the parentheses is implied 
by the fact that *not* being iterable is an error.
I think the right thing to do here (if possible!) is to distinguish 
between unpacking and other iteration. E.g.
for x in 20: ... 
=> TypeError: 'int' object is not iterable
a, b = 20
=> TypeError: cannot unpack 'int' object (not iterable)
or similar. I'm less concerned about the exact wording than the fact 
that we give a hint that it was the unpacking operation that failed, 
rather than iteration in some other context.
    
@Serhiy : I asked you to explain by what mean you supported the affirmation that our feeling (that the current message can be misleading in the specific situation) is wrong, but you didn't give us any element to understand your position. Steven and I did it: I gave you a sample code showing how and why a situation could be misinterpreted in regard of the current message, and Steven explained that as an experience python programmer, he experienced the same misunderstanding (I'm also an experienced programmer: started in the eighties, I'm new in Python only).
May be, if you gave us some sound argument element, we could also better understand the reason why you affirm that our feeling is wrong. Because in the current situation, you only declared your own feeling, but didn't give it any support, and all we can conclude is that you are a very experienced python programmer who can understand what was beyond the text of the message, but it seems in contradiction with the principle of making things clear.
--------------------
@Raymond : I do not understand how you make the link between both your affirmations. The initial problem is related with the fact that in some situation, the error message lead to think that there is a problem with the loop/generator/iterator, where the problem is in reality with a simple type which cannot be unpacked or iterated. 
Let's assume that the interpreter could never make the difference. In what way, does it support the fact that the message shouldn't in any way, hint the programmer on the fact that the problem is not with the generator/iterator/loop, but with the returned type ?
--------------------
@Steven (& everybody), 
How about : "[TYPE] object is not iterable/unpackable" ?
--------------------
About patching the sources in c, the message "object is not iterable" appears in four places in the version 3.6.3 : in ./Objects/object.c:1023, twice in ./Objects/typeobject.c (lines 6329 & 6344), and in ./Objects/abstract.c:3135. I have patched them to make them recognizable from one another, and it appear that all observed messages have been generated from abstract.c:3135 (in 3.6.3) and not from the other lines.
This poses problem, because not even the regression test show a case where those other 3 parts of the code are used, and I haven't figured out (yet?) what condition makes them occur.
    
> How about "[TYPE] object is not iterable/unpackable"
I just grepped the docs, and the term 'unpackable' does not appear anywhere in them.  I don't think this would be an improvement.
As for the earlier suggestion of adding "expected tuple" does not work, because it is not a tuple being expected, it is an iterable, and saying "expected iterable" would be redundant with saying "is not iterable".
What about "TypeError: expected iterable but found <type>"?  That's parallel to some other type error messages we already generate.  Would that be any clearer?  (I'm not sure it would be, it says the same thing.)
    
@Camion, please adopt a less insistent tone.  We've devoting time and thought to your concerns but are under no obligation to make a change just because you are demanding it.  
Part of learning any programming language is learning to reason from error message to root cause.  Good error messages help, but they don't eliminate the underlying not always comfortable path of learning the world-view of the language and its mechanisms.  That is partly why the devguide includes this passage, "Be careful accepting requests for documentation changes from the rare but vocal category of reader who is looking for vindication for one of their programming errors".
A instructor for several thousand engineers, I've not encountered a single case where the current message has been problematic, nor have we had tracker issue arise over Python's 27 year history.  And so far all the proposed wording changes make the message less accurate in other contexts where the message might arise. Based on that and on the comments by the other respondents, it would reasonable to close this issue.
    
Camion, the problem is not in the error message. The problem is in the complex expression that produces an error. If you have a complex expression you always have a change to misidentify the source of error. In your case the expression contains two implicit iter() invocations.
I have looked at the code. It is possible to make the error message in this case more specific. There are precedences of this when pass wrong type as var-positional or var-keyword arguments:
>>> print(*1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: print() argument after * must be an iterable, not int
>>> print(**1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: print() argument after ** must be a mapping, not int
In the following cases the error message is general. It is possible to make it more specific.
>>> a, b = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> {**1}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not a mapping
But PyObject_GetIter() is called in 88 places in CPython core and extensions. In all these cases we can make error messages more specific. But I have large doubts that it is worth to do. And I'm not sure even about the above examples.
If somebody provide a patch we could discuss it (I'm not sure it will be accepted at end).
    
@Raymond: I know that you have no obligation to make changes just because I'm demanding them and that's why I'm just trying to convince people of the validity of my observations, here. 
I apologize if my tone may seem "insistent", but please take in account than English is not my mother tongue.
I agree with the fact that part of learning a language is learning to reason from error message to root cause but I cannot ignore that Python's Zen asks for explicitness, and Steven agreed with that this situation was tricky to interpret.
It would be sly to suggest I'm be looking for vindication for one of my own programming errors. I'm experienced enough in trouble-shooting to have solved it by myself. It's just that I do not believe normal that this situation took me so much time and energy to solve from the error message **in regard with other similar situations**.
--------------------
@David : What other contexts are you talking about ? Instead of looking in the documentation which also talks about e.g.:unpacking archive files, You should probably grep the source code for strings containing the word unpack and it will show you that the word unpack is used in contexts fully related with the error in my example.
(SEARCH='".*unpack.*"'; find -name '*.c' -print0 | xargs -0 grep -li "$SEARCH" | while read F; do echo "$F" | sed 'p;s/./-/g'; grep -niC 5 "$SEARCH" "$F"; done) | less
--------------------
I don't know about what would be the right error message, but since I experimented on in and Serhiy asked for it, I made a sample patch on version 3.6.3 which contains all the change which I believe, would be required.
    
I think generically changing 'iterable' to 'iterable/unpackable' is wrong and would engender more confusion than at present.  Most uses of iteration have nothing to do with multiple assignment target unpacking.  Some minimal examples resulting in "TypeError: 'int' object is not iterable", where this change would be wrong, include
iter(1)
sorted(1)
for _ in 1: pass
What might be reasonable is to wrap iter(source) in multiple assignment target code with (the C equivalent, if it exists, of) try-except and augment the message before re-raising.  The addition could be something explicit like "and cannot be the source for multiple target assignment".
Camion, Steven gave you excellent advice about playing around with simplified code.  If you had split the multiple-operation error raising line,
for term, dgts in terms_generator(expected_precision):
into two simpler lines that each do less,
for value in terms_generator(expected_precision):
    term, dgts = value
you would have likely have spent much less time looking in the wrong place.  Isolating possible exception sources is a very useful debugging tool.
    
Jerry, I've been troubleshooting thing for 30 years and I'm quite experienced at it, and in the end I was perfectly able to manage this problem and solve this problem by myself. 
My point is not about my own difficulty to solve this problem, but about the fact that an information was lacking clarity in a language which claims that things must be told. You mention the fact that Steven suggested playing playing around with simplified code, but it's not what Steven suggested. Steven suggested to make the example simpler, and I replied him that this example was not chosen to describe the programming situation but illustrate the troubleshooting situation and provide a good way to show how a real situation might be problematic to analyse with this message - and Steven confirmed that he was also put out on a wrong track by this message. 
For sure, when you are an experienced programmer, you will develop a different way to understand this. My point is that it seems to go in contradiction with "Python's Zen".
You suggest that I should have split the packing and the unpacking on two separate lines. Ok, but this is an easy suggestion to make _when you already know the cause of the problem_. Each language has it's programming culture and; Ok, I'm new here but I haven't seen any recommendation going in that direction. On the contrary all the sample code and course I have seen on that topic encourage young programmer not to use this practice - and I believe it would be simpler to find a way to adapt the error message to make things clear that such an error can also arise for unpacking reasons (whatever the way you word it), than to change the programming/training culture around python.
I have been made multiple propositions to re-word this message, and Steven also made some. My last proposition was motivated by the idea of keeping the message short. Some other were longer. One might also write : "not iterable (this problem may sometimes arise from unpacking)" or something else.
The point is that in many situation, the problem comes from a mismatching between the number of values packed and the number of destination positions for unpacking; and in this case, you have a ValueError message stating clearly what happens. 
Only in this case, when the fact that you had only one value and didn't see it, makes that you no longer have an iterable type - do you get a TypeError. And Ok, I agree with Serhiy that the type of exception is correct, but I believe that the whole situation is misleading and should be corrected.
(I apologize for my long sentences : My mother tongue is french and it's not always easy for me to to make them short like you do in English XD )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot unpack int object
I don't know whether it is worth to do this change.

> I would add 'non-iterable', to get "cannot unpack non-iterable int object", to tell people what is needed instead.
Doesn't this imply that there are iterable int objects which can be unpacked?

In English, 'adjective noun' does not necessarily imply the existence of 'not-adjective' nouns, and the adjective may serve as a reminder or reason.  For instance, "This job is too dangerous for mortal humans!"
In the current context, failure of 'iter(ob)', by itself, only tells us that the particular object ob is not iterable. "'X' object is not iterable", rather that "'X' objects are not iterable", is correct.  (Ditto for other messages.)
That said, I think "cannot unpack int object, because not iterable" is better.

Is it worth to emit more specific (but possible uniform) error messages in other unpacking cases (see msg307999)?
FYI if a class implements __iter__ which returns non-iterable, the following error is raised:
>>> class C:
...     def __iter__(self):
...         return 1
>>> a, b = C()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: iter() returned non-iterator of type 'int'

I retested and iter(ob) (1) apparently raises 'not iterable' only when it can find neither __iter__ nor __getitem__.  It (2) raises 'non-iterator', as above, when it finds  __iter__, calls its, and get a non-iterator.  (3) Exceptions in __iter__ are passed through.  (If it finds and wraps __getitem__, errors only appear during iteration.)
The current patch for assignment unpacking catches case (1) with "v->ob_type->tp_iter == NULL && !PySequence_Check(v))".  Removing this would catch case (2) and incidentally case (3) TypeErrors.  Not catching user TypeErrors would require looking at the message.  I intentionally am not proposing to remove the TypeError check.
Presuming the existence of a PyErr_' function to get exception messages, we could simply prefix the existing message with a short 'Cannot unpack. '  The same could be done for the other cases in msg307999, but I don't know if they are subject to the same need.

thank You Serhiy for your C implementation. 
About this question of the dilemma around the meaning of "non iterable int" why not simply put the "non iterable" between parenthesis to avoid making it too verbose ?
"cannot unpack (non-iterable) int object"
or "cannot unpack int object (not iterable)
However, I wonder why nearly all the changes I've seen, seem to be about int ? This situation might arise with any non-iterable type (I got it first with Fraction).

New changeset 13a6c098c215921e35004f9d3a9b70f601e56500 by Serhiy Storchaka in branch 'master': bpo-32259 : Make a TypeError message when unpack non-iterable more specific. ( #4903 ) https://github.com/python/cpython/commit/13a6c098c215921e35004f9d3a9b70f601e56500 2022-04-11 14:58:55adminsetgithub: 76440 2017-12-26 10:32:10serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved 2017-12-26 10:30:43serhiy.storchakasetmessages: + msg309057 2017-12-19 10:29:36Camionsetmessages: + msg308625 2017-12-19 09:54:17terry.reedysetmessages: + msg308624 2017-12-19 08:17:04Camionsetmessages: + msg308620 2017-12-19 06:12:05terry.reedysetmessages: + msg308615 2017-12-19 04:31:36Camionsetmessages: + msg308612 2017-12-19 00:17:04terry.reedysetmessages: + msg308596 2017-12-18 21:40:26serhiy.storchakasetmessages: + msg308585 2017-12-18 20:50:41terry.reedysetmessages: + msg308584 2017-12-18 13:34:22serhiy.storchakasetmessages: + msg308556 2017-12-17 20:04:01terry.reedysetmessages: + msg308496 2017-12-17 15:29:14eric.smithsetmessages: + msg308490 2017-12-16 12:33:14serhiy.storchakasetmessages: + msg308477 2017-12-16 12:31:15serhiy.storchakasetstage: needs patch -> patch review
pull_requests: + pull_request4798 2017-12-16 11:03:53Camionsetmessages: + msg308472 2017-12-16 03:37:02terry.reedysetnosy: + terry.reedy
messages: + msg308443
2017-12-10 23:32:05Camionsetfiles: + issue-32259-iterable-unpackable.patch
keywords: + patch
messages: + msg308004
2017-12-10 23:20:42Camionsetmessages: + msg308003 2017-12-10 23:14:39Camionsetmessages: + msg308002 2017-12-10 22:54:01serhiy.storchakasetmessages: + msg307999 2017-12-10 21:47:16rhettingersetmessages: + msg307994 2017-12-10 21:19:12r.david.murraysetmessages: + msg307992 2017-12-10 21:15:55r.david.murraysetnosy: + r.david.murray
messages: + msg307991
2017-12-10 20:37:19Camionsetmessages: + msg307987 2017-12-10 19:36:54serhiy.storchakasetmessages: + msg307976 2017-12-10 18:52:01rhettingersetnosy: + rhettinger
messages: + msg307969
2017-12-10 10:31:41steven.dapranosetmessages: + msg307955 2017-12-10 10:22:34steven.dapranosetmessages: + msg307953 2017-12-10 10:00:27Camionsetmessages: + msg307951 2017-12-10 09:15:16serhiy.storchakasetmessages: + msg307946
versions: - Python 3.6 2017-12-10 06:25:59Camionsetmessages: + msg307944 2017-12-10 05:41:42Camionsetmessages: + msg307942 2017-12-10 01:31:49steven.dapranosetstatus: closed -> open

type: behavior -> enhancement
versions: + Python 3.7
nosy: + steven.daprano

messages: + msg307937
resolution: not a bug -> (no value)
stage: resolved -> needs patch 2017-12-09 17:33:38Camionsetmessages: + msg307904 2017-12-09 15:02:58serhiy.storchakasetstatus: open -> closed


messages: + msg307900
nosy: + serhiy.storchaka 2017-12-09 14:34:54Camionsetstatus: closed -> open

messages: + msg307899 2017-12-09 12:40:53eric.smithsetstatus: open -> closed

nosy: + eric.smith
messages: + msg307896

resolution: not a bug
stage: resolved 2017-12-09 12:10:32Camioncreate