The value is silently truncated, instead of raising an error that the decimal cannot be safely rescaled. This example would raise this error in pandas 2.1.4:
>>> ser = pd.Series(["1.2345"], dtype=pd.ArrowDtype(pa.string()))
>>> ser.astype(pd.ArrowDtype(pa.decimal128(1, 0)))
pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss
>>> pd.__version__
'2.1.4'
For decimal most usages values accuracy, and silently truncating the value does not seem desirable, also if the original series was a decimal, then this error would be raised:
>>> ser = pd.Series(["1.2345"], dtype=pd.ArrowDtype(pa.decimal128(5, 4)))
>>> ser.astype(pd.ArrowDtype(pa.decimal128(1,0)))
pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss
So it seems unusual when casting a decimal from a string, the value is silently truncated during rescale, but when casting it from another decimal, then there's no truncation during the rescale operation and an error is raised.
Expected Behavior
>>> ser = pd.Series(["1.2345"], dtype=pd.ArrowDtype(pa.string()))
>>> ser.astype(pd.ArrowDtype(pa.decimal128(1, 0)))
pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss
This was the behavior in pandas 2.1.4.
Installed Versions
INSTALLED VERSIONS
commit : f538741
python : 3.11.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 151 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252
pandas : 2.2.0
numpy : 1.26.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.17.2
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.2
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 14.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None
6399982 is the first bad commit
Author: Matthew Roeschke
Date: Thu Dec 14 13:06:54 2023 -1000
BUG: time strings cast to ArrowDtype with pa.time64 type (#56490)
cc @mroeschke
For decimal most usages values accuracy, and silently truncating the value does not seem desirable
but when casting it from another decimal, then there's no truncation during the rescale operation and an error is raised.
Agreed the inconsistency is not desirable, but with some dtypes we do perform lossy operations when the user requests it, e.g.
pd.Series([1.2345]).astype(int)
or from float64 to float32. It's not clear to me that we shouldn't allow this case to be lossy as well. I don't think it's surprising that .astype(pd.ArrowDtype(pa.decimal128(1, 0))) could be lossy, but as you point out perhaps special consideration should be given since Decimal types are specifically geared to avoid the issues of floating point data.
FWIW: The example of casting float to int fails with pyarrow types because it is lossy:
>>> a = pd.Series([1.2345], dtype="float64[pyarrow]")
>>> a.astype("int32[pyarrow]")
pyarrow.lib.ArrowInvalid: Float value 1.2345 was truncated converting to int32
Looks like numpy types allow the lossy conversion but pyarrow types not do not.
