Why do I get the error TypeError: cannot pickle object? - Archive

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

朝气蓬勃的猴子 · 它继承原有阿拉丁卡片的强样式展现，给搜索用户 ...· 5 月前 ·

腹黑的西装 · 一個人的婺源輕旅行· 6 月前 ·

儒雅的火腿肠 · 全链路拓扑_应用性能管理APM_用户指南（1 ...· 1 年前 ·

逃课的硬币 · 华语流行乐坛史略（一、起始和发展） - 知乎· 1 年前 ·

不羁的桔子 · 宜昌关陵庙是宜昌的著名景点吗？宜昌关陵庙好玩 ...· 1 年前 ·

1. Dask

The first is when using a DaskExecutor and using a task input or output that is not serializable by cloudpickle . Dask uses cloudpickle as the mechanism to send data from the client to the workers . This specific error is often raised with mapped tasks that use client-type objects, such as connections to databases or HTTP clients , as inputs.

Solution: Such database or HTTP connections need to be instantiated (and closed) inside your Prefect tasks.

2. Results

The second way this can happen is through Results . By default, task outputs are saved as LocalResults , and the default Serializer is the PickleSerializer , which uses cloudpickle. If you have a task that returns a client or connection , you can avoid serialization of this task by turning off checkpointing for that task with @task(checkpoint=False) .

View in #prefect-community on Slack

@Guillaume_Latour : Hi everyone, I stumbled upon an error as the prefect engine serializer tried to pickle a task result: TypeError: cannot pickle 'lxml.etree.XMLSchema' object
Of course the result of one of my task is of this type and I would prefer not to change it.

I tried to launch that process again with config.flows.checkpointing = "false" in the ~/.prefect/config.toml file but I got the same error…

This is an error that I cannot reproduce locally with prefect run <flow> . Do you guys have any leads on how I can solve this issue or at least reproduce the error locally?

@Kevin_Kho : config.flows.checkpointing = "false" gets overridden in Cloud backed runs. You need to turn it off in the task level @task(checkpoint=False) .

@Guillaume_Latour : oh.
But I am hosting the prefect server, would that change anything ?

I’ll try to put this option in the decorator nonetheless
ty for your quick response

@Anna_Geller : it works the same way for Cloud and Server

@Guillaume_Latour : yay it worked \o/