DOMException: Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'https://unpkg.com/@loaders.gl/[email protected]/dist/parquet-worker.js' failed to load.
worker-thread.js:97 Uncaught (in promise) Error: Failed to load worker parquet (#1 of 3) from https://unpkg.com/@loaders.gl/[email protected]/dist/parquet-worker.js. Uncaught NetworkError: Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'https://unpkg.com/@loaders.gl/[email protected]/dist/parquet-worker.js' failed to load. in :5:3
at WorkerThread._getErrorFromErrorEvent (worker-thread.js:97:14)
at Worker.worker.onerror (worker-thread.js:131:29)
Running the following lines generates the same DOMException error:
const workerUrl = "https://unpkg.com/@loaders.gl/[email protected]/dist/parquet-worker.js"
const workerSource = `\
try {
importScripts('${workerUrl}');
} catch (error) {
console.error(error);
throw error;
const blob = new Blob([workerSource], { type: "application/javascript" });
const loadableUrl = URL.createObjectURL(blob);
const worker = new Worker(loadableUrl, { name: "workerTest" });
If I replace the parquet worker url with an arrow worker url (like "https://unpkg.com/@loaders.gl/[email protected]/dist/arrow-worker.js"
) I don't get that error.
Let me know if I can add any more details, and thanks for your help!
I haven't personally worked the JS Parquet implementation at all, so I don't have any good information on this. My (possibly uniformed) impression is that the JS implementation can be hard to maintain because the spec is large. Our code is a fork of https://github.com/kbajalc/parquets which is itself a fork of https://github.com/ironSource/parquetjs. In #2144 I was trying to test out the current code and found it to fail on some basic files. In #2103 I have a WIP implementation that wraps https://github.com/kylebarron/parquet-wasm for a WASM parquet implementation, and found that to potentially be a lot faster.
I'm hoping to get #2103 wrapped up sometime in the near future, but if you have interest, a PR to finish that branch would be welcome.
Thanks for the response! Unfortunately it looks like #2103 is way outside my wheelhouse so I don't think I'll be much help to you there, but I'll keep an eye on it and #2144 in case those resolve something that helps out my issue as well. I found that the parquet loader (outside of the worker, at least) was pretty slow, so that's exciting that your implementation might be so much faster.
The parquet code, both the JS and WASM version still have serious JS bundling issues (inherited from the solutions we are building on). The JS code mainly because of its thrift
dependency. We are slowly working through these, some progress in 4.0 and hopefully we'll get further in 4.1.
Closing for now as this is not a quick fix.