$ python3
Python 3.6.8 (default, Oct 25 2023, 15:15:22)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.py as plt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/<...>/lib/python3.6/site-packages/matplotlib/__init__.py", line 917, in <module>
fail_on_error=True)
File "/<...>/lib/python3.6/site-packages/matplotlib/__init__.py", line 812, in _rc_params_in_file
with _open_file_or_url(fname) as fd:
File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/<...>/lib/python3.6/site-packages/matplotlib/__init__.py", line 786, in _open_file_or_url
fname = os.path.expanduser(fname)
File "/usr/lib64/python3.6/posixpath.py", line 235, in expanduser
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not PosixPath
This is on Rocky Linux 8.8.
So the history here is that I have a user who created a script that uses matplotlib. This import works fine in his original ‘/home/user/.local/’ environment. Note that the /home is not local but an NFS mount.
The purpose of his script is for other users to make use of it. So we took his entire ‘./local’ python directory and copied it out to a team NFS share and adjusted permissions so that everyone can use it.
We then configured a profile.d script on specific hosts that sets users who login to point their $PYTHONPATH, $PATH and $LD_LIBRARY_PATH to site-packages, bin and lib respectively to the new share. It all seems to works fine until matplotlib is exported. At which point it fails with the titled error.
I’ve been scouring the internet for a week (working on this as I have time) and I’ve uninstalled and reinstalled matplotlib a few times. Checked and tested different permissions but no go. Python appears to function fine. What I do not know is what it is about the copying of the environment that causes matplotlib to break. It even breaks for the original creator if we point him to the new share.
Any insight would be appreciated.
Edit: It shows installed on the same user it will fail on.
$ python3 -m pip show matplotlib
Name: matplotlib
Version: 3.3.4
Summary: Python plotting package
Home-page: https://matplotlib.org
Author: John D. Hunter, Michael Droettboom
Author-email: [email protected]
License: PSF
Location: /<…>/lib/python3.6/site-packages
Requires: cycler, kiwisolver, numpy, pillow, pyparsing, python-dateutil
Required-by: descartes, mizani, plotnine, scikit-image
I’m pretty new to Python and I’ve been banging my head against this for
a while. It seems like it would be a simple fix but I am missing what
it is.
So here is the error and I’ll explain a curve ball after. Note I’ve truncated the paths in the error.
File “/<…>/lib/python3.6/site-packages/matplotlib/init.py”,
line 786, in _open_file_or_url*
fname = os.path.expanduser(fname)*
File “/usr/lib64/python3.6/posixpath.py”, line 235, in expanduser*
path = os.fspath(path)*
TypeError: expected str, bytes or os.PathLike object, not PosixPath
This is odd, and I’m guessing (wildly, and only because I’ve had this
kind of thing in the past rarely, and not with matplotlib) that somehow
os.PathLike and PosixPath come from different implementations. Here I
mean that when os.fspath() is obtaining eg PosixPath it is getting a
different class than the PosixPath associated with whatever path
matplotlib is using during init.
Please fire up plain Python 3.6 on your host and try this:
>>> from pathlib import PosixPath
>>> from os import PathLike
>>> path = PosixPath('~/.something')
>>> isinstance(path, PathLike)
The >>> is the Python interactive prompt. I expect that to work, but
for something in your produced environment to obtain a PosixPath for
which that is False.
I think you should set up your environment differently, as a formal
venv with matplotlib etc installed normally, just to match you
user’s setup. i.e. not blithely copy their .local or whatever.
This is because whatever Python is running your code will get eg
PosixPath from its own base. If, somehow, the matplotlib startup is
getting an independent one, they’re distinct classes. I don’t have a
clear mechanism in mind for this failure, just inferring it from the
TypeError symptom.
This is on Rocky Linux 8.8.
So the history here is that I have a user who created a script that uses matplotlib. This import works fine in his original ‘/home/user/.local/’ environment. Note that the /home is not local but an NFS mount.
The NFSness should not matter.
The purpose of his script is for other users to make use of it. So we took his entire ./local python environment and copied it out to a team NFS share and adjusted permissions so that everyone can use it.
This I suspect is a source of your problem.
Are all your users running Rocky Linux 8.8?
We then configured a profile.d script on specific hosts that sets users
who login to point their $PYTHONPATH, $PATH and $LD_LIBRARY_PATH to
site-packages, bin and lib respectively to the new share. It all seems
to works fine until matplotlib is exported. At which point it fails
with the titled error.
Virtualenvs are tightly tied to their source Python install. If all your
users are using Rocky Linux 8.8 you’re probably good, but if not I’d be
making distinct virtualenvs for each end user host.
So you’ve got two valid ways forward:
if everyone is using the same Rocky Linux 8.8 you could make one
NFS-shared virtualenv
if there’s any variety, just make a virtualenv per host
You can still share the script etc via NFS.
So I’d be doing something shaped like this:
Create am epmty virtualenv:
/usr/bin/python3 -m venv /opt/venv-shared
That’s just an example path. It could b in /usr/local or in some
account which exists for shared software.
Note: the virtualenv is bound to the system /usr/bin/python3.
Get the modules you need in the virtualenv i.e. what your user have used
in their script, and put them in a requirements.txt file. Trite
initial setup:
echo matplotlib >/opt/venv-shared/requirements.txt
The name requirements.txt is an arbitrary convention. This way you
have some record of what’s going to be in the environment.
Install the required module into the virtualenv:
/opt/venv-shared/bin/python3 -m pip install -r /opt/venv-shared/requirements.txt
This way you know the environment will have what you wanted. Want more?
Update the requirements.txt file and rerun the command above.
Your end users can use this in a few ways:
directly invoke /opt/venv-shared/bin/python3 to run their scripts
have $PATH so that they find /opt/venv-shared/bin/python3 ahead of
/usr/bin/python3
make their onw virutalenvs (easy!) and install using the
/opt/venv-shared/requirements.txt file as their reference
If you’re sharing your user’s script, its shebang line should look like:
#!/usr/bin/env python3
to use whatever python3 the user has in their $PATH, or:
#!/opt/venv-shared/bin/python3
to hardwire the virtualenv you’ve build. Which would let you use it ad
hoc in other things because it knows where to go to get the necessary
virtualenv.
FINALLY, there’s the just-the-script approach. Put the script into your
NFS shared directory-of-scripts. Put the requirements.txt file there
too with a nice name. Any end user can prep their own local python to
match just be going:
python3 -m pip install -r /path/to/the/shared/requirements.txt
once, and then use the script from then on without further ado.
Happy to help troubleshoot this stuff and/or explain further.
Cheers,
Cameron Simpson [email protected]
On 28Dec2023 17:39, CJ via Matplotlib [email protected] wrote:
Thanks so much for your reply!
I’ve done the test as you suggested and it did in deed fail
$ python3
Python 3.6.8 (default, Oct 25 2023, 15:15:22)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import PosixPath
>>> from os import PathLike
>>> path = PosixPath('~/.something')
>>> isinstance(path, PathLike)
False
So I will try your idea of creating a shared venv. The two systems they will be using will be RL8.8 so the shared venv I think will work for this. Will reply when it is completed. Hopefully with success for posterity.
Thanks again!
On 03Jan2024 17:34, CJ via Matplotlib [email protected] wrote:
No response yet from the users but it worked great once I created the
shared venv. Thanks again!
