添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接

I'm trying to install Spacy/Prodigy on Google Compute Engine, and ran into problems so I was hoping someone here could share what works for them.

This is what I tried:

  • Created a "Debian GNU/Linux 9 (stretch)" image with 1 vCPU (n1-standard-1). It has Python 3.5.3 which isn't a great sign.
  • Installed PIP
  • Tried to install Spacy with pip install spacy but looks like the compiler isn't available. Below is my error message.
  • Is there another instance type that works better?

      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.5-3.5.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I./python -I./lib -I/usr/include/python3.5m -c ./python/ujson.c -o build/temp.linux-x86_64-3.5/./python/ujson.o -D_GNU_SOURCE
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    The following script should install all the system dependencies you need for spaCy and Prodigy for a clean Ubuntu 18.04 VM. It also installs a few other useful things:

    #!/usr/bin/env bash sudo apt-get update sudo apt-get install -y build-essential sudo apt-get install -y unzip libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils python-pip python-virtualenv python3-pip python3-venv \ python-dev python3-dev libopenblas-base libopenblas-dev

    You can then create a virtualenv and install your Prodigy wheel as follows:

    python3 -m venv spacy-env
    source spacy-env/bin/activate
    python3 -m pip install $path_to_your_Prodigy_wheel
    

    If you prefer to use Miniconda instead of pip, the following script will install Miniconda and create a conda environment with an optimized version of numpy:

    #!/usr/bin/env bash set -e MINICONDA_URL="https://repo.continuum.io/miniconda" MINICONDA_FILENAME="Miniconda3-latest-Linux-x86_64.sh" function print_usage { echo "Usage: ./install-miniconda" echo "This script installs Miniconda to /opt/miniconda and creates a conda environment with the optimized numpy" function run { # (re)create /opt/miniconda directory, give it to user sudo rm -rf /opt/miniconda sudo mkdir -p /opt/miniconda sudo chmod -R a+rwx /opt/miniconda # Download and run installer. wget $MINICONDA_URL/$MINICONDA_FILENAME -O /opt/miniconda/$MINICONDA_FILENAME sudo chmod a+rx /opt/miniconda/$MINICONDA_FILENAME /opt/miniconda/$MINICONDA_FILENAME -f -b -p /opt/miniconda # Allow others to execute miniconda sudo chmod -R a+rx /opt/miniconda/bin # Set up an environment and install optimized numpy /opt/miniconda/bin/conda create --prefix /opt/miniconda/numpy-mkl --copy -y numpy print_usage run "$@"

    If you want to use GPU, you’ll need to install CUDA and ideally the CUDNN library. You need to login to download the CUDNN installer, so there’s an extra step or two you need to do from your own computer. Once you have the installer on the VM, the following script should work:

    #!/usr/bin/env bash set -e # Install driver sudo add-apt-repository ppa:graphics-drivers/ppa sudo apt-get update sudo apt-get install -y nvidia-driver-396 libnvidia-compute-396 libnvidia-common-396 nvidia-utils-396 # Download toolkit and patch wget -O cuda_9.2.88_396.26_linux.run -c https://developer.nvidia.com/compute/cuda/9.2/Prod/local_installers/cuda_9.2.88_396.26_linux --quiet sudo mv cuda_9.2.88_396.26_linux.run /etc/install-cuda-9.2 wget https://developer.nvidia.com/compute/cuda/9.2/Prod/patches/1/cuda_9.2.88.1_linux --quiet sudo mv cuda_9.2.88.1_linux /etc/patch-cuda-9.2 sudo chmod a+rx /etc/install-cuda-9.2 sudo chmod a+rx /etc/patch-cuda-9.2 sudo /etc/install-cuda-9.2 --toolkit --silent --verbose sudo /etc/patch-cuda-9.2 --silent --accept-eula sudo cp /tmp/binaries/cudnn-9.2-linux-x64-v7.1.tgz /etc/cudnn.tgz sudo cp /tmp/runtime/cuda_bashrc /home/ubuntu/.bashrc sudo chmod a+rwx /home/ubuntu/.bashrc cd /tmp/binaries tar -xzf cudnn-9.2-linux-x64-v7.1.tgz sudo cp -r cuda/include/* /usr/local/cuda/include sudo cp -r cuda/lib64/* /usr/local/cuda/lib64 sudo chmod -R a+rx /usr/local/cuda/include/* sudo chmod -R a+rx /usr/local/cuda/lib64/*

    Once you have CUDA installed, you need to add the following lines to your .bashrc script:

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
    export PATH=$PATH:/usr/local/cuda/bin
    

    You can then run:

    pip install cupy
    pip install thinc_gpu_ops
                  

    I ran a few experiments to compare multiple vCPUs and a GPU for training a tagger. Here are the results:

  • 2 vCPUs: 12h15m
  • 4 vCPUs: 11h33m
  • 8 vCPUs: 9h37m
  • GPU: 3h43m
  • So you are much better off using a GPU and increasing the number of vCPUs is not worth the extra cost.

    @Jeff Yes, I agree: multiple CPUs for training are currently not advised, and in fact with the pip version of spaCy v2.0.18, too many CPUs can actually lead to too many threads being launched, which can decrease performance. In v2.1 of spaCy, we switch to single threading to prevent this kind of problem, and make it easier to run spaCy alongside other applications in a cloud environment.

    Hey Matthew,

    I don’t know if this is the place to ask this, but I am trying to run ner.batch-train on a GCE instance, with a GPU. I have followed your script to pretty much the line. However, when I come to running batch-train I get the error below:

        Exception ignored in: <bound method Stream.__del__ of <cupy.cuda.stream.Stream object at 0x7ff6db28cc88>>
    Traceback (most recent call last):
      File "cupy/cuda/stream.pyx", line 161, in cupy.cuda.stream.Stream.__del__
    AttributeError: 'Stream' object has no attribute 'ptr'
    Traceback (most recent call last):
      File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/mitch/spacy-env/lib/python3.6/site-packages/prodigy/__main__.py", line 331, in <module>
        controller = recipe(*args, use_plac=True)
      File "cython_src/prodigy/core.pyx", line 211, in prodigy.core.recipe.recipe_decorator.recipe_proxy
      File "/home/mitch/spacy-env/lib/python3.6/site-packages/plac_core.py", line 328, in call
        cmd, result = parser.consume(arglist)
      File "/home/mitch/spacy-env/lib/python3.6/site-packages/plac_core.py", line 207, in consume
        return cmd, self.func(*(args + varargs + extraopts), **kwargs)
      File "/home/mitch/spacy-env/lib/python3.6/site-packages/prodigy/recipes/ner.py", line 526, in batch_train
        baseline = model.evaluate(evals)
      File "cython_src/prodigy/models/ner.pyx", line 458, in prodigy.models.ner.EntityRecognizer.evaluate
      File "cython_src/prodigy/models/ner.pyx", line 460, in prodigy.models.ner.EntityRecognizer.evaluate
      File "/home/mitch/spacy-env/lib/python3.6/site-packages/spacy/language.py", line 548, in pipe
        for doc, context in izip(docs, contexts):
      File "/home/mitch/spacy-env/lib/python3.6/site-packages/spacy/language.py", line 572, in pipe
        for doc in docs:
      File "nn_parser.pyx", line 374, in pipe
      File "nn_parser.pyx", line 400, in spacy.syntax.nn_parser.Parser.parse_batch
      File "/home/mitch/spacy-env/lib/python3.6/site-packages/spacy/util.py", line 238, in get_cuda_stream
        return CudaStream() if CudaStream is not None else None
      File "cupy/cuda/stream.pyx", line 158, in cupy.cuda.stream.Stream.__init__
      File "cupy/cuda/runtime.pyx", line 331, in cupy.cuda.runtime.streamCreate
      File "cupy/cuda/runtime.pyx", line 334, in cupy.cuda.runtime.streamCreate
      File "cupy/cuda/runtime.pyx", line 144, in cupy.cuda.runtime.check_status
    cupy.cuda.runtime.CUDARuntimeError: cudaErrorUnknown: unknown error
    

    Using:
    Ubuntu 18.06
    Prodigy 1.7.1
    Spacy 2.0.18
    cupy 5.4.0
    cuda 9.2
    cudnn 9.2

    I can’t really workout where I have gone wrong, other than a different spacy to what you used?

    Cheers,
    Mitch

    arr1 = cupy.ones((4,4), dtype='float32') arr2 = copy.ones((4,4), dtype='float32') arr1 @ arr2

    I just want to test whether cupy can get the GPU working at all, basically.

    Hi Matthew,

    Thanks for a quick reply.

    So, the GPU is not working at all… I get the exact same unhelpful error when I run:

    import cupy
    arr1= cupy.ones((4,4), dtype='float32')
    

    Is it possible I have done something wrong when setting up the machine? The only thing that I can see is different is perhaps the cudnn version that I used. I used version 7.4.1.5

    Thanks Matthew, I managed to get to a point where your example cupy code works. I played about with versions until I got it to work - fun :).

    When running ner.batch-train is there anything specific I have to do to force spacy to use the GPU? I am seeing identical epoch times to when I was running on a CPU. My assumption (likely wrong) from reading thread was spacy would use the GPU if it is available? Or do I need to pass use_device=0 into begin training? Like I have seen you mention in another thread on this forum? Cheers

    You should be able to call spacy.prefer_gpu() in the recipe, somewhere before you call spacy.load(). I think that should be enough to use the GPU. I normally check either perf top or nvidia-smi to check that the GPU is being used. perf top is a bit less direct, but it shows you which C functions the time is being spent in.

    Thanks. nvidia-smi returns output that I would expect. Although, just looking at it again, I notice the CUDA version in the output is 10.0, where I am using 9.2 for everything else. I will get that to the right version and try again.

    Otherwise, I have a custom gpu wrapper recipe in which I call spacy.prefer_gpu() - which returns True. Also tried spacy.util.use_gpu(0) and spacy.require_gpu() And no luck. I get to a point where I get a ValueError: object __array__ method not producing an array from numpy. So my thinking is there is some incompatibility between thinc_gpu_ops/thinc/chainer and spacy…

    That seems strange. You’re still running the ner.batch-train recipe right? I know that the current version has some problems using the textcat on GPU, which are fixed in spaCy v2.1. But the NER recipes should work.

    Could you provide the full traceback?

    Yeah, I am still running the ner.batch-train recipe.

    Below is the full traceback. Let me know, if I can provide you with anymore information to help.

    spacy 2.0.18
    cupy 5.4.0
    thinc 6.12.1
    thinc_gpu_ops 0.0.4

    Loaded model en_core_web_lg
    Loaded 101839 evaluation examples from
    'attribute_dataset_home_garden_4676_cat_freq_thresh_1_data_read_mod_2642_tagging_cat_freq_thresh_combined_filtered_evaluation'
    Traceback (most recent call last):
    File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "main", mod_spec)
    File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
    File "/home/mitch/.local/lib/python3.6/site-packages/prodigy/main.py", line 331, in
    controller = recipe(args, use_plac=True)
    File "cython_src/prodigy/core.pyx", line 211, in prodigy.core.recipe.recipe_decorator.recipe_proxy
    File "/home/mitch/.local/lib/python3.6/site-packages/plac_core.py", line 328, in call
    cmd, result = parser.consume(arglist)
    File "/home/mitch/.local/lib/python3.6/site-packages/plac_core.py", line 207, in consume
    return cmd, self.func(
    (args + varargs + extraopts), **kwargs)
    File "/home/mitch/.local/lib/python3.6/site-packages/prodigy/recipes/ner.py", line 529, in batch_train
    baseline = model.evaluate(evals)
    File "cython_src/prodigy/models/ner.pyx", line 458, in prodigy.models.ner.EntityRecognizer.evaluate
    File "cython_src/prodigy/models/ner.pyx", line 460, in prodigy.models.ner.EntityRecognizer.evaluate
    File "/home/mitch/.local/lib/python3.6/site-packages/spacy/language.py", line 548, in pipe
    for doc, context in izip(docs, contexts):
    File "/home/mitch/.local/lib/python3.6/site-packages/spacy/language.py", line 572, in pipe
    for doc in docs:
    File "nn_parser.pyx", line 374, in pipe
    File "nn_parser.pyx", line 416, in spacy.syntax.nn_parser.Parser.parse_batch
    File "/home/mitch/.local/lib/python3.6/site-packages/numpy/core/numeric.py", line 632, in ascontiguousarray
    return array(a, dtype, copy=False, order='C', ndmin=1)
    ValueError: object array method not producing an array