The graphic card is GIGABYTE GTX 1070 mini ITX with the latest driver. The Matlab is 2016a. The CUDA ToolkitVersion reported by Matlab is 7.5.
-
gpuDevice(1)
call takes 4 minutes to complete. Each call. Later I realized that actually any GPU call after Matlab start causes NVIDIA to rebuild its cache in
...\AppData\Roaming\NVIDIA\ComputeCache\
. This is what takes so long. A remark: only the first call of
train(net1,x,t,'useGPU','yes')
has this overhead, the subsequent calls are fast. But
gpuDevice(1)
forces the NVIDIA cache rebuild always.
-
A call to
train(net2,xg,tg)
stops with error:
"Error using gpuArray/gather.An unexpected error occurred during CUDA execution. The CUDA error was:CUDA_ERROR_ILLEGAL_ADDRESS"
. And this is end of the game. After a
train
call with arguments prepared by
gpu2nndata
no other calls to GPU can be made. Not even gpuArray(). All calls end with some error. Matlab must be restarted (and NVIDIA cache rebuild) to be able to use train with
useGPU
flag at least.
Anybody has similar experience? Or does anybody has a clue what's going on and if this is solvable? Thank you.
AN Update: I can confirm that the first part of the issue, cache rebuild, was solved by defining Windows system variables:
CUDA_CACHE_MAXSIZE=2147483648
CUDA_CACHE_DISABLE=0
The cache folders size then grew from 256MB to 440MB. But the initial gpuDevice call takes now 4s, not 4 minutes. I found the solution in some other MATLAB Central question:
parallel.gpu.CUDAKernel slow on GTX 1080
Both these issues stem from the fact that Pascal architecture cards are not fully supported by the current CUDA toolkit. This means that all kernels have to be compiled on the fly. This includes the 400+MB of NVIDIA and MathWorks kernels in MATLAB.
The second issue is the first time I've heard of an actual runtime error on a GTX 10xx card when running cuDNN, but no doubt it's a manifestation of the same issue, see: https://www.mathworks.com/support/bugreports/details/1439741
The first issue can't be resolved until a future version of MATLAB, but the second issue will hopefully be resolved at some point with a patch.
Hi Joss,
I have a titan x pascal running with matlabR16a. I tried to test the performance with:
b=rand([10000 10000]); bg=gpuArray(b); a=rand([1000 1000]); ag=gpuArray(a); for ii=1:1000 bg=sum(imfilter(bg, ag)); display(ii); end
then I will get the error like:
Error using gpuArray/rot90 An unexpected error occurred during CUDA execution. The CUDA error was: an illegal memory access was encountered
Is matlab 16 generally not supporting nvidia pascal card?
Thanks, hope you hear from you.
Hi everybody, I had the same problem as Dalibor. I have a gtx 1070 (version 399.07) and Matlab R2018a. Last week the example from matlab "DeepLearningFasterRCNNObjectDetectionExample.mlx" run properly and today the program make an CUDA_ERROR_ILLEGAL_ADDRESS. I had to clean reinstall the new driver and the problem was solved.
I hope that's will can help somebody else.
Hi everybody, I have the same problem as you. I got a K80 (driver version 384.130) with Matlab R2017a. I am trying to use a VGG19 network with large images (800*800 pixels) and I got this following message : "An unexpected error occurred during CUDA execution. The CUDA error was: an illegal memory access was encountered ", "CUDA_ERROR_ILLEGAL_ADDRESS " whereas there is still 5Go of free memory on the GPU.
The error occurs during the backpropagation.
According to vl_simplenn_display the data will just take 2Go from the memory.
I also try on a P100 with the same driver version (384.130) and I still have the problem.
I don't have any problem if I use smaller image (250*250 pixels).
Do you have any idea of what can be the problem ?
Find the treasures in MATLAB Central and discover how the community can help you!
