YataiService gRPC server not accessible from outside Docker Container · Issue #1035 · bentoml/BentoML

link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

直爽的板凳 · GetProcessTimes· 2 月前 ·

发呆的春卷 · python如何去掉字符串首 | ...· 3 月前 ·

威武的火龙果 · 将域控制器升级到 Windows ...· 4 月前 ·

慷慨的蚂蚁 · TypeScript ...· 7 月前 ·

酒量小的拐杖 · [OpenWrt Wiki] Xiaomi ...· 10 月前 ·

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Describe the bug
I start up the YataiService on Docker (RHEL Base Image) turning OFF the web ui (--no-ui mode)

bentoml yatai-service-start --no-ui --debug --verbose
[2020-08-27 10:08:20,242] DEBUG - Enabling gRPC server reflection for debugging
* Starting BentoML YataiService gRPC Server
* Debug mode: on
* Web UI: off
* Running on 127.0.0.1:50051 (Press CTRL+C to quit)
* Usage:
*  Set config: `bentoml config set yatai_service.url=127.0.0.1:50051`
*  Set env var: `export BENTOML__YATAI_SERVICE__URL=127.0.0.1:50051`
* Help and instructions: https://docs.bentoml.org/en/latest/guides/yatai_service.html
At this point I have an empty model repository.

I ssh into the same docker container that is running the yatai-service and try running the following command:
bentoml list
BENTO_SERVICE    AGE    APIS    ARTIFACTS
I get an empty response as shown above - perfectly valid - the yatai-service is responding and accessible from within the container.
I have exposed the local docker port 50051 to the external world. I now try querying this yatai-service from other environments.
Attempt 1: Access yatai-service from my Windows Desktop
bentoml config set yatai_service.url=http://yatai-service-url-on-docker
bentoml list
  File "...\lib\site-packages\bentoml\yatai\client\bento_repository_api.py", line 189, in list
    return self.yatai_service.ListBento(list_bento_request)
  File "...\lib\site-packages\grpc\_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "...\lib\site-packages\grpc\_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "DNS resolution failed"
        debug_error_string = "{"created":"@1598523838.924000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1598523838.924000000","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":262,"referenced_errors":[{"created":"@1598523838.924000000","description":"DNS resolution failed","file":"src/core/ext/filters/client_channel/resolver/dns/native/dns_resolver.cc","file_line":202,"grpc_status":14,"referenced_errors":[{"created":"@1598523838.924000000","description":"OS Error","file":"src/core/lib/iomgr/resolve_address_windows.cc","file_line":96,"os_error":"The specified class was not found.\r\n","syscall":"getaddrinfo","wsa_error":10109}]}]}]}"
This is probably the most important part of the trace:
details = "DNS resolution failed"
"os_error":"The specified class was not found.\r\n","syscall":"getaddrinfo","wsa_error":10109
Now, I wasn't expecting a DNS Resolution failed - because I can confirm that the URL is accessible

For e.g. Running a

curl http://yatai-service-url-on-docker from my desktop (git bash) 

gives me an "Internal Server Error" - which atleast shows the domain can be resolved.

I must however admit - I was expecting the output from curl to look more like the below:
curl: (1) Received HTTP/0.9 when not allowed
The above message is when I run a similar curl command on port 50051 when the yatai-service is running on my local desktop
Attempt 2: Access yatai-service from another RHEL docker container (not the one running the yatai-service)
bentoml config set yatai_service.url=http://yatai-service-url-on-docker
bentoml list
  File ".../lib/python3.6/site-packages/bentoml/yatai/client/bento_repository_api.py", line 189, in list
    return self.yatai_service.ListBento(list_bento_request)
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 562, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Trying to connect an http1.x server"
        debug_error_string = "{"created":"@1598524163.124605195","description":"Error received from peer ipv4:XX.XXX.XXX.XX:XXXXX","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Trying to connect an http1.x server","grpc_status":14}
This is probably the most important part of the trace:

"grpc_message":"Trying to connect an http1.x server"
This error looks more promising than what I received on windows - but I now get an error message I don't understand very well.
We know for a fact that the yatai-service works when accessed from within the docker container.

But - why does it NOT work when accessed from the outside ?
The errors might be something very specific to grpc and my environment (and NOT Bento).

However, I need help on the following:
Are there any considerations I have to make in my environment given that yatai-service uses gRPC ? (I am new to gRPC)
By seeing the error messages above - Do you have any pointers on what I should be looking at next ?
Environment:

Here is the version of s/w I am using:
Python 3.8.0
bentoml, version 0.8.4
RHEL Docker Image (not using the Dockerfile that ships with BentoML)
          I got this working in my environment. It seems like I had to add a special Service Discovery configuration on my end to support gRPC Services. I am closing this item.
Having said that - #1034 (Yatai Web UI NOT responding is still an issue for me).

Thanks
I got this working in my environment. It seems like I had to add a special Service Discovery configuration on my end to support gRPC Services. I am closing this item.
Having said that - #1034 (Yatai Web UI NOT responding is still an issue for me).

Thanks
What did you add and where?
Thanks!
          @hrimhari  - My environment uses  traefik - https://docs.traefik.io/ for routing incoming requests to Services. I had to configure the environment to route requests by "port" (other options include routing by "path", "header" etc) in order for the bentoML CLI client to consume the remote yatai gRPC Service.
Hope that helps. Right now, I don't understand too much of either gRPC or traefik to now why that makes a difference.
          @jiyer2016 sorry for pinging you in this old issue.

but can you share some code for your route configuration?

actually i'm using something like this:
host: yatai.cluster-qa.misterturing.com

http:

paths:

- path: /

backend:

serviceName: yatai-service

servicePort: 3000

- path: /

backend:

serviceName: yatai-service

servicePort: 50051

but does not work.