添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接

Hey everyone. I'm interested in starting discussion about adding support to gRPC. Since cURL already support HTTP/2, I guess the main work would be parsing Protobuf (not sure though).

I am aware that there is a project called grpcurl but I believe adding native support in cURL would be bringing more adoptions to both gRPC and cURL.

I'm interested in getting opinion from you guys. Would that make sense? I would be happy totry wotking on that with other contributors.

I don't know much about gRPC, I have never seen nor tried grpcurl and I don't know the intended use cases for this. So let me ask:

Is the primary use case you see for this a way to specify the data to send to the gRPC server (text2binary) and then have the reverse functionality (binary2text) for the response, so that you can convert the response blob to something readable? Or is there more to it?

The MVP kind of feature is what you describe. gRPC is effectively Protobuf over HTTP/2. We need to be able to transform either JSON or Protobuf Text Format into binary (following Protobuf serialization rules) and do the opposite.

However, there is definitely more to it. For example:

  • gRPC supports unary, server stream, client stream, and bidi stream. This would probably need some way of sending and displaying multiple requests/responses. Currently grpcurl use interactive terminals to do that. An example of this is:
  • grpcurl -d @ 0.0.0.0:50051 $SERVICE.$METHOD <<EOF
    { "description": "new description" }
    { "description": "new description" }
    

    we use -d for specifying data and in this case we use @ to say that we send data through interactive terminal. And the <<EOF syntax is pretty standard.

  • Notice that in the previous example we send some JSON as data. This data will be matched against a Protobuf schema and if it doesn't comply to the schema (e.g unknown field, etc..), the command will fail (without the -allow-unknown-fields flag).
  • After that there is something called server reflection where the server exposes information about the services, methods and message that are register on it. This is nice for debugging and manual testing. An example of that is:
  • $ grpcurl 0.0.0.0:50051 describe [$SERVICE | $METHOD | $MESSAGE]
    $ grpcurl 0.0.0.0:50051 list

    where the first one give more info on the given parameter and the second one list the services register on the server.

  • Protobuf also has a text format which looks like JSON but is a little bit less verbose. I don't think grpcurl supports it but that could also be useful for test data in files.
  • Obviously there way more features that could be added but these are the ones that are majorly used outside of must have features like TLS, headers, ...

    Since gRPC is a data protocol over HTTP/2 without any URL syntax of its own, I don't think the support belongs in libcurl.

    As it is mostly a matter of converting from text to binary (and back) and send the data over h2, it seems to me like it should be perfectly doable to add support for gRPC in the curl tool.

    One of the largest work items for implementing something like this is to make sure that we have test cases to verify good and bad uses to the extent that we can be fairly sure things are working properly.

    Maybe a first work item would be to work out an idea and model for how to use the command tool to do gRPC. How would the curl command lines look for the N most common use cases?

    I was thinking of something like the following.

    Because we do not have GET, POST, ... in gRPC, we could replace the values of -X by something like: UNARY, SSTREAM (server streaming), CSTREAM (client streaming), BSTREAM (Bidirectional streaming).

    On top of that we will need to send some headers for gRPC to understand our requests. We need Content-Type: application/grpc and something like Method: /$Package.$Service/$Method or /$Service/$Method (if no package provided) to tell which endpoint we want to reach.

    For the data, we could use -d and pass some JSON data to it. We could have a simple object for UNARY and SSTREAM and an array of objects for streams. This would look like this:

    curl -H "Content-Type: application/grpc" -H "Method: /$Package.$Service/$Method" -X UNARY -d "{name: 'bagder'}" 0.0.0.0:50051
    curl -H ... -X SSTREAM -d "{name: 'bagder'}" 0.0.0.0:50051
    curl -H ... -X CSTREAM -d "[{name: 'bagder'}, {name: 'clement'}]" 0.0.0.0:50051
    curl -H ... -X SSTREAM -d "[{name: 'bagder'}, {name: 'clement'}]" 0.0.0.0:50051

    Finally, I believe the -X could be omitted when server reflection is enabled.

    Because we do not have GET, POST, ... in gRPC

    but since gRPC is done over HTTP/2, we still have them involved don't we? They're just in the layer outside of gRPC. What HTTP methods are gRPC using otherwise?

    we could replace the values of -X by something like: UNARY, SSTREAM (server streaming), CSTREAM (client streaming), BSTREAM (Bidirectional streaming).

    To me, it feels like this is better suited for a new option. An option that then also knows about the available options and will not accept any non-supported version. But maybe I'm not seeing the full picture yet.

    we will need to send some headers for gRPC to understand our requests

    I think we can be inspired by how --json works: when used it will set a few headers. A --grpc option could work similarly. We need a way to tell the tool that it should speak gRPC for this transfer. Perhaps it also sets the stream type?

    we could use -d and pass some JSON data to it

    this then requires JSON parsing ability, which curl currently does not have

    curl -H ... -X SSTREAM -d "[{name: 'bagder'}, {name: 'clement'}]" 0.0.0.0:50051

    In this case, is that "0.0.0.0:50051" a shortcut for saying you send this to the gRPC server on port 50051 on locahost or is it something else? (50051 seems like such an peculiar port number to use in an example)

    Do gRPC servers not act on a path on HTTP ? Can it not handle HTTP authentication or HTTP redirects?

    Right. What I meant was that GET, POST, etc.. are not gRPC primitives. They are encapsulated by it.

    To me, it feels like this is better suited for a new option.

    This sounds like a good idea. I don't know much about the internals of curl but if we can add a new dedicated option that would be nice.

    A --grpc option could work similarly

    This would also be nice. We wouldn't have the manually set the application/grpc Content-Type header. The use of this flag would add it automatically.

    Perhaps it also sets the stream type?

    I'm not really sure how the --json flag works (going to check out) but for the stream type what do you have in mind?

    this then requires JSON parsing ability, which curl currently does not have

    Either we go with JSON or we would have to chose another format. I can see two other ways:

  • A binary file. The problem is that this involves a first step for creating the binary file with code or with the protobuf compiler. I believe this is not user-friendly.
  • Using the Protobuf Text Format's message list. However I believe this is very similar to JSON and thus why not supporting JSON directly.
  • Now the question is: should we add support to that in curl or should we rely on other library?

    In this case, is that "0.0.0.0:50051" ...

    50051 is a port often used in gRPC. This is pretty much because most tutorials will use it but there is nothing tying gRPC application to this port.

    Do gRPC servers not act on a path on HTTP ? Can it not handle HTTP authentication or HTTP redirects?

    I'm not entirely sure about this for the lower level, but at the user level we never work with path, this is the point of RPCs. authentication is feasible but through other means (interceptors, ...), and I'm not aware of any redirects happening in gRPC.

    Could we default to HTTPS and fail miserably if the server doesn't use TLS?

    We could probably consider that, sure, but it seems a little premature to decide those teeny details already.

    Right. I was showing an example on localhost (in dev). For those we generally set an authority header and self-signed certificates.

    Self-signed is still signed, there's no difference in verification method for curl for that. Just a matter of that CA store you would use.

    We also have --insecure / -k in curl but we do not like to promote the use of this since it tends to stick and creep into production.

    So it seems that we could do it right?

    Should we try to list the options and the usage somehow? Maybe like an output of --help

    Is there any process that we need to go through in order to consider implementing this? a document, a vote, or something like this.

    Is there anything your not convinced of? The usefulness, the feasibility, etc...

    I think all this is doable and I don't think we have identified any show-stopper.

    Laying out some ideas in terms of how dedicated command line options are envisioned to work is good and useful, as it will also offer other interested people a chance to ponder and contribute feedback already before there is code to test. There is no doubt in my mind that these things might need a little experimenting and offering users to "get a feel" for them before we know for sure how everything should be done.

    There is no process. I think gRPC support already falls into the realm of what could be acceptable functionality for curl. 20% of curl users answered in the 2023 user survey that they would like to see "gRPC support".

    We normally ship brand new features/protocols as "experimental" first for some time, to let everyone try it out first and allow us to change things before we carve the rules in stone and they are never to be changed again. We should do that for gRPC as well.

    All this said: there is a lot of work needed before we are there:

  • code and sensibly working command line options (for old and new gPRC users, for new and old curl users, on multiple platforms)
  • documentation - everyone needs to be able to learn how to use this
  • tests - an important part as we cannot know that everything works solidly and safely until we have a decent set of dedicated tests for the functionality
  • Not really, no.

    I think you should start with not thinking about it as a new protocol. Protocols in curl are usually the URL scheme and since gRPC is still HTTP(S), we do not think of it as a new protocol. It is more of a feature for HTTP(S).

    As an absolutely first shot you probably want to try to just store a binary gRPC message in a file on disk and then send that to a gRPC server and see how that works:

    curl --data-binary @grpcfile -H "Content-Type: application/grpc" https://grpc.example.com:50051/ -o grpcresponse
    

    (I'm guessing, it might need a little more tweaking to be accepted.)

    Then consider the next steps to take to convert that into a more convenient command line.

    Ok. Using a unary API is indeed feasible. I will try my best to document what I learned so far.

    Let's start with the following proto file (greet.proto):

    syntax = "proto3";
    package greet;
    message GreetRequest {
      string first_name = 1;
    message GreetResponse {
      string result = 1;
    service GreetService {
      rpc Greet(GreetRequest) returns (GreetResponse);
    

    It is not that important what the RPC endpoint is doing but for this example I simply made it return "Hello " + the first_name sent in GreetRequest.

    From that we can create a binary representation of the GreetRequest message. To make it simple we can use the Protobuf Text Format and generate binary with the protoc --encode flag. request.txt will look like this:

    first_name: "Clement"
    

    and then we execute the following command to get the binary:

    (cat request.txt | protoc --encode=greet.GreetRequest greet.proto) > payload.bin

    Now, using the payload.bin is not enough to make a valid request. After checking some WireShark logs, I realized that we need to prepend the Protobuf binary with a byte saying wether or not the binary is compressed and the equivalent of Content-Length header (which seems to be skipped by gRPC). So let's start by serializing 0 (not compressed) in a request.bin file:

    printf "0: %.2x" 0 | xxd -r -g0 >> request.bin

    Then we append 4 bytes equal to 9 (the length of Clement + 2 bytes for Protobuf metadata) in tmp:

    printf "0: %.8x" 9 | xxd -r -g0 >> request.bin

    And finally, we can add the Protobuf payload after that:

    cat payload.bin >> request.bin

    request.bin is now usable for sending a request. For now, I only tried on localhost with and without TLS. Without TLS, we can execute the following command:

    curl --http2-prior-knowledge --insecure --data-binary @request.bin --header "Content-Type: application/grpc" http://0.0.0.0:50051/greet.GreetService/Greet --output response.bin

    and with TLS, we can use (insecure only because I'm on localhost):

    curl --http2 --cacert ssl/ca.crt --tlsv1.2 --tls-max 1.2 --insecure --data-binary @request.bin --header "Content-Type: application/grpc" https://0.0.0.0:50051/greet.GreetService/Greet --output response.bin

    We now have the response.bin. Once again, there is some wrapping happening. We can skip the first 5 bytes (1 for compression, 4 for Content-Length) and decode the payload into GreetResponse:

    tail -c +6 response.bin | protoc --decode=greet.GreetResponse greet.proto
    result: "Hello Clement"

    I need to admit this is quite hard for now since I did it by hand in bash. However, this can definitely by solved in code. I will try the other kind of Endpoints (server, client and bidi streaming). Hope this helps