Constructing gateway for ML media services using Go Use each REST and gRPC Common logic Handle media files Conclusion:

Artificial Intelligence

Constructing gateway for ML media services using Go Use each REST and gRPC Common logic Handle media files Conclusion:

admin

March 18, 2023

Constructing gateway for ML media services using Go
Use each REST and gRPC
Common logic
Handle media files
Conclusion:

At Neiro.ai we construct Generative AI tools for personality cloning. We have now , , and other AI techs in a single user-friendly web interface. We also provide an API for Business Customers.

Our machine-learning services are maintained by different engineering teams and have different APIs. We developed the gateway to observe these services and wrap them into one convenient service.

We’d like the gateway to satisfy the next requirements:

requests coming from our mobile apps
requests coming from our Web Studio
All infrastructure — rate limits, token verification, tracing, monitoring, and so forth
Generally, requests Depending on the case, we wish to send either URL or binary file to the gateway.

As a startup, we wish to maneuver fast and decided to construct a gateway using Go infrastructure.

We offer this text with a sample echo application. The source code might be found here: https://github.com/mynalabsai/grpc_gateway_media_example

Service interface described in protobuf format

With a view to fulfill the primary requirement, we decided to make use of , which generates a proxy from JSON to protobuf object format and might redirect the remaining request to the gRPC server.

gRPC-gateway generates reverse proxy from REST to gRPC

Listed below are three things we want to do:

Add a handler from that may broadcast the request from to . It gets attached to http.ServeMux. Here’s the way it’s done:

Mux will handle /v1/echo by proxying request to localhost:9090 grpc endpoint

Start the gRPC server
For the HTTP server, filter out Content-type: application/grpc, and send it on to the gRPC-handler. We do that by utilizing a method from this text. The remaining of the requests are processed with the prevailing mux and are assigned to proxy.

The redirect of the request to the specified ML service is a callback. It known as contained in the shared wrapper. The shared wrapper handles the logic related to the infrastructure and isn’t influenced by request type.

In our example, the echo service callback waits for a second after which returns its input as a response. Wrapper measures the latency of callback and might be reused for other requests.

Through the use of the wrapper we are able to measure the duration for each form of requests

When you run with requests containing raw data in bytes, an error occurs.

curl for HTTP endpoint

grpcurl for gRPC endpoint

It is because data field type in our data model is bytes . It’s unattainable to transmit bytes within the JSON text format, which is why we use base64 representation of the info. For it to work, we’ll pass base64(abacaba), which is YWJhY2FiYQ==, as a substitute of abacaba for field data

That is an expected echo response for HTTP

That is an expected echo response for gRPC

Mainly, if the goal is to transmit a media file within the request body while preserving the structure, we create a string with a base64 representation of the file.

Nonetheless, this string will probably be too long for an audio or a brief video, which makes it inconvenient to work with JSON. Besides, the user must have the option to manually check that the request will return a particular file. In the primary iteration, we made a python script that encodes a file and outputs a line that should be explicitly copied to the suitable place in JSON. Besides being inconvenient, anyone who desires to make a request might want to have a script.

An alternative choice is to support files via URLs within the API. We allow this feature, but manual testing could also be tricky: you want to send the audio to the item storage. We wanted an choice to send a neighborhood file.

The proper approach to send files could be to support multipart/form-data requests. Yet, the ecosystem doesn’t allow it. The authors suggest handling file downloads individually. We want to make a request in the identical format as before, but with sending files via multipart/form-data as a substitute of explicitly sending them contained in the structure with the request.

That’s why we arrived at the answer of using macros that suggest which string fields have to be moreover expanded through the base64 representation of a specified file.

Here is how our request looks like

Now we are able to send request and specify local file

Here’s how you can do such processing. First, we install middleware. After receiving multipart/form-data, middleware recursively parses the JSON with data and expands macros.

The brand new request follows the identical path as before. That is the way it looks in Go:

That is how we process multipart/form-data and expand macros

And the handler is barely updated:

Filter application/grpc for gRPC server, expand macros and use reverse proxy for the remaining

That is an outline of how we used to handle media files in our ML services.

We hope you enjoyed the ride and learned something useful on your projects. Once more, yow will discover full example in our github: https://github.com/mynalabsai/grpc_gateway_media_example

At Neiro.ai, we’re all about moving fast, and the gateway we built using infrastructure has allowed us to just do that. Through the use of , we were capable of fulfill our requirements to process each and requests, and share infrastructure logic for rate limits, token verification, tracing, monitoring, and more. We even found a neat workaround to handle media files by utilizing macros expansion. If you happen to’re in an analogous boat, we highly recommend trying out as an answer. Thanks for reading, and stay tuned for more AI adventures.