gRPC

Overview

gRPC is an open-source framework for remote procedure calls, a way for one program to call a function in another program over the network as if it were local. You describe a service and its messages in a schema file using Protocol Buffers. Tools then generate client and server code in many languages. At runtime, gRPC sends compact binary messages over HTTP/2, supports streaming in both directions, and handles deadlines, cancellation, metadata and status codes in a standard way.

An everyday analogy is a restaurant with a printed order pad. Instead of customers shouting free-form requests at the kitchen, every order uses the same pad with fixed boxes: dish, quantity, table, allergies. The kitchen knows exactly what to expect, orders are quick to write and hard to misread, and a new waiter learns the system in minutes. Protocol Buffers are the order pad, and gRPC is the reliable runner who carries pads between dining room and kitchen, sometimes many at once.

gRPC is popular for internal communication between microservices, where strong types, low overhead and streaming matter. In AI infrastructure it is common too: model servers such as Triton and KServe offer gRPC endpoints, the OpenTelemetry protocol OTLP can run over gRPC, and many vector databases, including Milvus and Qdrant, provide gRPC APIs.

What it is

gRPC combines a few layers.

Official implementations exist for many languages, including C++, Java, Go, Python, C#/.NET, Node.js, Ruby, PHP, Kotlin, Dart and Swift. gRPC-Web lets browsers call gRPC services through a proxy, because browsers do not expose the low-level HTTP/2 control gRPC needs.

Where it came from

Google used an internal RPC system called Stubby for many years to connect its services. In February 2015 Google released gRPC as an open-source, general-purpose successor built on public standards such as HTTP/2 and Protocol Buffers. Version 1.0 followed in 2016. In March 2017 Google contributed gRPC to the Cloud Native Computing Foundation (CNCF), where it is an incubating project. The project is licensed under Apache 2.0.

Protocol Buffers itself was developed inside Google and open-sourced in 2008. The current language versions are proto3 and the newer "editions" model, which lets features be set per file rather than by a fixed syntax version.

How it works

1. Define the contract

You write a .proto file describing the service:

syntax = "proto3";
package scoring.v1;

service Scorer {
  rpc Score (ScoreRequest) returns (ScoreResponse);
  rpc ScoreStream (stream ScoreRequest) returns (stream ScoreResponse);
}

message ScoreRequest  { string id = 1; repeated float features = 2; }
message ScoreResponse { string id = 1; float probability = 2; }

Each field has a number. The numbers, not the names, identify fields on the wire, which is what allows schemas to evolve safely.

2. Generate code

Running the compiler with language plugins produces message classes, a client stub with a Score method, and a server base class to implement.

3. Implement and serve

The server implements the methods and registers them. The client creates a channel to the server's address and calls methods on the stub.

4. Send over HTTP/2

Each call becomes an HTTP/2 stream. Request metadata travels as headers, the serialised message as length-prefixed frames, and the result ends with trailers containing a status code. Many calls share one long-lived TCP connection.

5. Stream when useful

In streaming calls, either side can send many messages over the same stream. A client can stream sensor readings; a server can stream tokens from a language model as they are generated; both can talk at once in a bidirectional stream.

6. Control time and failure

The client sets a deadline. If it passes, the call ends with DEADLINE_EXCEEDED and the server can stop work. Cancellation propagates to downstream calls. Status codes such as UNAVAILABLE, INVALID_ARGUMENT and PERMISSION_DENIED give consistent error semantics, and retry policies can be configured.

Key concepts and terms

A worked example

This scenario is illustrative. A payments company runs a fraud model behind a JSON REST API. The checkout service calls it once per transaction, about 3,000 times a second at peak. Payloads contain 400 numeric features. JSON encoding makes each request about 9 KB, parsing costs significant CPU, and the p99 latency of the call is 45 ms. Field name typos between teams cause silent errors twice in one quarter.

The team defines a FraudScorer service in a .proto file with a Score unary method and a ScoreBatch client-streaming method for back-office replays. Features become a repeated float field. Code is generated for the Java checkout service and the Python model server. The same 400 features now encode to about 1.6 KB per request, because floats are packed binary values without field names. Connections are long-lived and multiplexed.

Measured in load tests, p99 latency for the call drops from 45 ms to 18 ms, and CPU on the checkout service falls noticeably. The client sets a 50 ms deadline; if scoring misses it, checkout applies a rules-based fallback. Mismatched fields now fail at compile time. When the data science team adds a feature, they add a new field number; older clients keep working and simply do not send it.

Strengths and limitations

Strengths:

• Strongly typed contracts with generated code in many languages.
• Compact binary encoding and efficient HTTP/2 transport.
• Built-in streaming in both directions.
• Standard deadlines, cancellation and status codes across languages.
• Clear schema evolution rules through field numbers.
• Good fit for service meshes, which understand HTTP/2 traffic.

Limitations:

• Browsers cannot call gRPC directly; gRPC-Web and a proxy are needed.
• Binary payloads are harder to inspect than JSON without tooling.
• Load balancing long-lived HTTP/2 connections needs L7-aware proxies or client-side balancing.
• Less natural for public APIs aimed at many third parties.
• Code generation adds build steps and tooling to manage.

How it compares with alternatives

Practical guidance and best practices

1. Version packages in .proto files, for example scoring.v1, and keep backward compatibility within a version.
2. Never reuse or change field numbers; reserve numbers of removed fields.
3. Set a deadline on every call and propagate it to downstream calls.
4. Reuse channels; creating a channel per request wastes connections.
5. Use client-side load balancing or an L7 proxy, so long-lived connections spread across replicas.
6. Use TLS, and mTLS between internal services, with token metadata for user identity.
7. Return meaningful status codes and use rich error details rather than generic UNKNOWN.
8. Enable the health checking service and, in development, server reflection for debugging tools.
9. Lint and check .proto changes for breaking changes in CI.

Common mistakes

• Omitting deadlines, so stuck calls hold resources indefinitely.
• Reusing a removed field's number for a new field, corrupting data for old clients.
• Exposing gRPC directly to browsers without gRPC-Web or a compatible gateway.
• Relying on a plain L4 load balancer and sending all traffic from a client to one server.
• Creating huge messages instead of streaming, hitting default size limits.
• Retrying non-idempotent methods automatically.

Frequently asked questions

Is gRPC faster than REST?

Often, for service-to-service calls, because of binary encoding and connection reuse over HTTP/2. The real gain depends on payload size, network and implementation, so measure for your workload.

Can I use gRPC from a web browser?

Not directly. Browsers need gRPC-Web, which requires a proxy such as Envoy or server support, or an alternative protocol such as Connect that is compatible with gRPC services.

Does gRPC require Protocol Buffers?

Protocol Buffers is the default and by far the most common choice. gRPC can in principle carry other serialisation formats, but most tooling assumes Protocol Buffers.

How does gRPC handle errors?

Every call ends with a status code and optional message and details. Codes such as NOT_FOUND, UNAVAILABLE and DEADLINE_EXCEEDED have defined meanings across all languages.

Is gRPC good for streaming LLM output?

Yes. Server streaming lets a server send tokens as they are generated over one call, and many model servers offer gRPC streaming endpoints for this purpose.

How do I change a gRPC API safely?

Add new fields with new numbers, keep old ones, and reserve numbers of fields you delete. For incompatible changes, create a new package version such as v2 and run both during migration.

Glossary

• gRPC: an open-source RPC framework from Google, now in the CNCF.
• Protobuf: short for Protocol Buffers.
• IDL: interface definition language, such as the .proto syntax.
• HTTP/2: the HTTP version with multiplexed streams that gRPC uses.
• Stub: generated client code.
• Bidirectional streaming: both sides sending message streams concurrently.
• Trailer: HTTP/2 headers sent after the body, carrying gRPC status.
• Reflection: a service that lets tools discover a server's API at runtime.
• Field number: the identifier of a protobuf field on the wire.

Summary

gRPC is a high-performance RPC framework that uses Protocol Buffers for typed contracts and compact messages, and HTTP/2 for multiplexed, streaming transport. Open-sourced by Google in 2015 as a successor to its internal Stubby system and contributed to the CNCF in 2017, it is a standard choice for internal service communication and for many AI serving and data systems. Use it with versioned schemas, never-reused field numbers, deadlines on every call, proper load balancing and TLS, and keep REST or GraphQL for browser-facing and public APIs.

Related

It belongs to Architecture in my stack, under MLOps and AI infrastructure. Others in the same group: