gRPC Protobuf Reference

gRPC Protobuf is the foundation of high-performance, strongly-typed service communication in distributed systems. Use this reference when defining service interfaces, generating client and server code, or debugging serialization issues in gRPC services. This guide answers: *How do I define a service? How do I encode data? How do I version a service? What happens when a field is missing or a type mismatches?* When a gRPC call fails silently, this is the first place to look.

Defining Services and Messages

Message Structure

Define data structures using message blocks in .proto files. Each message is a named collection of fields, each with a type and a unique field number.

syntax = "proto3";

package example.v1;

import "google/protobuf/timestamp.proto";

message User {
  int64 id = 1;
  string name = 2;
  string email = 3;
  bool is_active = 4;
  repeated string roles = 5;
  google.protobuf.Timestamp created_at = 6;
  map<string, string> metadata = 7;
}

Field numbers must be unique within a message. Use int32, int64, string, bool, bytes, and double as built-in types. Protobuf supports repeated (array), map, and nested messages.

Service Definition

Define a gRPC service using the service keyword. Each method has a request and response message.

service UserService {
  rpc GetUser(GetUserRequest) returns (User) {
    option (google.api.http) = {
      get: "/v1/users/{id}"
    };
  }

  rpc ListUsers(ListUsersRequest) returns (ListUsersResponse) {
    option (google.api.http) = {
      get: "/v1/users"
    };
  }

  rpc CreateUser(CreateUserRequest) returns (User);
}

Methods may be rpc, streaming, or server-streaming. Use stream keyword for bidirectional streaming.

Field Encoding and Options

Field numbers must be non-negative integers. The first 16 bits of a field number are used in the varint encoding, so field numbers 1–16 are encoded in 1 byte, and higher numbers use more bytes.

message Order {
  uint32 id = 1;                   // 1 byte if id < 128
  string customer_name = 2;        // UTF-8 encoded
  bytes raw_data = 3;              // length-prefixed bytes
  repeated string items = 4;       // each string is length-prefixed
  int64 created_at = 5;            // varint-encoded
  bool active = 6 [packed = true]; // packed array of bools
  map<string, double> prices = 7;  // key-value pairs, each key and value varint-encoded
}

Use packed = true to store repeated fields as a single contiguous sequence of values. This reduces size and improves decoding speed.

Generating Code and Compiling Protobuf

`protoc` Command Line

Use protoc to compile .proto files into language-specific code.

protoc \
  --proto_path=src/proto \
  --go_out=gen/go \
  --go-grpc_out=gen/go \
  --java_out=gen/java \
  --python_out=gen/python \
  src/proto/example/v1/user.proto

--proto_path: directory where .proto files are located.
--go_out: generates Go code (.pb.go).
--go-grpc_out: generates gRPC service stubs (.pb.gw.go).
--java_out: generates Java .java files.
--python_out: generates Python .py files.

Language-Specific Code Generation

Go

Generated Go code includes:

Message structs with GetField() and SetField() methods.
Marshal() and Unmarshal() methods.
func (m *User) ProtoReflect() protoreflect.Message.

user := &example.User{
  Id:       123,
  Name:     "Alice",
  Email:    "[email protected]",
  IsActive: true,
}

data, err := user.Marshal()
if err != nil {
  log.Fatal(err)
}

var parsed example.User
err = parsed.Unmarshal(data)

Java

Java uses GeneratedMessageV3 as base class. Use Builder pattern for construction.

User user = User.newBuilder()
  .setId(123)
  .setName("Alice")
  .setEmail("[email protected]")
  .setActive(true)
  .build();

Protobuf uses ProtoReflection for dynamic access and schema introspection.

Data Serialization and Wire Format

Wire Types

Protobuf encodes data using wire types. Each field is serialized as a tag (field number + wire type) followed by the value.

Wire Type	Meaning
0	Varint (int32, int64, bool, enum)
1	64-bit (double, fixed64)
2	Length-delimited (string, bytes, message, repeated)
5	32-bit (float, fixed32)

Example: User message with id = 123 and name = "Alice":

Tag: 1 (field 1, wire type 0) → 123 in varint → 0x7B
Tag: 2 (field 2, wire type 2) → length 5 → "Alice"

Result: 0x08 0x7B 0x12 0x05 0x41 0x6C 0x69 0x63 0x65

Field Numbering and Backward Compatibility

Field numbers are critical. Adding a field with the same number as a previous one overwrites data.

// v1
message User {
  int64 id = 1;
  string name = 2;
}

// v2: new field
message User {
  int64 id = 1;
  string name = 2;
  string email = 3;  // added in v2
  string phone = 4;  // new field, but typo: should be "phone_number"
}

// v3: renamed and reordered
message User {
  int64 id = 1;
  string name = 2;
  string email = 3;
  string phone_number = 4;  // corrected
}

When a v1 client talks to a v3 server:

phone field is ignored (unknown field 4).
phone_number field is read correctly.
id and name are decoded from varint.

Silent failure: a field added in v2 but not used in v3 causes data loss.

Repeated Fields and Packed Encoding

Repeated fields are encoded as length-delimited values.

repeated string roles = 5;

When packed = true, repeated values are packed into a single buffer.

repeated string roles = 5 [packed = true];

A roles list ["admin", "user"] is encoded as:

Tag: 5 (wire type 2) → length 11 → "admin" (5 bytes) + "user" (4 bytes)

Packed vs. Unpacked:

Unpacked: each value is length-delimited → more overhead.
Packed: single length-delimited block → smaller, faster.

Use packed = true for all repeated fields unless you need sparse or non-contiguous data.

Versioning and Interoperability

Field Numbering Best Practices

Use 1–1000 for core fields; leave room for expansion.
Field numbers are immutable after release.
Use optional, required, and repeated fields wisely.

message Config {
  int32 version = 1 [required = true];
  string name = 2;
  repeated string features = 3 [packed = true];
  bool enable_logging = 4 [default = true];
  int32 max_connections = 5 [default = 100];
}

required fields must be present in serialized data; missing fields cause deserialization error.
default values are used when a field is omitted.

Default Values and Missing Fields

Type	Default Value
`int32`	0
`int64`	0
`float`	0.0
`double`	0.0
`bool`	false
`string`	""
`bytes`	""
`enum`	first value
`message`	new instance

Silent failure: a field is expected but missing. If Config.enable_logging is false in proto, and the client sends enable_logging: false, the server receives true (default).

Optional vs. Required Fields

required fields must be present in the serialized message. If missing, Deserialize() returns nil or throws an error.
optional fields may be omitted. If omitted, the field uses its default value.

message Request {
  string query = 1 [required = true];
  int32 timeout = 2 [optional = true, default = 30];
  string format = 3 [optional = true, default = "json"];
}

A request with only query but no timeout or format is valid. If query is missing, deserialization fails.

Backward and Forward Compatibility

Adding a new field in a message is backward-compatible.
Removing a field is forward-compatible.
Changing field type (e.g., int32 → int64) is backward-compatible.

Critical failure: a new server expects a timestamp field but the client sends int64 instead. The field is present but misinterpreted.

Field Options and Custom Options

Use option to add metadata.

syntax = "proto3";

package example.v1;

option go_package = "github.com/example/v1;v1";
option java_multiple_files = true;
option java_outer_classname = "UserServiceProto";

message User {
  int64 id = 1 [(gogoproto.moretags) = `json:"id,omitempty" xml:"id" yaml:"id"`];

  // Custom options
  string email = 2 [
    (gogoproto.customname) = "EmailAddress",
    (gogoproto.jsontag) = "email",
    (gogoproto.moretags) = `xml:"email,attr"`
  ];
}

Use gogoproto (Go-specific) or google.api (HTTP annotations) to customize code generation.

Debugging and Failure Modes

Common Errors and Their Causes

invalid field number: a field number is missing or out of range in the wire format.
malformed message: length-delimited field has incorrect length.
missing required field: a required field is not present in the serialized data.
unexpected field: a field number is unknown in the message schema.

Protobuf Debugging Tools

Use protoc to inspect binary data.

# Print decoded message from binary file
protoc --decode=example.v1.User user.bin

# Convert binary to JSON
protoc --decode=example.v1.User --json_output user.bin

Use --include_imports to show all dependencies.

protoc --include_imports --decode=example.v1.User user.bin

Runtime Debugging in Code

In Go:

// Check if a field is set
if user.GetId() != 0 {
  // id is set
}

// Check if a field is explicitly set (not just default)
if user.HasEmail() {
  // email was sent, even if empty
}

In Java:

if (user.hasEmail()) {
  System.out.println("Email: " + user.getEmail());
}

Field Number Confusion: 1 vs. 100

A common failure: using field number 100 for a new field but forgetting to add optional or default.

message User {
  int64 id = 1;
  string name = 2;
  string email = 100;  // new field, but no default
}

A v1 client sends User with id = 1, name = "Alice". The server expects email but gets email = "". Because email is string, default is "", so email is present but empty.

But the client expects email to be non-empty, so the server returns User with email = "", and the client fails silently.

Fix: add default = "" and optional = true.

Advanced Topics

Oneof and Nested Messages

Use oneof to represent a value that is exactly one of several types.

message Event {
  oneof payload {
    User user = 1;
    Post post = 2;
    Comment comment = 3;
  }
}

Only one of user, post, or comment may be present. The wire format includes a discriminant (tag) for the selected field.

Custom Types and Extensions

Define custom types using extensions.

extend User {
  string nickname = 1000;
  repeated string tags = 1001;
}

// Usage
User user = User.newBuilder()
  .setId(1)
  .setName("Alice")
  .setNickname("alice123")
  .addTags("backend")
  .build();

Extensions allow third-party services to extend a message without recompiling.

Schema Evolution

When evolving a schema:

Add new fields with unused numbers.
Mark new fields as optional or required.
Use default values for backward compatibility.
Test with old clients and new servers.

Use protoc with --include_imports and --experimental_allow_unknown_fields to catch issues early.

Tip: Always use packed = true on repeated fields. Always add default values. Use required for essential fields. Use oneof for variant types. Use protoc to inspect and validate binary data.

This page was rewritten on 10 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.