diff --git a/A110-child-channel-plugins.md b/A110-child-channel-plugins.md new file mode 100644 index 000000000..6f2102bbc --- /dev/null +++ b/A110-child-channel-plugins.md @@ -0,0 +1,437 @@ +A110: Child Channel Options +---- + +* Author(s): [Abhishek Agrawal](mailto:agrawalabhi@google.com) +* Approver: @markdroth, @ejona86, @dfawley +* Status: In Review +* Implemented in: +* Last updated: 2026-03-18 +* Discussion at: https://groups.google.com/g/grpc-io/c/EBIp3uud-Bo + +## Abstract + +There are several use cases where gRPC internally creates a "child channel". +Because these channels are created internally rather than being created by the +application, it is currently difficult to inject necessary configuration for +these channels, which makes it hard to configure things like metrics or tracing +from the application. This design proposes a mechanism to allow applications to +pass configuration options to these child channels. + +## Background + +Complex gRPC ecosystems often require the creation of auxiliary channels that +are not directly instantiated by the user application. The primary examples are: + +1. xDS: When a user creates a channel with an xDS target, the gRPC library + internally creates a separate channel to communicate with the xDS control + plane. (Note: Child channel options are passed to the `XdsClient` when the + `XdsClient` is created, and that `XdsClient` instance will use those same + child channel options for any child channel it creates over its lifetime.) +2. External Authorization (ext_authz): As described + in [gRFC A92](https://github.com/grpc/proposal/pull/481), the gRPC server or + client may create an internal channel to contact an external authorization + service. +3. External Processing (ext_proc): As described + in [gRFC A93](https://github.com/grpc/proposal/pull/484), filters may create + internal channels to call external processing servers. + +The primary motivation for this feature is the need to configure observability +on a per-child-channel basis. + +* StatsPlugins: Users use these plugins to configure metrics and tracing (as + described in + gRFC [A66](https://github.com/grpc/proposal/blob/master/A66-otel-stats.md) + and [A72](https://github.com/grpc/proposal/blob/master/A72-open-telemetry-tracing.md)) + so that telemetry from internal channels is correctly tagged and exported. +* Interceptors: Users may need to apply specific interceptors (e.g., for + logging, or tracing) to internal traffic. + +Global configuration is not sufficient for these use cases for the reasons +described in the 'Rationale' section below. + +### Related Proposals + +* [A27: xDS-Based Global Load Balancing](https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md) +* [A66: Otel Stats](https://github.com/grpc/proposal/blob/master/A66-otel-stats.md) +* [A72: OpenTelemetry Tracing](https://github.com/grpc/proposal/blob/master/A72-open-telemetry-tracing.md) +* [A92: xDS ExtAuthz Support](https://github.com/grpc/proposal/pull/481) +* [A93: xDS ExtProc Support](https://github.com/grpc/proposal/pull/484) + +## Proposal + +We introduce the concept of **Child Channel Options**. This is a configuration +container attached to a parent channel/server that is strictly designated for +use by its children. + +The user API must allow "nesting" of channel options (specifying child channel +options within parent channel options). A user creating a Parent +Channel/Server `P` can provide a set of options `O_child`. + +* `O_child` is opaque to `P`. `P` does not apply these options to itself. +* `O_child` is carried in `P`'s state, available for extraction by internal + components. +* The configuration provided by `O_child` is strictly uniform across all child + channels of a particular parent channel/server. + +When an internal component (e.g., an xDS client factory) attached to `P` +needs to create a Child Channel `C`: + +1. It retrieves `O_child` from `P`. +2. It applies `O_child` to the configuration of `C`. +3. It should also configure channel `C` to use `O_child` for its children. + +* **Multi-level Propagation**: The child options `O_child` apply recursively to + all child channels, no matter how deeply nested. For example, if a parent + channel `P` creates a child channel `C` (e.g., to an `ext_authz` server), and + `C` itself is configured to use `xds:///` (which requires creating an xDS + client control plane channel), `C` will pass `O_child` to its own child + channels. + +The Child Channel `C` typically requires some internal +configuration `O_internal` (e.g., target URIs, or internal interceptors). + +* Merge Rule: `O_child` and `O_internal` are merged. If the environment supports + global channel options, `O_child` options override global channel options. +* Conflict Resolution: Mandatory internal settings (`O_internal`) generally take + precedence over user-provided child options (`O_child`) to ensure correctness. + +In some cases, child channels may be shared across multiple parent +channels/servers. For example, the xDS control plane channel is shared across +multiple channels or servers as described +in [gRFC A27](https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md). +However, it is possible for each parent channel or server to be created with +different child options. + +Consider an example where Parent Channels/Servers (`P1`, `P2`) point to the +same target but provide *different* Child Channel +Options (`O_child1`, `O_child2`): + +* Behavior: The shared client is created using the options from the first parent + channel or server that triggers its creation (e.g., `O_child1`). +* Subsequent Usage: When `P2` requests the client, it receives the existing + shared client. `O_child2` is effectively ignored for that specific shared + resource. + +**LB Policies and Resolvers** + +Some LB policies and resolvers may need to create child channels. We use +`grpclb` as an example for how this plumbing will be handled in LB policies. +Note that this proposal does not mandate any behavior changes for `grpclb` +specifically. + +To support this, the child channel options must be plumbed down into +resolvers and LB policies. Each internal component that creates a child +channel `C` is explicitly responsible for applying and propagating those +options: + +* Java: When an LB policy creates an out-of-band (OOB) child channel via + `LoadBalancer.Helper` (e.g., `createResolvingOobChannelBuilder()` or + `createOobChannel()`), the `Helper` implementation (`ManagedChannelImpl`) + is responsible for automatically calling + `channelConfigurator.configureChannelBuilder(builder)` to apply the + options to `C` and `builder.childChannelConfigurator(channelConfigurator)` + to propagate `O_child` recursively to any further child channels. For + resolvers and other internal components (e.g., `XdsClient` via + `GrpcXdsTransportFactory`) that independently create a child channel `C`, + `channelConfigurator` is plumbed via + `NameResolver.Args.getChildChannelConfigurator()`. That internal + component is responsible for explicitly calling both + `channelConfigurator.configureChannelBuilder(channelBuilder)` and + `channelBuilder.childChannelConfigurator(channelConfigurator)` when + constructing `C`. +* Go: A new field (`ChildChannelOptions`) will be added to + `resolver.BuildOptions` and `balancer.BuildOptions` (passed when + creating a resolver or LB policy) to contain the child channel options + (`[]grpc.DialOption`). When a resolver (e.g., the xDS resolver creating + a control plane `XdsClient`) or an LB policy (e.g., `grpclb` creating an + out-of-band `ClientConn`) creates a child channel `C` (e.g., via + `grpc.NewClient`), that resolver or LB policy is responsible for applying + `ChildChannelOptions` to `C` and calling + `grpc.WithChildChannelOptions(ChildChannelOptions...)` so that any further + child channels created by `C` also inherit the options. +* C-core: No special plumbing is needed to pass child channel options to LB + policies and resolvers because they are already contained within the + channel arguments (`grpc_channel_args`). When an LB policy or resolver + (e.g., `grpclb` or an xDS resolver) creates a child channel `C`, that LB + policy or resolver is responsible for propagating both the individual + child channel args (`O_child` applied to `C`) and the + `GRPC_ARG_CHILD_CHANNEL_ARGS` argument containing the child channel args + (`O_child` propagated recursively) to `C`. + +### Language Implementations + +#### Java + +In Java, the configuration will be achieved by accepting functional +interfaces. The API allows users to register a configurator on a +`ManagedChannelBuilder` or `ServerBuilder`. When an internal library +or component creates a child channel `C`: +1. If created via `LoadBalancer.Helper` (`createResolvingOobChannelBuilder()` + or `createOobChannel()`), `ManagedChannelImpl` automatically calls + `channelConfigurator.configureChannelBuilder(builder)` and + `builder.childChannelConfigurator(channelConfigurator)` on `C`'s builder. +2. If created independently by a resolver or internal transport factory + (e.g., `GrpcXdsTransportFactory`), that component retrieves + `channelConfigurator` via + `NameResolver.Args.getChildChannelConfigurator()` and explicitly calls + `channelConfigurator.configureChannelBuilder(builder)` and + `builder.childChannelConfigurator(channelConfigurator)` on `C`'s builder. + +* ##### Configuration Interface + + Define a new public API interface, `ChannelConfigurator`, to encapsulate the + configuration logic for channels. + + ```java + + import io.grpc.ManagedChannelBuilder; + + // Captures the intent of the plugin. + // Consumes a builder to modify it before further configuring the channel + public interface ChannelConfigurator { + /** + * Configures the given channel builder. + * + * @param builder the channel builder to configure + */ + void configureChannelBuilder(ManagedChannelBuilder builder); + } + ``` + +* ##### API Changes + + * ManagedChannelBuilder: + Add `ManagedChannelBuilder#childChannelConfigurator(ChannelConfigurator` + `channelConfigurator)` to allow users to register this configurator. + * XdsServerBuilder: + Add `XdsServerBuilder#childChannelConfigurator(ChannelConfigurator` + `configurator)` to allow users to provide configuration for any + internal channels created by the server (e.g., connections to external + authorization or processing services). + * NameResolver.Args: + Add `NameResolver.Args#getChildChannelConfigurator()` and + `NameResolver.Args.Builder#setChildChannelConfigurator(` + `ChannelConfigurator channelConfigurator)` to allow resolvers to access + the child channel configurator when creating internal child channels. + +* ##### Usage Example + + ```java + // Define the configurator for internal child channels + ChannelConfigurator myInternalConfig = new ChannelConfigurator() { + @Override + public void configureChannelBuilder(ManagedChannelBuilder builder) { + builder.maxInboundMessageSize(4 * 1024 * 1024); + } + }; + + // Apply it to the parent channel + ManagedChannel channel = ManagedChannelBuilder.forTarget("xds:///my-service") + .childChannelConfigurator(myInternalConfig) // <--- Configuration injected here + .build(); + ``` + +#### Go + +In Go, both the Client (`grpc.NewClient`) and the Server (`NewGRPCServer`) +create internal child channels. We introduce mechanisms to pass `DialOption`s +into these internal channels from both entry points. + +* ##### New API for Child Channel Options + +* Client-Side: `WithChildChannelOptions` + + For standard clients, we introduce a `DialOption` wrapper. + + ```go + // WithChildChannelOptions returns a DialOption that specifies a list of + // DialOptions to be applied to any internal child channels. + func WithChildChannelOptions(opts ...DialOption) DialOption { + return newFuncDialOption(func(o *dialOptions) { + o.childChannelOptions = opts + }) + } + ``` + +* Server-Side: `ChildChannelOptions` + + For xDS-enabled servers, we introduce a `ServerOption` wrapper. + Since `xds.NewGRPCServer` creates an internal xDS client to fetch listener + configurations, it requires a way to apply `DialOptions` (such as **Socket + Options** or **Stats Handlers**) to that internal connection. + + ```go + // ChildChannelOptions returns a ServerOption that specifies a list of + // DialOptions to be applied to the server's internal child channels + // (e.g., the xDS control plane connection). + func ChildChannelOptions(opts ...DialOption) ServerOption { + return newFuncServerOption(func(o *serverOptions) { + o.childDialOptions = opts + }) + } + ``` + +* Resolvers and LB Policies: `BuildOptions` + + To allow resolvers and load balancers to access child channel options when + creating child channels, `ChildChannelOptions` will be added to both + `resolver.BuildOptions` and `balancer.BuildOptions`. + + ```go + type BuildOptions struct { + // ... existing fields ... + + // ChildChannelOptions contains the options to be applied to any + // internal child channels created by the resolver or load balancer. + ChildChannelOptions []DialOption + } + ``` + +* ##### Usage Example (User-Side Code) + + This design provides users with the flexibility to define independent + configurations for parent and child channels within a single NewClient call. + For example, a parent channel can be configured with transport security (mTLS) + while the internal child channels (such as the xDS control plane connection) + are configured with specific interceptors or a custom authority. + + ```go + func main() { + // Define configuration specifically for the internal control plane + internalOpts := []grpc.DialOption{ + // Inject the OTel handler here. It will only measure traffic on the + // internal child channels (e.g., to the xDS server). + grpc.WithStatsHandler(otelHandler) + } + + // Create the Parent Channel + conn, err := grpc.NewClient("xds:///my-service", + // Parent channel configuration (Data Plane) + grpc.WithTransportCredentials(insecure.NewCredentials()), + + // Child channel configuration (Control Plane) + // The OTel handler inside here applies ONLY to the child channels. + grpc.WithChildChannelOptions(internalOpts...), + ) + + if err != nil { + log.Fatalf("failed to create client: %v", err) + } + defer conn.Close() + + // ... use conn ... + } + ``` + +#### Core (C/C++) + +In gRPC Core, we utilize the existing `ChannelArgs` mechanism recursively to +pass configuration to internal channels. We define a standard argument key whose +value is a pointer to another `grpc_channel_args` structure. This "Nested +Arguments" pattern allows the parent channel or server to carry a specific +subset of arguments intended solely for its children. + +* ##### Configuration Mechanism + + We define a new channel argument key. The value associated with this key is a + pointer to a `grpc_channel_args` struct, managed via a pointer vtable to + ensure correct ownership and copying. + + ```c + // A pointer argument key. The value is a pointer to a grpc_channel_args + // struct containing the subset of options for child channels. + #define GRPC_ARG_CHILD_CHANNEL_ARGS "grpc.child_channel.args" + ``` + +* ##### API Changes + + We add a helper method to the C++ `ChannelArguments` and `ServerBuilder` + classes to simplify packing the nested arguments safely. + + ```cpp + // Sets the channel arguments to be used for child channels. + void SetChildChannelArgs(const ChannelArguments& args); + ``` + +* ##### Usage Example (User-Side Code) + + An example of how this will work on the channel side: + + ```cpp + // Create OTel StatsPlugin. + grpc::OpenTelemetryPluginBuilder ot_plugin_builder; + // ...set options on builder... + auto ot_plugin = ot_plugin_builder.Build(); + assert(ot_plugin.ok()); + // Add the StatsPlugin to both child args and parent args. + grpc::ChannelArguments child_args; + grpc::ChannelArguments parent_args; + ot_plugin->AddToChannelArguments(&child_args); + ot_plugin->AddToChannelArguments(&parent_args); + // Add child args to parent args. + parent_args.SetChildChannelArgs(child_args); + // Create channel with parent args. + auto channel = grpc::CreateCustomChannel( + "xds:///my-service", credentials, parent_args); + ``` + + An example of how this will work on the server side: + + ```cpp + grpc::ServerBuilder server_builder; + // Create OTel StatsPlugin. + grpc::OpenTelemetryPluginBuilder ot_plugin_builder; + // ...set options on builder... + auto ot_plugin = ot_plugin_builder.Build(); + assert(ot_plugin.ok()); + // Add the StatsPlugin to both child args and the server builder. + grpc::ChannelArguments child_args; + ot_plugin->AddToChannelArguments(&child_args); + ot_plugin->AddToServerBuilder(&server_builder); + // Add the child args to the server builder. + server_builder.SetChildChannelArgs(child_args); + // Start the server. + auto server = server_builder.BuildAndStart(); + ``` + +## Rationale + +The proposed mechanism of Child Channel Options provides a targeted way to +propagate configuration from a parent channel to its children. This approach is +chosen because it allows configuration to be scoped to specific parent-child +hierarchies, which is necessary for accurate telemetry and interceptor +application without affecting unrelated channels. + +### Why not Global Configuration? + +The primary use-case we care about is setting a `StatsPlugin` for one particular +channel, in which case we want that same `StatsPlugin` to also be used for any +child of that channel. + +For example, let's say that we create two channels, one to target A and one to +target B, both of which create their own child channel to an `ext_authz` server +target Z. If we create the channel to target A with a specific `StatsPlugin`, +then we want that `StatsPlugin` to also be used for the child channel to target +Z created by the channel to target A. We do *not* want it to be used for the +child channel to target Z created by the channel to target B, because we did not +configure the `StatsPlugin` for the channel to target B. + +We cannot achieve this using the global registry, for a couple of reasons. +First, the global registry can only select channels based on parameters like the +target URI. To attach our `StatsPlugin` to the internal target Z, we would have +to select it based on target Z, which would erroneously attach it to the child +channels for both A and B. Second, it is hard for the application that registers +the global `StatsPlugin` to know what target URIs will be used for internal +child channels. + +## Non-Goals and Future Work + +### Differentiating Configuration Per Child Channel + +This proposal mandates that the child channel options provided by a parent are +uniform across all its child channels. We considered allowing different +configurations for different child channels (e.g., based on the purpose of the +channel like xDS vs RLS). However, this would require a mechanism to categorize +or identify the purpose of each child channel, which adds significant +complexity. Since no strong need for this was identified during initial +discussions, it is left for future work if and when it becomes necessary.