Execution protocol

The executor is an additional layer of abstraction that provides portability and versioning for the execution of build steps. Additionally, because Nyb talks to the executor through an RPC protocol (see below), there is only one executor process per Nyb invocation. This peculiarity can be used to implement the equivalent of Bazel workers and speed up compilation.

The executor is an external program, developed on a per-project basis, called once by Nyb at the beginning of nyb run. Communication goes through an STDIN/STDOUT duplex. Requests are issued on the executor’s STDIN file as a stream of newline-delineated JSON objects. Responses are parsed from the executor’s STDOUT file, also a stream of newline-delineated JSON objects. A request can have many responses (this is the equivalent of a gRPC server stream). Additionally, in verbose mode the executor’s STDERR file is read and piped to Nyb’s output. Multiplexing over this duplex is implemented with the help of a request ID, passed along the associated responses.

Nyb communicates with the executor through an STDIN/STDOUT duplex.

The format for the requests and responses are given below, using the TypeScript type system:

/**
 * DuplexPayload wraps both requests and responses to provide a requestID.
 * Request IDs are used for multiplexing.
 */
type DuplexPayload = {
  /** id is the request ID. */
  id: string;

  /** payload is either a Request object, or a Response object. */
  payload: Request | Response;
};

/**
 * Request is either a request for the versions of a list of tools
 * or a request for execution.
 */
type Request = VersionsRequest | ExecutionRequest;

/**
 * Response is either a response for VersionsRequest or for
 * ExecutionRequest.  In the case of VersionsRequest, the response is
 * unary, meaning that there is only one response for a VersionsRequest.
 * In the case of an ExecutionRequest, the executor sends a stream
 * of ExecutionResponse objects.  The stream ends when the "status"
 * field of ExecutionResponse has a value other than "pending", in which
 * case the ExecutionResponse is the last object in the stream and gives
 * the overall status of the execution.
 */
type Response = VersionsResponse | ExecutionResponse;

/**
 * VersionsRequest is a request for the versions of a list of tools.
 * The versions are arbitrary strings.  If the version of a tool changes,
 * all the action steps that depend on this tool are invalidated (the
 * versions are embedded in the action digests used for caching).
 */
type VersionsRequest = {
  method: 'versions';
  params: {
    /**
     * tools is a list of tools to retrieve the version of.  As versions
     * can depend on system calls, giving a list of tools allows the
     * executor to make these system calls lazily, only when an execution
     * actually takes place.
     */
    tools: string[];
  };
};

/**
 * VersionsResponse is the response for VersionsRequest.
 */
type VersionsResponse = {
  /**
   * versions is a list of versions.  The order is the same as the
   * one used for the "tools" field in the VersionsRequest.
   */
  versions: []string;
};

/**
 * ExecutionRequest is a request for executing an action step.
 * As a rule can be comprised of multiple action steps, executing a rule
 * can mean sending multiple ExecutionRequest payloads and waiting for
 * the executions to complete, sequentially.
 */
type ExecutionRequest = {
  method: 'execute';
  params: {
    /**
     * payload is the step to execute.
     */
    payload: Step;
  };
};

/**
 * Step is an action step.  For the full shape, see the shape of Nyb's
 * dependency graph.
 */
type Step = any;

/*
 * ExecutionResponse is one item in the stream that is sent by the executor
 * following an ExecutionRequest.
 */
type ExecutionResponse = {
  /**
   * status is the status of the execution.  If status is other than
   * "pending", the payload is the last payload in the server stream, and
   * gives the overall result for the action step.
   *
   * "pending" is the default status.
   */
  status?: Status;

  /**
   * output is an optional text output.  Typically, this would be
   * a line of the STDERR/STDOUT of an external compiler that is invoked
   * by the executor.  The output could be buffered in any way deemed
   * the most appropriate by the executor.  One common approach is
   * to send one line of output by ExecutionResponse item.
   */
  output?: string;
};

/**
 * Status is the overall status of an execution.
 */
const enum Status {
  /**
   * pending is issued with an ExecutionResponse when the execution is
   * still in progress.
   */
  pending = 0,

  /**
   * success is issued when the execution completed successfully.
   */
  success = 1,

  /**
   * failure is issued when the execution completed to failure.
   */
  failure = 2,

  /**
   * cancelled is issued when the execution was cancelled midway and
   * did not complete.
   */
  cancelled = 3,
}

Docs

Execution protocol