Sceneis an opaque, immutable representation of composited UI. Each frame, the rendering pipeline produces a scene that is uploaded to the engine for rasterization (via
Window.render). A scene can also be rasterized directly into an
Scene.toImage). Generally, a
Sceneencapsulates a stack of rendering operations (e.g., clips, transforms, effects) and graphics (e.g., pictures, textures, platform views).
SceneBuilderis lowest level mechanism (within the framework) for assembling a scene.
SceneBuildermanages a stack of rendering operations (via
SceneBuilder.pop, etc) and associated graphics (via
SceneBuilder.addRetained, etc). As operations are applied,
EngineLayerinstances that it both tracks and returns. These may be cached by the framework to optimize subsequent frames (e.g., updating an old layer rather than recreating it, or reusing a previously rasterized layer). Once complete, the final
Sceneis obtained via
- Drawing operations are accumulated in a
PictureRecorder) which is added to the scene via
- External textures (represented by a texture ID established using the platform texture repository) are added to the scene via
SceneBuilder.addTexture. This supports a “freeze” parameter so that textures may be paused when animation is paused.
- Previously rasterized
EngineLayerscan be reused as an efficiency operation via
SceneBuilder.addRetained. This is known as retained rendering.
Pictureis an opaque, immutable representation of a sequence of drawing operations. These operations can be added directly to a
SceneBuilder.addPicture) or combined into another picture (via
Canvas.drawPicture). Pictures can also be rasterized (via
Picture.toImage) and provide an estimate of their memory footprint.
PictureRecorderis an opaque mechanism by which drawing operations are sequenced and transformed into an immutable
Picture. Picture recorders are generally managed via a
Canvas. Once recording is completed (via
PictureRecorder.endRecording), a final
Pictureis returned and the
Canvas) are invalidated.
Layerrepresents one slice of a composited scene. Layers are arranged into a hierarchy with each node potentially affecting the nodes below it (i.e., by applying rendering operations or drawing graphics). Layers are mutable and freely moved around the tree, but must only be used once. Each layer is composited into a
Layer.addToScene); adding the root layer is equivalent to compositing the entire tree (e.g., layers add their children). Note that the entire layer tree must be recomposited when mutated; there is no concept of dirty states.
EngineLayerinstance when a layer is added. This handle can be stored in
Layer.engineLayerand later used to optimize compositing. For instance, rather than repeating an operation,
Layer.addToScenemight pass the engine layer instead (via
Layer._addToSceneWithRetainedRendering), potentially reusing the rasterized texture from the previous frame. Additionally, an engine layer (i.e., “
oldLayer”) can be specified when performing certain rendering operations to allow these to be implemented as inline mutations.
- Layers provide support for retained rendering (via
Layer.markNeedsAddToScene).This flag tracks whether the previous engine layer can be reused during compositing (i.e., whether the layer is unchanged from the last frame). If so, retained rendering allows the entire subtree to be replaced with a previously rasterized bitmap (via
SceneBuilder.addRetained). Otherwise, the layer must be recomposited from scratch. Once a layer has been added, this flag is cleared (unless the layer specifies that it must always be composited).
- All layers must be added to the scene initially. Once added, the cached engine layer may be used for retained rendering. Other layers disable retained rendering altogether (via
Layer.alwaysNeedsAddToScene). Generally, when a layer is mutated (i.e., by modifying a property or adding a child), it must be re-added to the scene.
- If a layer must be added to the scene, all ancestor layers must be added, too. Retained rendering allows a subtree to be replaced with a cached bitmap. If a descendent has changed, this bitmap becomes invalid and therefore the ancestor layer must also be added to the scene (via
- Note that this property isn’t enforced during painting; the layer tree is only made consistent when the scene is composited (via
- A layer’s parent must be re-added when it receives a new engine layer [?]
- Generally, only container layers use retained rendering?
- Metadata can be embedded within a layer for later retrieval (via
ContainerLayermanages a list of child layers, inserting them into the composited scene in order. The root layer is a subclass of
ContainerLayeris responsible for compositing the full scene (via
ContainerLayer.buildScene). This involves making retained rendering state consistent (via
ContainerLayer.updateSubtreeNeedsAddToScene), adding all the children to the scene (via
ContainerLayer.addChildrenToScene), and marking the container as no longer needing to be added (via
ContainerLayerimplements a number of child operations (e.g.,
ContainerLayer.removeAllChildren), and multiplexes most of its operations across these children (e.g.,
ContainerLayer.find; note that later children are “above” earlier children).
ContainerLayerwill use retained rendering for its children provided that they aren’t subject to an offset (i.e., are positioned at the layer’s origin). This is the entrypoint for most retained rendering in the layer tree.
ContainerLayer.applyTransformtransforms the provided matrix to reflect how a given child will be transformed during compositing. This method assumes all children are positioned at the origin; thus, any layer offsets must be removed and expressed as a transformation (via
SceneBuilder.pushOffset). Otherwise, the resulting matrix will be inaccurate.
ContainerLayersubclass that supports efficient repainting (i.e., repaint boundaries). Render objects defining repaint boundaries correspond to
OffsetLayersin the layer tree. If the render object doesn’t need to be repainted or is simply being translated, its existing offset layer may be reused. This allows the entire subtree to avoid repainting.
OffsetLayerapplies any offset as a top-level transform so that descendant nodes are composited at the origin and therefore eligible for retained rendering.
OffsetLayer.toImagerasterizes the subtree rooted at this layer (via
Scene.toImage). The resulting image, consisting of raw pixel data, is rendered into a bounding rectangle using the specified pixel ratio (ignoring the device’s screen density).
ContainerLayersubclass that applies an arbitrary transform; it also happens to (typically) be the root layer corresponding to the
RenderView. Any layer offset (via
Layer.addToScene) or explicit offset (via
OffsetLayer.offset) is removed and applied as a transformation to ensure that
PhysicalModelLayeris the engine by which physical lighting effects (e.g., shadows) are integrated into the composited scene. This class incorporates an elevation, clip region, background color, and shadow color to cast a shadow behind its children (via
AnnotatedRegionLayerincorporates metadata into the layer tree within a given bounding rectangle. An offset (defaulting to the origin) determines the rectangle’s top left corner and a size (defaulting to the entire layer) represents its dimensions. Hit testing is performed by
AnnotatedRegionLayer.findAll. Results appearing deeper in the tree or later in the child list take precedence.
LeaderLayerallow efficient transform linking (e.g., so that one layer appears to scroll with another layer). These layers communicate via
LayerLink, passing the leader’s transform (including
layerOffset) to the follower. The follower uses this transform to appear where the follower is rendered.
CompositedTransformTargetwidgets create and manage these layers.
- There are a variety of leaf and interior classes making up the layer tree.
- Leaf nodes generally extend
PictureLayerdescribes a single
Pictureto be added to the scene.
TextureLayeris analogous, describing a texture (by ID) and a bounding rectangle;
PlatformViewLayeris nearly identical, applying a view instead of a texture.
- Interior nodes generally extend
ContainerLayeror one of its subclasses.
OffsetLayeris key to implementing efficient repainting.
OpacityLayer, etc., apply the corresponding effect by pushing a scene builder state, adding all children (via
ContainerLayer.addChildrenToScene, potentially utilizing retained rendering), then popping the state.
EngineLayerand its various specializations (e.g.,
OffsetEngineLayer) represent opaque handles to backend layers.
SceneBuilderproduces the appropriate handle for each operation it supports. These may then be used to enable retained rendering and inline updating.
Canvasprovides an interface for performing drawing operations within the context of a single
PictureRecorder. That is, all drawing performed by a
Canvasis constrained to a single
PictureLayer). Rendering and drawing operations spanning multiple layers are supported by
PaintingContext, which manages
Canvaslifecycle based on the compositing requirements of the render tree.
PaintingContext.repaintCompositedChildis a static method that paints a render object into its own layer. Once nodes are marked dirty for painting, they will be processed during the painting phase of the rendering pipeline (via
PipelineOwner.flushPaint). This performs a depth-first traversal of all dirty render objects. Note that only repaint boundaries are ever actually marked dirty; all other nodes traverse their ancestor chain to find the nearest enclosing repaint boundary, which is then marked dirty. The dirty node is processed (via
PaintingContext._repaintCompositedChild), retrieving or creating an
OffsetLayerto serve as the subtree’s container layer. Next, a new
PaintingContextis created (associated with this layer) to facilitate the actual painting (via
- Layers can be attached and detached from the rendering pipeline. If a render object with a detached layer is found to be dirty (by
PipelineOwner.flushPaint), the ancestor chain is traversed to find the nearest repaint boundary with an attached (or as yet uncreated) layer. This node is marked dirty to ensure that, when the layer is reattached, it will be repainted as expected (via
PaintingContextprovides a layer of indirection between the paint method and the underlying canvas to allow render objects to adapt to changing compositing requirements. That is, a render object may introduce a new layer in some cases and re-use an existing layer in others. When this changes, any ancestor render objects will need to adapt to the possibility of a descendent introducing a new layer (via
RenderObject.needsCompositing); in particular, this requires certain operations to be implemented by introducing their own layers (e.g., clipping).
PaintingContextmanages this process and provides a number of methods that encapsulate this decision (e.g.,
PaintingContextalso ensures that children introducing repaint boundaries are composited into new layers (via
PictureRecorderprovided to the
Canvas, appending picture layers (i.e.,
PictureLayer) whenever a recording is completed.
PaintingContextis initialized with a
ContainerLayersubclass (i.e., the container layer) to serve as the root of the layer tree produced via that context; painting never takes place within this layer, but is instead captured by new
PictureLayerinstances which are appended as painting proceeds.
PaintingContextmay manage multiple canvases due to compositing. Each
Canvasis associated with a
PictureLayer(i.e., the current layer) for the duration of its lifespan. Until a new layer must be added (e.g., to implement a compositing effect or because a repaint boundary is encountered), the same canvas (and
PictureRecorder) is used for all render objects.
- Recording begins the first time the canvas is accessed (via
PaintingContext.canvas). This allocates a new picture layer, picture recorder, and canvas instance; the picture layer is appended to the container layer once created.
- Recording ends when a new layer is introduced directly or indirectly; at this point, the picture is stored in the current layer (via
PictureRecorder.endRecording) and the canvas and current layer are cleared. The next time a child is painted, a new canvas, picture recorder, and picture layer will be created.
- Composited children are painted using a new
PaintingContextinitialized with a corresponding
OffsetLayer. If a composited child doesn’t actually need to be repainted (e.g., because it is only being translated), the
OffsetLayeris reused with a new offset.
PaintingContextprovides a compositing naive API. If compositing is necessary, new layers are automatically pushed to achieve the desired effect (via
PaintingContext.pushLayer). Otherwise, the effect is implemented directly using the
- Pushing a layer ends the current recording, appending the new layer to the container layer. A
PaintingContextis created for the new layer (if present, the layer’s existing children are removed since they’re likely outdated). If provided, a painting function is applied using the new context; any painting is contained within the new layer. Subsequent operations with the original
PaintingContextwill result in a new canvas, layer, and picture recorder being created.
PaintingContext’s base class. Its primary utility is in providing support for clipping without introducing a new layer (e.g., methods suitable for render objects that do not need compositing).
- Compositing describes the process by which layers are combined by the engine during rasterization. Within the context of the framework, compositing typically refers to the allocation of render objects to layers with respect to painting. Needing compositing does not imply that a render object will be allocated its own layer; instead, it indicates that certain effects must be implemented by introducing a new layer rather than modifying the current one (e.g., applying opacity or clipping). For instance, if a parent first establishes a clip and then paints its child, a decision must be made as to how that clip is implemented: (1) as part of the current layer (i.e.,
Canvas.clipPath), or (2) by pushing a
ClipPathLayer). The “needs compositing” bit is how this information is managed.
- Conceptually, this is because this node might eventually paint a descendent that pushes a new layer (e.g., because it is a repaint boundary). Thus, any painting that might have non-local consequences must be implemented in a way that would work across layers (i.e., via compositing).
- Note that this is different than from marking a render object as being a repaint boundary. Doing this causes a new layer to be introduced when painting the child (via
PaintingContext.paintChild). This invalidates the needs compositing bits for all the ancestors of the repaint boundary. This might cause some ancestors to introduce new layers when painting, but only if they utilize any non-local operations
RenderObject.markNeedsCompositingBitsUpdatemarks a render object as requiring a compositing bit update (via
_nodesNeedingCompositingBitsUpdate). If a node is marked dirty, all of its ancestors are marked dirty, too. As an optimization, this walk may be cut off if the current node or the current node’s parent is a repaint boundary (or the parent is already marked dirty).
- If a node’s compositing bits need updating, it’s possible that it will now introduce a new layer (i.e., it will need compositing). If so, all ancestors will need compositing, too, since they may paint a descendent that introduces a new layer. As described, certain non-local effects will need to be implemented via compositing.
- The walk may be cut off at repaint boundaries since all ancestors must already have been marked as needing compositing.
- Adding or removing children (via
RenderObject.dropChild) might alter the compositing requirements of the render tree. Similarly, changing the
RenderObject.alwaysNeedsCompositingbits would require that the bits be updated.
- During the rendering pipeline,
PipelineOwner.flushCompositingBitsupdates all dirty compositing bits (via
RenderObject._updateCompositingBits). A given node needs compositing if (1) it’s a repaint boundary (
RenderObject.isRepaintBoundary), (2) it’s marked as always needing compositing (
RenderObject.alwaysNeedsCompositing), or (3) any of its descendants need compositing.
- This process walks all descendants that have been marked dirty, updating their bits according to the above policy. Note that this will typically only walk the path identified when the compositing bits were marked dirty.
- If a node’s compositing bit is changed, it will be marked dirty for painting (as painting may now require additional compositing).
- If a render object always introduces a layer, it should toggle
alwaysNeedsCompositing. If this changes (other than when altering children, which calls this automatically),
markNeedsCompositingBitsUpdateshould be called.
- Certain render objects introduce repaint boundaries within the render tree (via
RenderObject.isRepaintBoundary). These render objects are always painted into a new layer, allowing them to paint separately from their parent. This effectively decouples the subtree rooted at the repaint boundary from previously painted nodes.
- This is useful when part of the UI remains mostly static and part of the UI updates frequently. A repaint boundary helps to avoid repainting the static portion of the UI.
- Render objects that are repaint boundaries are associated with an
OffsetLayer, a special type of layer that can update its position without re-rendering.
- Render objects that form repaint boundaries are handled differently during painting (via
- If the child doesn’t need painting, the entire subtree is skipped. The corresponding layer is updated to reflect the new offset (via
PaintingContext._compositeChild) and appended to the current parent layer.
- If the child needs painting, the current offset layer is cleared or created (via
PaintingContext._repaintCompositedChild) then used for painting (via
- Otherwise, painting proceeds through
- Since repaint boundaries always push new layers, all ancestors must be marked as needing compositing. If any such node utilizes a non-local painting effect, that node will need to introduce a new layer, too.
PaintingContext.paintChildmanages this process by (1) ending any ongoing recording, (2) creating a new layer for the child, (3) painting the child in the new layer using a new painting context (unless it is a clean repaint boundary, which will instead be used as is), (4) appending that layer to the current layer tree, and (5) creating a new canvas and layer for any future painting with the original painting context.
- Methods in
PaintingContextwill consulting the compositing bits to determine whether to implement a behavior by inserting a layer or altering the canvas (e.g., clipping).
- All render objects may be associated with a
RenderObject.layer). If defined, this is the last layer that was used to paint the render object (if multiple layers are used, this is the root-most layer).
- Repaint boundaries are automatically assigned an
PaintingContext._repaintCompositedChild). All other layers must specifically set this field when painting.
- Render objects that do not directly push layers must set this to null.
ContainerLayertracks any associated
ContainerLayer.engineLayer). Certain operations can use this information to allow a layer to be updated rather than rebuilt.
- Methods accept an “
oldLayer” parameter to enable this optimization.
- Stored engine layers can be reused directly (via
SceneBuilder.addRetained); this is retained rendering.
- A layer can be rasterized to an image via
OffsetLayer.toImage. A convenient way to obtain an image of the entire UI is to use this method on the
RenderView’s own layer. To obtain a subset of the UI, an appropriate repaint boundary can be inserted and its layer used similarly.
- There are two code paths. Layers that do not require compositing handle transforms and clips within their own rendering context (thus, the effects are limited to that single layer), via
Canvas. Those that are composited, however, introduce these effects as special
- A variety of useful
AnnotatedRegionLayer, which is useful for storing metadata within sections of a layer,
BackdropFilterLayer, which allows the background to be blurred or filtered,
OpacityLayer, which allows opacity to be varied, and
OffsetLayer, which is the key mechanism by which the repaint boundary optimization works.
Canvas.restoreallow drawing commands to be grouped such that effects like blending, color filtering, and anti-aliasing can be applied once for the group instead of being incrementally stacked (which could lead to invalid renderings).
- Textures are entirely managed by the engine and referenced by a simple texture ID. Layers depicting textures are associated with this ID and, when composited, integrated with the backend texture. As such, these are painted out-of-band by the engine.
- Texture layers are integrated with the render tree via
TextureBox. This is a render box that tracks a texture ID and paints a corresponding
TextureLayer. During layout, this box expands to fill the incoming constraints.