Rendering

The kOfxImageEffectActionRender action is passed to plugins when the host requires them to render an output frame.

All calls to the kOfxImageEffectActionRender are bracketed by a pair of kOfxImageEffectActionBeginSequenceRender and kOfxImageEffectActionEndSequenceRender actions. This is to allow plugins to prepare themselves for rendering long sequences by setting up any tables etc.. it may need.

The kOfxImageEffectActionBeginSequenceRender will indicate the frame range that is to be rendered, and whether this is purely a single frame render due to interactive feedback from a user in a GUI.

Identity Effects

If an effect does nothing to its input clips (for example a blur with blur size set to ‘0’) it can indicate that it is an identity function via the kOfxImageEffectActionIsIdentity action. The plugin indicates which input the host should use for the region in question. This allows a host to short circuit the processing of an effect.

Rendering and The Get Region Actions

Many hosts attempt to minimise the areas that they render by using regions of interest and regions of definition, while some of the simpler hosts do not attempt to do so. In general the order of actions, per frame rendered, is something along the lines of….

  • ask the effect for it’s region of definition,

  • clip the render window against that

  • ask the effect for the regions of interest of each of it’s inputs against the clipped render window,

  • clip those regions of interest against the region of definition of each of those inputs,

  • render and cache each of those inputs,

  • render the effect against it’s clipped render window.

A host can ask an effect to render an arbitrary window of pixels, generally these should be clipped to an effect’s region of definition, however, depending on the host, they may not be. The actual region to render is indicated by the kOfxImageEffectPropRenderWindow render action argument. If an effect is asked to render outside of its region of definition, it should fill those pixels in with black transparent pixels.

Note thate OfxImageEffectSuiteV1::clipGetImage() function takes an optional region parameter. This is a region, in Canonical coordinates, that the effect would like on that input clip. If not used in a render action, then the image returned should be based on the previous get region of interest action. If used, then the image returned will be based on this (usually be clipped to the input’s region of definition). Generally a plugin should not use the region parameter in the render action, but should leave it to the ‘default’ region.

Multi-threaded Rendering

Multiple render actions may be passed to an effect at the same time. A plug-in states it’s level of render thread safety by setting the kOfxImageEffectPluginRenderThreadSafety string property. This can be set to one of three states….

kOfxImageEffectRenderUnsafe

String used to label render threads as un thread safe, see, kOfxImageEffectPluginRenderThreadSafety.

Indicating that only a single ‘render’ action can be made at any time among all instances

kOfxImageEffectRenderInstanceSafe

String used to label render threads as instance thread safe, kOfxImageEffectPluginRenderThreadSafety.

Indicating that any instance can have a single ‘render’ action at any one time

kOfxImageEffectRenderFullySafe

String used to label render threads as fully thread safe, kOfxImageEffectPluginRenderThreadSafety.

Indicating that any instance of a plugin can have multiple renders running simultaneously

Rendering in a Symmetric Multi Processing Environment

When rendering on computers that have more that once CPU (or this new-fangled hyperthreading), hosts and effects will want to take advantage of all that extra CPU goodness to speed up rendering. This means multi-threading of the render function in some way.

If the plugin has set kOfxImageEffectPluginRenderThreadSafety to kOfxImageEffectRenderFullySafe, the host may choose to render a single frame across multiple CPUs by having each CPU render a different window. However, the plugin may wish to remain in charge of multithreading a single frame. The plugin set property kOfxImageEffectPluginPropHostFrameThreading informs the host as to whether the host should perform SMP on the effect. It can be set to either…

  • 1, in which case the host will attempt to multithread an effect instance by calling it’s render function called simultaneously, each call will be with a different renderWindow, but be at the same frame

  • 0, in which case the host only ever calls the render function once per frame. If the effect wants to multithread it must use the OfxMultiThreadSuite API.

A host may have a render farm of computers. Depending exactly how the host works with it’s render farm, it may have multiple copies on an instance spread over the farm rendering separate frame ranges, 1-100 on station A, 101 to 200 on station B and so on…

Rendering Sequential Effects

Some plugins need the output of the previous frame to render the next, typically they cache some information about the last render and use that somehow on the next frame. Some temporally averaging degraining algorithms work that way. Such effects cannot render correctly unless they are strictly rendered in order, from first to last frame, on a single instance.

Other plugins are able to render correctly when called in an arbitrary frame order, but render much more efficiently if rendered in order. For example a particle system which maintains the state of the particle system in an instance would simply increment the simulation by a frame if rendering in-order, but would need to restart the particle system from scratch if the frame jumped backwards.

Most plug-ins do not have any sequential dependence. For example, a simple gain operation has no dependence on the previous frame.

Similarly, host applications, due to their architectures, may or may not be able to guarantee that a plugin can be rendered strictly in-order. Node based applications typically have much more difficulty in guaranteeing such behaviour.

To indicate whether a plugin needs to be rendered in a strictly sequential order, and to indicate whether a host supports such behaviour we have a property, kOfxImageEffectInstancePropSequentialRender. For plug-ins this can be one of three values…

  • 0, in which case the host can render an instance over arbitrary frame ranges on an arbitrary number of computers without any problem (default),

  • 1, in which case the host must render an instance on a single computer over it’s entire frame range, from first to last.

  • 2, in which case the effect is more efficiently rendered in frame order, but can compute the correct result regardless of render order.

For hosts, this property takes three values…

  • 0, which indicates thet the host can never guarantee sequential rendering,

  • 1, which indicates thet the host can guarantee sequential rendering for plugins that request it,

  • 2, which indicates thet the host can sometimes perform sequential rendering.

When rendering, a host will set the in args property on kOfxImageEffectPropSequentialRenderStatus to indicate whether the host is currently supporting sequential renders. This will be passed to the following actions,

  • the begin sequence render action

  • the sequence render action

  • the end sequence render action

Hosts may still render sequential effects with random frame access in interactive sessions, for example when the user scrubs the current frame on the timeline and the host asks an effect to render a preview frame. In such cases, the plugin can detect that the instance is being interactively manipulated via the kOfxImageEffectPropInteractiveRenderStatus property and hack an approximation together for UI purposes. If eventually rendering the sequence, the host must ignore all frames rendered out of order and not cache them for use in the final result.

A host may set the in args property kOfxImageEffectPropRenderQualityDraft in :c:macro:kOfxImageEffectActionRender` to ask for a render in Draft/Preview mode. This is useful for applications that must support fast scrubbing. These allow a plug-in to take short-cuts for improved performance when the situation allows and it makes sense, for example to generate thumbnails with effects applied. For example switch to a cheaper interpolation type or rendering mode. A plugin should expect frames rendered in this manner that will not be stuck in host cache unless the cache is only used in the same draft situations.

OFX : Fields and Field Rendering

Fields are evil, but until the world decides to adopt sensible video standard and casts the current ones into the same pit as 2 inch video tape, we are stuck with them.

Before we start, some nomenclature. The Y-Axis is considerred to be up, so in a fielded image,

  • even scan lines 0,2,4,6,… are collectively referred to as the lower field,

  • odd scan lines 1,3,5,7… are collective referred to as the upper field.

We don’t call them odd and even, so as to avoid confusion with video standard, which have scanline 0 at the top, and so have the opposite sense of our ‘odd’ and ‘even’.

Clips and images from those clips are flagged as to whether they are fielded or not, and if so what is the spatial/temporal ordering of the fields in that image. The kOfxImageClipPropFieldOrder clip and image instance property can be…

kOfxImageFieldNone

String used to label imagery as having no fields

kOfxImageFieldLower

String used to label the lower field (scan lines 0,2,4…) of fielded imagery

kOfxImageFieldUpper

String used to label the upper field (scan lines 1,3,5…) of fielded imagery

Images extracted from a clip flag what their fieldedness is with the property kOfxImagePropField, this can be….

kOfxImageFieldNone

String used to label imagery as having no fields

kOfxImageFieldBoth

String used to label both fields of fielded imagery, indicating interlaced footage

kOfxImageFieldLower

String used to label the lower field (scan lines 0,2,4…) of fielded imagery

kOfxImageFieldUpper

String used to label the upper field (scan lines 1,3,5…) of fielded imagery

The plugin specifies how it deals with fielded imagery by setting this property:

kOfxImageEffectPluginPropFieldRenderTwiceAlways

Controls how a plugin renders fielded footage.

  • Type - integer X 1

  • Property Set - a plugin descriptor (read/write)

  • Default - 1

  • Valid Values - This must be one of

    • 0 - the plugin is to have its render function called twice, only if there is animation in any of its parameters

    • 1 - the plugin is to have its render function called twice always

The reason for this is an optimisation. Imagine a text generator with no animation being asked to render into a fielded output clip, it can treat an interlaced fielded image as an unfielded frame. So the host can get the effect to render both fields in one hit and save on the overhead required to do the rendering in two passes.

If called twice per frame, the time passed to the render action will be frame and frame+0.5. So 0.0 0.5 1.0 1.5 etc…

When rendering unfielded footage, the host will only ever call the effect’s render action once per frame, with the time being at the integers, 0.0, 1.0, 2.0 and so on.

The render action’s argument property kOfxImageEffectPropFieldToRender tells the effect which field it should render, this can be one of…

Note

kOfxImageEffectPropFieldToRender will be set to kOfxImageFieldBoth if kOfxImageEffectPluginPropFieldRenderTwiceAlways is set to 0 on the plugin

A plugin can specify how it wishes fielded footage to be fetched from a clip via the clip descriptor property kOfxImageClipPropFieldExtraction. This can be one of…

Fetch a full frame interlaced image

Fetch a single field, making a half height image

Fetch a single field, but doubling each line and so making a full

height image (default)

If fetching a single field, the actual field fetched from the source frame is…

  • the first temporal field if the time passed to clipGetImage has a fractional part of 0.0 <= f < 0.5

  • the second temporal field otherwise,

To illustrate this last behaviour, the two examples below show an output with twice the frame rate of the input and how clipGetImage maps to the input. The .0 and .5 mean first and second temporal fields.

Behaviour with unfielded footage

output 0       1       2       3
source 0       0       1       1

Behaviour with fielded footage

output 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
source 0.0 0.0 0.5 0.5 1.0 1.0 1.5 1.5

NOTE

  • while some rarely used video standards can have odd number of scan-lines, under OFX, both fields always consist of the same number of lines. Pad with black where needed.

  • host developers, for single field extracted images, you don’t need to do any buffer copies, you just need to set the row bytes property of the returned image to twice the normal value, and maybe tweak the start address by a scanline.

Rendering In An Interactive Environment

Any host with an interface will most likely have an interactive thread and a rendering thread. This allows an effect to be manipulated while having renders batched off to a background thread. This will mean that some degree of locking will go on to prevent simultaneous read/writes occurring, see this section for more on thread safety.

A host may need to abort a backgrounded render, typically in response to a user changing a parameter value. An effect should occasionally poll the OfxImageEffectSuiteV1::abort() function to see if it should give up on rendering.

Rendering on GPU

page ofxOpenGLRender

Introduction

The OfxOpenGLRenderSuite allows image effects to use OpenGL commands (hopefully backed by a GPU) to accelerate rendering of their outputs. The basic scheme is simple….

OpenGL House Keeping

If a host supports OpenGL rendering then it flags this with the string property kOfxImageEffectPropOpenGLRenderSupported on its descriptor property set. Effects that cannot run without OpenGL support should examine this in kOfxActionDescribe action and return a kOfxStatErrMissingHostFeature status flag if it is not set to “true”.

Effects flag to a host that they support OpenGL rendering by setting the string property kOfxImageEffectPropOpenGLRenderSupported on their effect descriptor during the kOfxActionDescribe action. Effects can work in three ways….

Hosts can examine this flag and respond to it appropriately.

Effects can use OpenGL accelerated rendering during the following action…

If an effect has indicated that it optionally supports OpenGL acceleration, it should check the property kOfxImageEffectPropOpenGLEnabled passed as an in argument to the following actions,

If this property is set to 0, then it should not attempt to use any calls to the OpenGL suite or OpenGL calls whilst rendering.

Getting Images as Textures

An effect could fetch an image into memory from a host via the standard Image Effect suite “clipGetImage” call, then create an OpenGL texture from that. However as several buffer copies and various other bits of house keeping may need to happen to do this, it is more efficient for a host to create the texture directly.

The OfxOpenGLRenderSuiteV1::clipLoadTexture function does this. The arguments and semantics are similar to the OfxImageEffectSuiteV2::clipGetImage function, with a few minor changes.

The effect is passed back a property handle describing the texture. Once the texture is finished with, this should be disposed of via the OfxOpenGLRenderSuiteV1::clipFreeTexture function, which will also delete the associated OpenGL texture (for source clips).

The returned handle has a set of properties on it, analogous to the properties returned on the image handle by OfxImageEffectSuiteV2::clipGetImage. These are:

The main difference between this and an image handle is that the kOfxImagePropData property is replaced by the kOfxImageEffectPropOpenGLTextureIndex property. This integer property should be cast to a GLuint and is the index to use for the OpenGL texture. Next to texture handle the texture target enumerator is given in kOfxImageEffectPropOpenGLTextureTarget

Note, because the image is being directly loaded into a texture by the host it need not obey the Clip Preferences action to remap the image to the pixel depth the effect requested.

Render Output Directly with OpenGL

Effects can use the graphics context as they see fit. They may be doing several render passes with fetch back from the card to main memory via ‘render to texture’ mechanisms interleaved with passes performed on the CPU. The effect must leave output on the graphics card in the provided output image texture buffer.

The host will create a default OpenGL viewport that is the size of the render window passed to the render action. The following code snippet shows how the viewport should be rooted at the bottom left of the output texture.

  // set up the OpenGL context for the render to texture
  ...

  // figure the size of the render window
  int dx = renderWindow.x2 - renderWindow.x1;
  int dy = renderWindow.y2 - renderWindow.y2;

  // setup the output viewport
  glViewport(0, 0, dx, dy);

Prior to calling the render action the host may also choose to bind the output texture as the current color buffer (render target), or they may defer doing this until clipLoadTexture is called for the output clip.

After this, it is completely up to the effect to choose what OpenGL operations to render with, including projections and so on.

OpenGL Current Context

The host is only required to make the OpenGL context current (e.g., using wglMakeCurrent, for Windows) during the following actions:

For the first 3 actions, Render through EndSequenceRender, the host is only required to set the OpenGL context if kOfxImageEffectPropOpenGLEnabled is set. In other words, a plug-in should not expect the OpenGL context to be current for other OFX calls, such as kOfxImageEffectActionDescribeInContext.

group CudaRender

Version

CUDA rendering was added in version 1.5.

Defines

kOfxImageEffectPropCudaRenderSupported

Indicates whether a host or plug-in can support CUDA render.

  • Type - string X 1

  • Property Set - plug-in descriptor (read/write), host descriptor (read only)

  • Default - “false” for a plug-in

  • Valid Values - This must be one of

    • ”false” - the host or plug-in does not support CUDA render

    • ”true” - the host or plug-in can support CUDA render

kOfxImageEffectPropCudaEnabled

Indicates that a plug-in SHOULD use CUDA render in the current action.

If a plug-in and host have both set kOfxImageEffectPropCudaRenderSupported=”true” then the host MAY set this property to indicate that it is passing images as CUDA memory pointers.

kOfxImageEffectPropCudaStreamSupported

Indicates whether a host or plug-in can support CUDA streams.

  • Type - string X 1

  • Property Set - plug-in descriptor (read/write), host descriptor (read only)

  • Default - “false” for a plug-in

  • Valid Values - This must be one of

    • ”false” - in which case the host or plug-in does not support CUDA streams

    • ”true” - which means a host or plug-in can support CUDA streams

kOfxImageEffectPropCudaStream

The CUDA stream to be used for rendering.

This property will only be set if the host and plug-in both support CUDA streams.

If set:

  • this property contains a pointer to the stream of CUDA render (cudaStream_t). In order to use it, reinterpret_cast<cudaStream_t>(pointer) is needed.

  • the plug-in SHOULD ensure that its render action enqueues any asynchronous CUDA operations onto the supplied queue.

  • the plug-in SHOULD NOT wait for final asynchronous operations to complete before returning from the render action, and SHOULD NOT call cudaDeviceSynchronize() at any time.

If not set:

  • the plug-in SHOULD ensure that any asynchronous operations it enqueues have completed before returning from the render action.

group MetalRender

Version

Metal rendering was added in version 1.5.

Defines

kOfxImageEffectPropMetalRenderSupported

Indicates whether a host or plug-in can support Metal render.

  • Type - string X 1

  • Property Set - plug-in descriptor (read/write), host descriptor (read only)

  • Default - “false” for a plug-in

  • Valid Values - This must be one of

    • ”false” - the host or plug-in does not support Metal render

    • ”true” - the host or plug-in can support Metal render

kOfxImageEffectPropMetalEnabled

Indicates that a plug-in SHOULD use Metal render in the current action.

If a plug-in and host have both set kOfxImageEffectPropMetalRenderSupported=”true” then the host MAY set this property to indicate that it is passing images as Metal buffers.

kOfxImageEffectPropMetalCommandQueue

The command queue of Metal render.

This property contains a pointer to the command queue to be used for Metal rendering (id<MTLCommandQueue>). In order to use it, reinterpret_cast<id<MTLCommandQueue>>(pointer) is needed.

The plug-in SHOULD ensure that its render action enqueues any asynchronous Metal operations onto the supplied queue.

The plug-in SHOULD NOT wait for final asynchronous operations to complete before returning from the render action.

group OpenClRender

Version

OpenCL rendering was added in version 1.5.

Defines

kOfxImageEffectPropOpenCLRenderSupported

Indicates whether a host or plug-in can support OpenCL Buffers render.

  • Type - string X 1

  • Property Set - plug-in descriptor (read/write), host descriptor (read only)

  • Default - “false” for a plug-in

  • Valid Values - This must be one of

    • ”false” - the host or plug-in does not support OpenCL Buffers render

    • ”true” - the host or plug-in can support OpenCL Buffers render

kOfxImageEffectPropOpenCLSupported

Indicates whether a host or plug-in can support OpenCL Images render.

  • Type - string X 1

  • Property Set - plug-in descriptor (read/write), host descriptor (read only)

  • Default - “false” for a plug-in

  • Valid Values - This must be one of

    • ”false” - in which case the host or plug-in does not support OpenCL Images render

    • ”true” - which means a host or plug-in can support OpenCL Images render

kOfxImageEffectPropOpenCLEnabled

Indicates that a plug-in SHOULD use OpenCL render in the current action.

If a plug-in and host have both set kOfxImageEffectPropOpenCLRenderSupported=”true” or have both set kOfxImageEffectPropOpenCLSupported=”true” then the host MAY set this property to indicate that it is passing images as OpenCL Buffers or Images.

When rendering using OpenCL Buffers, the cl_mem of the buffers are retrieved using kOfxImagePropData. When rendering using OpenCL Images, the cl_mem of the images are retrieved using kOfxImageEffectPropOpenCLImage. If both kOfxImageEffectPropOpenCLSupported (Buffers) and kOfxImageEffectPropOpenCLRenderSupported (Images) are enabled by the plug-in, it should use kOfxImageEffectPropOpenCLImage to determine which is being used by the host.

kOfxImageEffectPropOpenCLCommandQueue

Indicates the OpenCL command queue that should be used for rendering.

This property contains a pointer to the command queue to be used for OpenCL rendering (cl_command_queue). In order to use it, reinterpret_cast<cl_command_queue>(pointer) is needed.

The plug-in SHOULD ensure that its render action enqueues any asynchronous OpenCL operations onto the supplied queue.

The plug-in SHOULD NOT wait for final asynchronous operations to complete before returning from the render action.

kOfxImageEffectPropOpenCLImage

Indicates the image handle of an image supplied as an OpenCL Image by the host.

  • Type - pointer X 1

  • Property Set - image handle returned by clipGetImage

This value should be cast to a cl_mem and used as the image handle when performing OpenCL Images operations. The property should be used (not kOfxImagePropData) when rendering with OpenCL Images (kOfxImageEffectPropOpenCLSupported), and should be used to determine whether Images or Buffers should be used if a plug-in supports both kOfxImageEffectPropOpenCLSupported and kOfxImageEffectPropOpenCLRenderSupported. Note: the kOfxImagePropRowBytes property is not required to be set by the host, since OpenCL Images do not have the concept of row bytes.

kOfxOpenCLProgramSuite

Typedefs

typedef struct OfxOpenCLProgramSuiteV1 OfxOpenCLProgramSuiteV1

OFX suite that allows a plug-in to get OpenCL programs compiled.

This is an optional suite the host can provide for building OpenCL programs for the plug-in, as an alternative to calling clCreateProgramWithSource / clBuildProgram. There are two advantages to doing this: The host can add flags (such as -cl-denorms-are-zero) to the build call, and may also cache program binaries for performance (however, if the source of the program or the OpenCL environment changes, the host must recompile so some mechanism such as hashing must be used).

struct OfxOpenCLProgramSuiteV1
#include <ofxGPURender.h>

OFX suite that allows a plug-in to get OpenCL programs compiled.

This is an optional suite the host can provide for building OpenCL programs for the plug-in, as an alternative to calling clCreateProgramWithSource / clBuildProgram. There are two advantages to doing this: The host can add flags (such as -cl-denorms-are-zero) to the build call, and may also cache program binaries for performance (however, if the source of the program or the OpenCL environment changes, the host must recompile so some mechanism such as hashing must be used).

Public Members

OfxStatus (*compileProgram)(const char *pszProgramSource, int fOptional, void *pResult)

Compiles the OpenCL program.