Core Image workingColorSpace & outputColorSpace-CodePudding

I am rendering video frames using Metal Core Image shaders. One of the requirements I have is to be able to pick a particular color (and user selected nearby range) from the CIImage, keep that color in the output and turn every other else black and white (color splash). But I am confused about the right approach that would work for videos shot in all kinds of color spaces (including 10 bit HDR):

First job is to extract the color value from the CIImage at any given pixel location. From my understanding, this can be extracted using the following API:

func render(_ image: CIImage, 
  toBitmap data: UnsafeMutableRawPointer, 
  rowBytes: Int, 
  bounds: CGRect, 
  format: CIFormat, 
  colorSpace: CGColorSpace?)

The API says passing NULL to colorSpace will cause the output to be in ciContext outputColorSpace. It's not clear how to correctly use this API to extract the exact color at given pixel locations, given the possibility of both 8 bit and 10 bit input images?

Having extracted the value, the next issue is how to pass the values to Metal Core Image shader? Shaders use normalized color ranges dependent on the workingcolorSpace of ciContext. Do I need to create a 1D texture with the color that should be passed to shader or there is a better way?

CodePudding user response：

Based on your comment, here is another alternative:

You can read the pixel value as floats using the context's working color space. By using float values, you ensure that the bit depth of the input doesn't matter and that extended color values are correctly represented. So for instance, a 100% red in BT.2020 would result in an extended sRGB value of (1.2483, -0.3880, -0.1434).

To read the value, you could use our small helper library CoreImageExtensions (or check out the implementation to see how to use render to get float values):

let pixelColor = context.readFloat32PixelValue(from: image, at: coordinate, colorSpace: context.workingColorSpace)
// you can convert that to CIVector, which can be passed to a kernel
let vectorValue = CIVector(x: pixelColor.r, y: pixelColor.g, ...)

In your Metal kernel, you can use a float4 input parameter for that color.

You can store and use the color value on later rendering calls as long as you are using the same workingColorSpace for the context.

CodePudding user response：

I think you can achieve that without worrying about color spaces and even without the intermediate rendering step (which should speed up performance a lot).

You can simply crop your image to a 1x1 px square image that contains the specific color and make the image to extent virtually infinitely in all directions. You can then pass that image into your next kernel and sample it anywhere to retrieve the color value (in the same color space as before).

let pixelCoordinate: CGPoint // the coordinate of the pixel that contains the color

// crop down to a single pixel
let colorPixel = inputImage.cropped(to: CGRect(origin: pixelCoordinate, size: CGSize(width: 1, height: 1))
// make the pixel extent infinite
let colorImage = colorPixel.clampedToExtent()

// simply pass it to your kernel
myKernel.apply(..., arguments: [colorImage, ...])

In the Metal kernel code, you can simply access it via sampler (or sample_t in a color kernel) and sample it like this:

// you can sample at any coord since the image contains the single color everywhere
float4 pickedColor = colorImage.sample(colorImage.coord());