Comparison of Scaler Kernels in y4mscaler
Step Response

This page illustrates the step response of the various kernels implemented in y4mscaler's default scaling engine.

The synthetic source image used here contains three horizontal step functions, like this:

The first step is from 1 to 254 --- these are the limits of "valid" pixel data. The second is 16 to 235 --- these are the black and white luma values. The third is 72 to 184 --- these are just grayscale values. (Here, the first step looks no different from the second, because its values are outside of the range of R'G'B' colors.)

In the table below, each row corresponds to a different "M/N" downscaling factor, and each column corresponds to a different kernel in the scaler engine. The source frame for each row is "N/M" times the final horizontal size, and is then scaled horizontally using y4mscaler:

    y4mscaler -I bg=rgb:128,128,128 -O yscale=1/1 -O xscale=M/N -S option=KERNEL -S mode=mono
The output of each step is plotted in the little graphs shown below. The graphs focus on the 5 pixels before and after the step.

Here is the step response table:

M/N box linear quadratic cubicB cubicCR cubic cubicK4 sinc:4 sinc:6
2/3
1/2
1/3
1/4

Higher order filters with negative coefficients (e.g. cubicCR, sinc:N) exhibit a small amount of overshoot and ringing. (The Y'CbCr colorspace has its min and max values at 16 and 235/240 precisely to accommodate this kind of overshoot.)

You may also want to see the resulting scaled step-function frames themselves, a comparison which more directly illustrates the effects of ringing, or even a brief comparison with yuvscaler.


return to y4mscaler

maximum impact research
Digital Media Group
<dmg at mir.com>
Last modified: Sun Dec 11 14:55:46 EST 2005

©2003 Matthew Marjanovic.
This material may not be republished in any form without express written consent of the author.