Code for YUV video stream processing. This code extends code from the MJPEG tools and requires the mjpeg libraries. If you have difficulty compiling these contact me, mjpeg0 at silicontrip dot org.
You will need the mjpeg libraries to compile the yuv tools. information on the mjpeg libraries and download.
The Libav tools require the libavcodec library which is part of the ffmpeg tools
libav-bitrate
libav2yuv
libavmux
utilyuv
vf_broadcast
vf_overlay
vf_watermark
yuv2jpeg
yuvaddetect
yuvadjust
yuvafps
yuvaifps
yuvbilateral
yuvCIFilter
yuvconvolve
yuvcrop
yuvdeinterlace
yuvdeinterlaceB
yuvdiag
yuvdiff
yuvfade
yuvfieldrev
yuvGENERIC
yuvhsync
yuvilace
yuvnlmeans
yuvpixelgraph
yuvrfps
yuvsubtitle
yuvtbilateral
yuvtout
yuvtshot
yuvvalues
yuvwater
yuvyadif
Produces an ASCII file suitable for plotting in gnuplot of the bitrate per frame. Also includes a rolling average algorithm that helps smooth out differences between I frame, P frame and B frames. Constant bitrate is not so constant. Requires about 50 frames rolling average window to get an idea of the average bitrate.
using a post processing tool such as octave to average the graph may be useful
VBR example
CBR example
Tested the EDL mode of this program. Discovered what appears to be a bug in the libav library in which frames are unexpectedly dropped. I have attempted to work around the bug, however it appears to be filetype specific. Currently m2v files work accurately. Use this with the updated yuvdiag tool to stamp your video for EDL file creation. Or use a frame accurate video player.
This is a very simple generic anything to yuv tool, using the libav library from ffmpeg. It supports any video file that libav supports (which includes WMV3s now!!!) it can be used to replace the -vo yuv4mpeg option of mplayer or the -f yuv4mpegpipe option of ffmpeg. The tool can also override yuv header values, in the case of incorrect aspect ratio or interlacing etc. And will perform simple chroma subsampling conversion, if the source file is not in the desired chroma subsampling mode.
This release now includes an EDL file parser, for joining or editing files. This is a first working version is not optimised and may contain bugs. I hope to have more testing over the next few months. See sample EDL file below.
Now uses sws_scaler for chroma resampling rather than the deprecated img_convert, So will compile with the newer version of FFMPEG. Also have provision for decode_audio3 and decode_video2 which has appeared in the latest ffmpeg snapshot.
I use this regularily for file conversions, so I hope the code is quite mature, with a thorough set of options and help. Any bugs should be picked up quickly.
As I do want this to be a one program fits all for yuv video conversion any bugs or issues you have with the software I would like to hear about so I can make this the best software.
libav2yuv strangefile.avi | y4m-yuvfilter | ffmpeg -f yuv4mpegpipe -i - -vcodec whatever wantedfile.avi
#[num] [duration] [srcIn] [srcOut] [recIn] [recOut] /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_01.mpg VA C 0:0:0:0 0:4:9:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_02.mpg VA C 0:0:0:0 0:3:39:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_03.mpg VA C 0:0:0:0 0:2:38:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_04.mpg VA C 0:0:0:0 0:3:9:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_05.mpg VA C 0:0:0:0 0:4:39:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_06.mpg VA C 0:0:0:0 0:4:9:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_07.mpg VA C 0:0:0:0 0:2:9:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_08.mpg VA C 0:0:0:0 0:3:9:0 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_09.mpg VA C 0:0:0:0 0:3:38:15 /Users/d332027/Movies/TheFaceOfTheEnemy/the_face_of_the_enemy_10.mpg VA C 0:0:0:0 0:5:46:0
The fields are:
This is a very simple avi/mov/mpg demuxer and muxer.
The demuxer works quite well, however the muxing part still has bugs and has not correctly produced a working file. This is still a work in progress.
Converts full swing levels to broadcast swing for file which have been incorrectly labelled.
Performs clipping, scaling or detection of out of range Luma values (16-235)
This was my first paid video filter job, to add a static image overlay to a video file.
This filter handles PNGs with alpha chanels (and any other image format that libavcodec can read). The image can be positioned and scaled. And also has an interval on and off.
The filter uses the arguments:
image_name[:X:Y[:W:H[:on_time:off_time]]] or config_name.cfg (the .cfg is required)
The config file is a quick way to store frequently used settings. The config file consists of:
filename mywatermark.png x 640 y 400 w 64 h 16 ontime 5 offtime 55
The interval uses the source video's DTS value to calculate if the watermark is shown or not. This may produce unexpected results if the source video has timecode breaks, or if part of the video is skipped using the ffmpeg command line option -ss. ontime and offtime must both be specified for interval display to work. The default values for ontime and offtime are -1 meaning do not use interval display, the watermark remains visible for the entire duration of the video.
Most of the functionality of this code has been replaced by yuvvalues.c any further work on this will be focussed there.
Currently only produces a text file (which can be graphed by gnuplot) of the average lightness of the sourceframes. The idea is that there is a black fade before and after advertisements. To produce chapter markers for DVDs from TV.
From looking at this above graph it appears that this program is doing a frame difference, rather than averaging luma. Which is the same function as yuvdiff. Now we can quite easily see scene changes, shown as sharp spikes. At about 810 and 1500
We can also tell that the video has gone through a frame rate doubling by frame dupliation. Shown by the high frequency component.
I should move this image down to YUV diff
Allows changes such as brightness, contrast, saturation, hue shift and hue rotation. Supports negative values for inversion.
A linear frame averaging, frame rate converter. Does a better job than the frame dropping/duplicating converters.
I did think that this no longer needed maintaining since yuvfps now has a weighted average conversion. However I find this programs conversion much nicer than yuvfps. This will work with converting very slow or very fast frame rates and averages all frames that overlap.
It supports interlaced and progressive streams, and can force a conversion from progressive to interlace (although I'm not sure if it properly handles progressive to interlaced chroma)
I have re-written this to use integers rather than floats, appears to increase speed by about 3 times.
It may overflow with long files (greater than about 80 minutes) and large prime number frame rate values. However these values are not very common. I've tried to detect this occurance, but it simply reports an error on stderr and continues with a corrupted stream.
performs a bilateral filter over the video. Is horribly slow, and a little too agressive.
Based on code from m.eitz
Useful options are:
-D 3 -R 3
Higher values of R cause more smearing. Higher values of D increase the search radius, and increase processing time.
This allows Core Image filters to operate on yuv streams.
Obviously this program only works under OSX.
To choose a Core Image filter use the -f option: | yuvCIFilter -f CILineScreen |
Default filter settings are shown:
{CILineScreen {
inputAngle = 0;
inputCenter = [150 150];
inputImage = ;
inputSharpness = 0.7;
inputWidth = 6;
}}
Any of the filter settings can be changed using the -i option: yuvCIFilter -f CILineScreen -i inputAngle=45.0
Multiple settings are separated by a comma: yuvCIFilter -f CILineScreen -i inputAngle=45.0,inputWidth=12.0
Some settings require a space and will need to be quoted on the command line.
A list of filters is obtained by giving an invalid filtername to the -f option (such as help)
CIAdditionCompositing CIAffineClamp CIAffineTile CIAffineTransform CIAreaAverage CIAreaHistogram CIAreaMaximum CIAreaMaximumAlpha CIAreaMinimum CIAreaMinimumAlpha CIBarsSwipeTransition CIBlendWithMask CIBloom CIBoxBlur CIBumpDistortion CIBumpDistortionLinear CICheckerboardGenerator CICircleSplashDistortion CICircularScreen CICircularWrap CICMYKHalftone CIColorBlendMode CIColorBurnBlendMode CIColorControls CIColorCube CIColorDodgeBlendMode CIColorInvert CIColorMap CIColorMatrix CIColorMonochrome CIColorPosterize CIColumnAverage CIComicEffect CIConstantColorGenerator CICopyMachineTransition CICrop CICrystallize CIDarkenBlendMode CIDifferenceBlendMode CIDiscBlur CIDisintegrateWithMaskTransition CIDisplacementDistortion CIDissolveTransition CIDotScreen CIEdges CIEdgeWork CIEightfoldReflectedTile CIExclusionBlendMode CIExposureAdjust CIFalseColor CIFlashTransition CIFourfoldReflectedTile CIFourfoldRotatedTile CIFourfoldTranslatedTile CIGammaAdjust CIGaussianBlur CIGaussianGradient CIGlassDistortion CIGlassLozenge CIGlideReflectedTile CIGloom CIHardLightBlendMode CIHatchedScreen CIHeightFieldFromMask CIHexagonalPixellate CIHoleDistortion CIHueAdjust CIHueBlendMode CIKaleidoscope CILanczosScaleTransform CILenticularHaloGenerator CILightenBlendMode CILinearGradient CILineOverlay CILineScreen CILuminosityBlendMode CIMaskToAlpha CIMaximumComponent CIMaximumCompositing CIMedianFilter CIMinimumComponent CIMinimumCompositing CIModTransition CIMotionBlur CIMultiplyBlendMode CIMultiplyCompositing CINoiseReduction CIOpTile CIOverlayBlendMode CIPageCurlTransition CIParallelogramTile CIPerspectiveTile CIPerspectiveTransform CIPinchDistortion CIPixellate CIPointillize CIRadialGradient CIRandomGenerator CIRippleTransition CIRowAverage CISaturationBlendMode CIScreenBlendMode CISepiaTone CIShadedMaterial CISharpenLuminance CISixfoldReflectedTile CISixfoldRotatedTile CISoftLightBlendMode CISourceAtopCompositing CISourceInCompositing CISourceOutCompositing CISourceOverCompositing CISpotColor CISpotLight CIStarShineGenerator CIStripesGenerator CISunbeamsGenerator CISwipeTransition CITorusLensDistortion CITriangleTile CITwelvefoldReflectedTile CITwirlDistortion CIUnsharpMask CIVortexDistortion CIWhitePointAdjust CIZoomBlur
Performs a generic convolution filter on the video. Quite slow. Support any odd dimension matrix (3x3, 5x5, 7x7...) uses the command line argument -m 1,2,3,4,5,6,7,8,9.
I am thinking about adding support for predefined matricies, such as blur, sharpen, edge detection, emboss.
I now know that it's not wise to run sharpening on low bitrate mpeg files, as it highlights the macroblocks.
Used to detect matting in yuv sources. It can also be used to crop and matte a yuv video stream.
Cropping makes the destination video frame dimensions smaller, while matting will keep the video the same size, but replace pixels with a solid colour.
A non destructive deinterlacer. Converts to a double frame rate, progressive yuv stream for further processing by temporal based non interlace aware filters. Then re-interlaced or frame rate converted before encoding.
This program uses an experimental detection algorithm to detect interlaced pixels and then uses cubic interpolation to replace pixels that are suffering comb effect. Image data is not lost as the frame rate is doubled. This adaptive algorithm is quite effective, however not ideal. Noisey artefacts may be apparent in (some) most material.
I have been looking into implimenting a 4 point DFT on the data and looking at the amplitude of the high (/2) frequency component. This has lead to better results but appears to still have issues at the boundary of interlace and non interlace material.
The above code could be extended to an arbitary number of pixels, comparing the /2 (interlace) frequency with the /4 frequency.
This release of the code I have played with comparing both the high frequency and the next frequency component. The high frequency shows areas of interlace however also detects edges. The second frequency component detects edges. By balancing these two results, by trial and error, I beleive to have come up with a better detection algorithm. However non interpolated interlace artefacts are still apparent. This is the best results so far.
Just when I thought I tuned the DFT algorithm to the best parameters. I came up with another method that simply looked for a "greater than, smaller than, greater than" pattern and as a result are now detecting interlace much more accurately than before. I also added an additional test on the AC value to eliminate non noticable interlace (or noise). This version is now available.
I am extremely happy with the results. Although not perfect, there are false positives, and a few false negatives. Of course I will be fine tuning this filter even more over time. For now I would use it as part of a yuv filter chain for my production videos.
Write a pixel merging algorithm for conversion of 25i material to 25p (rather than 50p).
Look at ways to perform anti-aliasing for fixing up nearest neighbour de-interlace filters. These sorts of filters
should be shot, as it's quite easy to write a linear interpolator, not to difficult to write a cubic interpolator,
and look I just wrote an adaptive type de-interlacer. I can't believe I saw this kind of de-interlace filter used
on an episode of Top Gear the other night. Disgraceful!
Provide a screenshot...
I have been experimenting with training a neural net to *learn* what the missing pixel may be from the surrounding pixels. My first attempts have not worked, if it is trained from images with many uphill lines, then uphill interpolation looks fantastic but downhill is very jaggered.
I am thinking about increasing the number of hidden layers in the neural net to see if it improves interpolation. The code is in java, I'll just need some encouragement to look at it again.
This part of the code would replace the cubic interpolator. Detection would still be separate algorithm, however comparing the interpolated pixel with the real pixel and replacing the real pixel with the interpolated if it is different by a threshold amount. May prove a better algorithm.
by default yuvdeinterlace will double the frame rate and convert fields into full height frames. No arguments needed.
This tool has 5 operating modes.
This program needs some work to add labels to the histogram scopes, but not being a video engineer, I do not know exactly what to add.
feedback would be appreciated.
By default this program produces a video of frame 2 minus frame 1 frame 3 minus frame 2 and so on.
The -g option suppresses video output and produces an ASCII file suitable for plotting in gnuplot of the difference between consecutive frames or fields. This can be used to detect 3-2 pulldown, or any other pulldown (frame duplication) frame rate conversion. If the output is ran through an FFT it can even detect other frequency rate changes such as 2.39 (25 to 59.94, progressive PAL to interlaced NTSC)
This program does not remove pulldown. yuvrfps can be used for frame duplication removal. I do not know of a way to remove frame blended frame rate conversion.
I have used this extensively to detect pulldown material.
This progeam can also be an aid to trimming video. It can compare multiple reference frames and produce an ASCII file suitable for plotting. The frame with the least difference is the closest matching frame.
With the -b option it will also search for frames closely matching a black frame.
To produce a video showing the differences between each frame: | yuvdiff |
To produce an ASCII file showing the differences between each frame for detecting pulldown or frame rate conversion: |yuvdiff -g > output.txt
To search for a reference frame: | yuvdiff -g search_frame.y4m > output.txt
To search for multiple reference frames and the black level: | yuvdiff -g -b start_frame.y4m end_frame.y4m > output.txt
The program produces this ASCII output:
Interlace, with multiple reference files (if -b specified, is always the last column)
1 20422241 15400627 24882428 1.5 20482772 15072749 24912090 2 20436052 15407218 24887759 2.5 20495250 15077157 24917718 3 20433069 15413509 24896056 3.5 20487508 15087745 24917554 4 20425924 15401108 24869589 4.5 20502369 15072110 24894271
Progressive difference only
1 1147577 2 1003628 3 1078191 4 1260775 5 1253182 6 1364214 7 1364779 8 1234701 9 1306411 10 1443550
Will fade the video to black after X number of frames. Uses a trial and error method of fading to black, could be better.
I used this program to give a more professional feel to the downloaded internet videos I was converting
Works by delaying the fields by 1 field. May also require audio adjustment to keep the audio in sync
I've noticed that some broadcast DVBT material is field reversed progressive material. This filter can return the content back to progressive.
Attempts to perform interlace detection, by FFT the vertical lines in the screen, and averaging the resulting frequency spectra. Also tried experimenting with filtering to intelligently remove interlace.
This code simply performs a 1d vertical FFT on the video
In a full height progressive display, if the frame is treated as occuring at the same temporal value, stationary images retain their full height resolution, however moving images suffer from "comb" effect
If the entire frame is treated as two separate temporal images, then the vertical resolution is halved. A full height frame can be produced by interpolation, however this is noticable and would needlessly degrade the the image. Stationary portions would lose information. Moving portions have already lost information due to the nature of interlace.
Neither case is true, as part of the image may be stationary and part may be moving. Detecting the moving portion and interpolating only the needed pixels is the goal of this code.
An NL-means spacial filter. this performs an NL-means spatial filter on the video stream. It is slow, approximately 15 seconds per SD frame (2.4ghz Core 2 Duo) The noise reduction from the NL-means algorithm is quite effective.
I am looking at methods to optimise this code
produces a text file of the chroma channels of a single pixel.
I have been using this to analyse some chroma-luma bleeding (rainbow) I've found on commercial DVDs. In order to write a filter to remove this rainbow effect.
The pixel location is hard coded.
Performs frame rate reduction by simple fractional amounts, by dropping 1 out of every X frames or fields.
This is to work with progressive 3:2 pulldown files performing conversions such as 29.97 to 23.976 similar to yuvkineco or even 30 to 25 fps. It analyses X frames and finds the least different fram e and drops it, in theory removing any duplicated frames.
It is possible that the 3:2 pulldown cadence starts on an interlace frame in which the detection will make all frames interlace (field reversed progressive) this can be corrected by the -s option to start the sequence on a progressive frame. It should be less than the Decimation number.
-Freduces X frames to X-1 Common Decimation ratios 5: 29.970 -> 23.976 6: 30 -> 25 -f force drop frame X (or X:Y fields) -I t|b|p Force interlace mode -s skip X frames. Output X frames unchanged before starting detection.
a subtitle rendering utility for yuv streams
Reads an ascii subtitle file and renders a TTF in the image
The subtitle file is in the format
10,100,Leah and the wookie must never again leave this city. 100,170,That was never a condition of our arrangment.
-fpath to TTF file. -s size to render the subtitle -y vertical position on screen -c colour in the format y,u,v -u subtitle file filename
performs a temporal bilateral filter over the video.
As the bilateral filter is edge preserving, this filter has innate motion awareness This filter attempts to perform a time based noise reduction while minimising ghosting.
Based on code from m.eitz
Useful options are:
-D 3 -R 3
Higher values of R cause more ghosting. Higher values of D increase the search radius, and increase processing time.
This is what the yuvtshot filter is meant to do for VHS speck noise
This is a spatial filter which detects and removes the white "speck" noise from VHS tapes.
I have spent a while tuning the detection algorithm, it appears quite effective. This filter is more useful at removing VHS noise than the tshot filter.
Removes temporal shot noise by use of a median filter in 1 or more dimensions
-a process all pixels. Do not adaptively select noise pixels -c process chroma only -y process luma only -m modes: OR'd flags together
Can remove VHS "sparkle" noise with mode 4.
prints timecode, difference and min/average/max of the yuv channels for each frame
-d to use NTSC drop frame timecode
1 00:00:00:01 1.45315 10 17.4508 145 126 127.005 128 125 127.995 130 2 00:00:00:02 0.297374 12 17.4305 145 126 127.005 128 125 127.999 130 3 00:00:00:03 0.189955 12 17.4328 145 126 127.005 128 126 127.997 130 4 00:00:00:04 0.324564 13 17.3704 145 126 127.005 128 125 127.994 130 5 00:00:00:05 0.348811 13 17.3848 145 126 127.005 128 125 127.996 130 6 00:00:00:06 0.268432 14 17.3737 145 126 127.005 128 125 127.997 130 7 00:00:00:07 0.650675 13 17.8667 145 125 126.992 130 124 128.065 131 8 00:00:00:08 3.21166 12 21.0664 145 119 126.622 135 117 129.807 140 9 00:00:00:09 4.08278 13 25.1366 145 116 126.314 141 116 131.849 145 10 00:00:00:10 4.6215 13 29.6829 145 114 125.886 145 115 134.196 150
Attempts to detect and remove semi-transperant watermarks from the source. Produces a PGM file of the detected watermark which is used to remove or reduce the effect. This is a two pass process, the documentation is a little sparse.
The first pass produces a grey image file (PGM format) this is done by averaging luma of all the frames. The idea is that the watermark is consistantly brighter than the surrounding video. The PGM file may need to be edited (blurred) in places as other long term artifacts may be visible.
The second pass modifies the luma of the video reducing the brightness by the amount in the pgm file. This is not a linear reduction. Trial and error was used to determine the correct formula to reduce the luma by. For flat watermarks, this function is ideal.
Non flat watermarks cannot be removed by this program. Colour watermarks cannot be removed by this program. Solid watermarks cannot be removed by this program.
Pass 1: | yuvwater -d > watermark.pgm
May also use a black frame with the watermark showing, extracted by other means.
pass 2 | yuvwater yuvwater -m 145 -l 72 -u 384 -i watermark.pgm | -m specifies the amount to remove, the lower the number the darker the resulting watermark, good starting values are 140. -l specifies the black level for normalisation, good starting values between 32-80. If the black is too light, increase this value. -u specifies the white level, good starting value 384, if the white is too dark, decrease this value.
An implementation of the YADIF deinterlace filter for yuv streams.
-I force interlace mode t|b. top or bottom field first.
Isn't Chyler Leigh a goddess?