Codec shoot-out 2003 - 1st installment
Table of content:
2: Test setup
3: Test 1: The Matrix
4: Test 2: Saving Private Ryan
5: Test 3: Futurama
7: Future outlook
Phew. For the past 3 weeks I've literally spent every free minute in preparation for a new codec comparison. The new comparison was supposed to be the biggest one so far, increasing the number of sources and codecs to a previously unseen level. I also meant to include a H.264 (AKA MPEG-4 AVC) codec into the test, but having spoken to the developers of the prospective candidate (Vanguard Software H.264 beta2) I decided to postpone the test because this codec currently does not have any postprocessing, and does not correspond to the latest revision of the standard (the standard has been finalized a few weeks back). Beta3, which should fix those "issues" was supposed to be released in time for the comparison, but is still not out at the time I'm writing this. This left me with 7 codecs to be tested on the two well known sources and a new one: Futurama (an animated series from the creators of the Simpsons).
Now a quick overview of what has happened since the last comparison:
The missing contestants: The following codecs were supposed to be included but didn't make it for various reasons: Dicas mpegable X4, the commercial encoding application by Dicas, comes as a 30-day trial. However, that trial ran out after only starting the program twice and I couldn't find a way to circumvent the limits (OK, reinstalling would've done it but that's too much to ask for). I got an unlocked version by Dicas, which puts a logo in every frame and which I did not deem acceptable for a production environment. I requested a key to unlock my locked 30 day trial but I never got one. I also asked On2 a 3rd time to evaluate VP5 and I did not get a reply for the 3rd time. On2 has publicly announced VP6 to be released in mid-May, we'll see if there's a change in their stance towards public scrutiny by then. I also did not include the Sigma Designs MPEG-4 codec. Forst of all, the license violation issue has never been properly settled, and it's nothing more than a rebranded XviD codec.
Last but not least: In order to present each codec the best way possible, I contacted each codec maker 1-2 weeks prior to the start of encoding to give them a chance to suggest the best settings for the movies used in this test. So, if you think my testing methods and settings are flawed, you're welcome to take it up with the developers because I stuck to their recommendations and if they gave sub-optimal settings it's their loss. At this point I'd like to thank all people who've worked with me in this area. The only codecs where I did not get any settings for were 3ivX and WMV9, despite having contacted the two companies.
Now let me present our contestants:
Dicas mpegable AVI 2.0.3
DivX3 in the form of SBC (Smart Bitrate Control) as done in Nandub (hereafter referred to as SBC)
RealNetworks RealVideo9 based on HelixProducer 9.1 M6 Gold (hereafter referred to as RV9)
Windows Media Video V9 (VCM beta1, hereafter referred to as WMV9)
XviD (Isibaar's development build dated 4/24, Isibaar branch in the CVS)
All codecs were tested in a 2 pass setup where this was applicable (3ivX and mpegable AVI only offer 1 pass encoding).
All testing was done on the following hardware:
AMD Athlon XP 2800+
Shuttle SN41 mainboard (part of the Shuttle SN41G2 barebone system)
2x256MB PC2700 Apacer CL2.5 DDR RAM
Creative 3d Blaster 4 Titanium4600 (NVidia Geforce4 TI4600)
Samsung SyncMaster 191T-P Display connected via DVI
To encode I used the following software:
Avisynth 2.51 April 16. for frameserving
VirtualDubMod 126.96.36.199b2 for DivX5/XviD encoding, VirtualDub 1.5.1 for 3ivX, mpegable AVI and WMV9
Nandub 1.0 RC2 lumafix ecffix for SBC
Helix Producer 9.1 M6 gold (build number 188.8.131.52)
To simplify things I used GordianKnot for the DivX3/5/XviD encoding and AutoRV9 for the RealVideo encoding.
Movies I encoded:
- Region1, NTSC, length: 2h16
Saving Private Ryan - Region1, NTSC, length: 2h49. I will call this movie SPR from here on
Futurama Season 2 Episode 1 - Region2, PAL, length: 0h21
I used a 128kbit/s ABR audio track created in lame 3.93 (using --alt-preset 128 setting) for all movies, then tried to put Matrix onto 1 700MB CD, SPR onto 2 700MB CDs and 4 Futurama Episodes onto one 700 MB CD (resulting in 175MB for an episode). The size of the audio track was 124221KB, 157455KB and 20582KB respectively. This resulted in an effective video bitrate of 581kbit/s for Matrix, 1016kbit/s for SPR and 987kbit/s for Futurama (here kbit = 1000bit, DivX3 uses kbit = 1024bits so you have to divide that bitrate by 1.024 to get the effective DivX3 bitrate, all other codecs use k = 1000). Please note that multiplexing a VBR MP3 with an AVI gives more overhead than a regular CBR MP3 file. I used GKnot to calculate the bitrate as it's known to be accurate in such situations.
As you may know, not every rate control mechanism is perfect so here are the final movie sizes I got
|3ivX||723'574 KB||1'493'398 KB||178'754 KB|
|DivX5||717'642 KB||1'435'154 KB||179'250 KB|
|mpegable||920'234 KB||1'144'774 KB||171'168 KB|
|RV9||722'977 KB||1'447'144 KB||180'976 KB|
|SBC||716'658 KB||1'434'004 KB||179'304 KB|
|WMV9||727'886 KB||1'460'384 KB||183'288 KB|
|XviD||717'242 KB||1'434'320 KB||179'288 KB|
Note that 700MB equals 716'800KB, 1400MB equals 1'433'600KB and 175 MB equals 179'200 KB. SBC is still best to match a given target size, closely followed by XviD and DivX5. RV9 still gets oversized (the longer the movie, the more), so did 3ivX and WMV9. As you can see the results for mpegable AVI are catastrophic. This has a reason: the WfV configuration interface is buggy (the people at Dicas admit that) and it happens quite often that your set bitrate gets reset to the default bitrate of 800 kbit/s. This explains why Matrix got oversized and SPR got undersized. As the difference in bitrate is considerable, it would not have been fair to compare mpegable AVI to the other codecs, and it was therefore excluded from the comparison at this point. Bottom line: If you need size predictability (who doesn't?), the codec choice narrows considerably already at this point.
Here I take Matrix as an example. The resolution of SPR was bigger, hence the performance also dropped. Note that you should take these values with a grain of salt. Only the first 10'000 frames were used for speed measures so the overall average FPS might be a bit different. Furthermore, I did not reset the PC before each measurement and I did not perform several measurements per codec. For good absolute values, more testing would be required, but these values should nevertheless give you a good impression of the relative speed difference between codecs. Note that the values are averaged over the two passes where applicable.
As you can see, speed has augmented considerably for both DivX5 and SBC as
compared to the last comparison. This is due to a much faster CPU and the fact
that all processing is now done in YV12 mode (AviSynth 2.51). A word about XviD
encoding speed: Futurama was encoded faster than the other two movies, mainly
because I didn't use B-frames, QPel and Chroma motion (all options that have
a tendency to slow things down considerably). It is to be expected that once
all the current features are stable, programmers will work on optimizing the
codec for speed. If you recall from the last comparison (where these features
were not present), XviD was the fastest codec of the bunch.
Other important stuff:
Here's the Avisynth script I used to encode the movies, so that you can perfectly reproduce my results. I used force film in DVD2AVI in Matrix and SPR. Futurama was interlaced, I used Decomb's FieldDeinterlace with no blending to get a progressive image both movies. Note that I used neutral bicubic resizing for every clip. I don't like the blurryness of the bilinear resize filter and the neutral setting is a good compromise between crispness and better compressibility but it doesn't blur.
For SBC and DivX5 I used the DivX5 filter at strength 4. I have been using the DivX5 filter to play back DivX3 movies a lot in the past and whenever I thought that something was odd I switched back to the DivX3 filter only to find out that the problem was the source, not the filter. I have also done the same during the comparison but I'm confident to say that the results would not be any different had I used the DivX3 filter for SBC playback (except for being more of a hassle to compare and make screenshots). For XviD I used ffdhow and set it to use XviD rather than ffdshow's own decoding routines. The build I got had internal XivD postprocessing turned on by default, in future versions you might have to turn on deblocking and deringing in the decoder options in the codec setup. Note that this postprocessing is brand new and the lead programmer suggesed that I used it for my comparisons. In SPR and Futurama, the CPU usage was above 90% so if you don't have a 2.8 GHz or higher CPU you should use Nic's filter (ffdshow has/had a bug when playing back XviD files where QPel was used). The other codecs do not permit postprocessing configuration.
All files were reviewed using Media Player Classic as it can play back AVIs and RV9 files (also QT by the way). Furthermore, the player can forward to any frame in the video which literally saved me days when making screenshots. Thank you Gabest for making such a great player! As I have a flatscreen monitor, I used the monitor's native resolution of 1280x1024 to review all clips. Flatscreens are not good at displaying lower resolutions full screen, plus zooming the video helps to accentuate any problems a codec might have (and which I would've certainly missed when reviewing at a low quality 640x480 resolution). Note that the screen used has a reaction time lower than 25ms which allows for flawless movie playback (and 3D shooters are okay, too).
A note about the screenshots: I did my best to present shots from the exact same frame. Some codecs exhibited delays (for instance in codec A you take frame X, in codec B you have to take frame X+1), and RV9 behaved rather erratically. Sometime I was unable to locate the frame that corresponded to the one used in all other codecs.
Now proceed to the first test.
This document was last updated on May 10, 2003