Nvidia released a new video driver for Linux, the version 180.25 (x86, x86_64).
They also released a new mplayer-vdpau, many people have been waiting for, since there have been memory allocation issues with the old version.
This incarnates in the notorious ‘error 25’ or sometimes also error 23:

Error 25 at libvo/vo_vdpau.c:990

Activating TwinView and starting a clip, that was working with a disabled TwinView still gives the Error 23 thing:

Error 23 at libvo/vo_vdpau.c:790

And tearing is still present.

Like mentioned by the Nvidia developers on their forum, no big changes were done to the code, except the changes, that have been supplied by the mplayer team itself have added to this release. Why there is the nvidia-settings tool in the x86 directory the very first time at all, and this only there and not in the x86_64 directory, I didn’t find out.

Unfortunately there are still issues left with the memory allocation, and either tweaking line 94 in ./mplayer-vdpau/libvo/vo_vdpau.c, where the value for NUM_VIDEO_SURFACES_H264 modified, by setting it to anything highter like instead of 16 up to 32 or to something lower, like 9.
Changing the line 705 produces more or less the same result:

--- mplayer-vdpau~/libvo/vo_vdpau.c	2009-01-22 05:30:12.000000000 +1100
+++ mplayer-vdpau/libvo/vo_vdpau.c	2009-01-22 01:20:23.000000000 +1100
@@ -701,7 +701,7 @@
             uint32_t round_width = (vid_width + 15) & ~15;
             uint32_t round_height = (vid_height + 15) & ~15;
             uint32_t surf_size = (round_width * round_height * 3) / 2;
-            max_references = (12 * 1024 * 1024) / surf_size;
+            max_references = ((12 * 1024 * 1024) / surf_size) + 11;
             if (max_references > 16) {
                 max_references = 16;
             }

This results only in the fact, that some files can be played back, now, but files, which have been already working, don’t work anymore.

Restarting X (Ctrl+Alt-Backspace) is the most efficient way to solve this problem, at least sometimes.

My workaround
I’ve found finally a workaround, which works for really all of my .mkv (Matroska), .mts (AVCHD), .ts (VC-1). The only file with WMV3 I have, played also well.

Most of you won’t like my solution, but it shows once more it’s matter of available memory.

The solution I’m using now on my Media Center PC, which is set up with My Media System (MMS-1.1.0-RC9), is running now completely without any window manager (kdm/gdm/xdm), but only a tiny (I mean really tiny and minimalistic) window manager (matchbox) is used for a basic window management. Actually I wouldn’t need this window manager either, if I don’t launch other application from within MMS.
Window manager like fvwm or even XFCE or IceWM I considered as much too big.

MMS has a OpenGL (animated) output method (mms -o opengl), but also here, I have to use the SDL output method (mms -o sdl)to make all HD videos work.

All is done on a Nvidia 8400GS 256MB and an Ubuntu 8.10 (Intrepid Ibex).
I’ve had also a few promising test series on a Fedora8 with KDE3.5 (no Compiz and no compositing enabled). There, I have only one sample, that refuse to play.

All tests were done with the line 704 +11 tweak.

Here’s simple method the test, if the mplayer also works for you flawlessly, if you start it without any window neither desktop managers.
Here are my not very difficult six steps: