I’ve reported several times about tearing with Nvidia’s new VDPAU driver. Now it seems I’ve found a working solution for me, to get rid of this pretty ugly tearing effect. Here are my steps:

• Use latest driver (>=180.27) from the launchpad PPA repository. Add these lines into your /etc/apt/sources.list if you are using Debian/Ubuntu. If not, get Nvidias installer here: x86, x86_64

  # nvidia experimental driver
  deb http://ppa.launchpad.net/anders-kaseorg/ubuntu intrepid main

  Perform an apt-get update, and install all the packages listed here:

  apt-get update
  apt-get install nvidia-180-kernel-source nvidia-180-libvdpau nvidia-180-libvdpau-dev nvidia-glx-180 nvidia-glx-180-dev

• Backup your existing /etc/X11/xorg.conf first. Now you have two possibilities:
  1. add these lines into your xorg.conf

     Section "Extensions"
         Option         "Composite" "Disable"
     EndSection

  or perform

  2. nvidia-xconfig -no-composite
• restart your KDE or Gnome session (Ctrl+Alt+Backspace)

In other words: Make sure you’re not using compositing. This means you can’t use all these fancy 3D effect, that Compiz, Metacity or KWin offers. To be honest, I do not miss them on my Media Center PC with MMS installed. Although My Media System (MMS) does also have nice animations, which also won’t work either.

[UPDATE] don’t read further if: I’ve you suffer from the green screen crash/problem: here’s how I fixed it: [1]

[UPDATE] [ *** don’t use this – see UPDATE2 below *** ]
Disabling Composite had a strange side effect. Some of my movies just stop playing after approximately 30-60 seconds, Xorg and/or mplayer began to suck almost 100% CPU, and the video stopped/crashed after another minute while showing nothing but strange green to yellow screen.
After doing a lot of senseless things, I’ve stumbled across a strange effect. I found out, that the crash only occur if the graphics adapter was connect via DVI and was running 50Hz (PAL/Europe). Setting the frequency to 60Hz on my Sharp 37″ DFP solved the problem. Unfortunately I didn’t find out how to force 60Hz with the nvidia-settings tool. Each time I restarted my X, the frequency fell back onto 50Hz (EDID default). Only an entry into xorg.conf, which is explicitly forcing to use 60Hz led to a success. Using the analog VGA connector works always fine.
This is how my “Device” section looks now:

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce 8400 GS"
    Option "HorizSync"
    Option "VertRefresh" "60"
    Option "UseEdidDpi" "flase"
EndSection

[UPDATE2] [ *** !!for experts only!! *** ]
It turned out, that it’s not the selected frequency, but the necessary switch of any frequency while X is running. You can e.g. start nvidia-settings, change the frequency, and hit the “apply” button. After this action the mpayer player playes fine, and no green screen appears. Doing this a seconds time and so on doesn’t work for me anymore. No I had the problem to implement this workaround. I did it like this (I know, very odd, but it works now reliably ).

1. create two different xorg.conf, one with two different frequencies (here are mine [1],[2])
2. Assuming you know what you’re doing and you are using your own .xinirc do the following
3. sudo cp /etc/X11/xorg.conf2 /etc/X11/xorg.conf
4. sudo nvidia-settings –load-config-only &
5. sleep 4
6. sudo cp /etc/X11/xorg.conf1 /etc/X11/xorg.conf
7. xterm -font 12×24 -fg white -bg black -geometry 80×25+100+100 &
8. sleep 2
9. sudo nvidia-settings –load-config-only &
10. …here the rest of your  .xinitrc script

I didn’t tested this thoroughly, I only saw, that the first few minutes are playing fine now.

Boy it’s late. To be continued… oops, it crashed again!, it seems I only deffered the crash. Ok, have to investigate a bit more, but later…
I’ll check tonight, if Nvidia 180.29 with latest mplayer-vdpau in version 3482714 has solved this issue. See here.