Overlays and diagonal line

It seems nVidia Geforce FX cards don't support overlays smaller than 1x or
non 4:3 overlays. When trying to use an overlay non 4:3 it just looks very
distorted (hardly see anything) and when trying to make an overlay smaller,
instead the overlay stays at original size and gets clipped.

Both of these are from getting overlays directly from a Bt878 card, no such
problems in playing videos (presumably because it's using GDI hwstretch for
it in that cases.

Also, I'm getting this odd diagonal line in overlay playback, noticible in
flashes (when video goes suddenly from dark to light and back). I thought
this may have been a form of tearing, but the line looks exactly the same at
all refresh rates.

I also get this line (although fainter) in MAME with Direct 3D blitter,
however it's in the opposite direction. I never noticed these lines with
other cards.