VP8

• VP8 Encode Parameter Guide
• An Analysis of VP8, a new video codec for the web
• webm wiki: FFmpeg VP8 Encoding Options
• ffmpeg wiki: Encode VP8
• ffmpeg libvpx documentation

With the defaults, vpxenc will create low quality encodes, its bitrate is too low. It will have to be bumped up to get some good quality.

When using ffmpeg, pass -b 0 (bitrate 0) so that it doesn't limit bitrate. Also use a CRF. For very high quality encodes, I'd say around 6 to 8.

ffmpeg -i video.y4m -vcodec libvpx -crf 6 -b:v 0 -y video.mkv

HandBrake:

HandBrakCLI -i video.y4m -e VP8 -q 6 -o video.mkv