I haven't used any youtube clones and I haven't seen their code, so I can't recommend any of them to you. You can see their demo versions, read reviews, but most of the time it is just like buying a black box anyway Besides all these products might not satisfy your needs (or may have features that you don't need, but eat the resources of your server).

The best way to restrict media usage is probably by using DRM (Digital Rights Management). If you choose this way, then Windows Media will be a good solution. However most web-sites prefer to offer flash videos. One of the reasons for that is that Windows Media DRM is limited to Windows only.

Some membership web-sites just use simple authentication and timestamp URLs. Such URLs expire shortly so there is very little opportunity for them for being watched by non legitimate users.