secure RTMPE protocol. Is it really secure???

[any ANONYMOUS forum user]

RTMPE
-----
RTMPE is an extension to RTMP to include encryption of content. Adobe uses industry standard crypto primitives for RTMPE although it calls this protocol "proprietary".

This document is a clean-room specification of the RTMP "Encryption" scheme called RTMPE. It contains industry-standard crypto primitives, ARC4, HMACSHA256 and Diffie-Hellman. The specification was created by reviewing the source code of rtmpdump v1.6.

Academic and other discussion is invited. Distribution of this document is unlimited and encouraged. Implementations even more so.

More info: http://lkcl.net/rtmp
Revisions: http://rab.zapto.org/RTMPE/
RTMPE recording: stream-recorder.com: RTMPE stream recording/downloading/capturing/ripping/saving

Conventions
-----------

data[x:y] means "bytes x through y, inclusive" - like in python
x+y on bytes means "append the two byte streams, consecutively"
data[x] means "the byte offset by x" - like in python.
/* ... */ means comments
bigendian32(x) means create 4 bytes in big-endian order, from a 32-bit integer.


Constants
---------

RTMP_SIG_SIZE = 1536
SHA256DL = 32 /* SHA 256-byte Digest Length */

RandomCrud = {
    0xf0, 0xee, 0xc2, 0x4a,
    0x80, 0x68, 0xbe, 0xe8, 0x2e, 0x00, 0xd0, 0xd1,
    0x02, 0x9e, 0x7e, 0x57, 0x6e, 0xec, 0x5d, 0x2d,
    0x29, 0x80, 0x6f, 0xab, 0x93, 0xb8, 0xe6, 0x36,
    0xcf, 0xeb, 0x31, 0xae
}

SWFVerifySig = { 0x1, 0x1 }

/* data in quotes does not include quotes as part of data */
GenuineFMSKey = "Genuine Adobe Flash Media Server 001" 
GenuineFPKey = "Genuine Adobe Flash Player 001" 

GenuineFMSKeyCrud = GenuineFMSKey + RandomCrud
GenuineFPKeyCrud = GenuineFPKey + RandomCrud

GetServerDHOffset
-----------------

The purpose of this function is to calculate the offset of the Server's Diffie-Hellmann key.

Its input is 4 consecutive bytes.

    offset = byte[0] + byte[1] + byte[2] + byte[3]
    offset = modulo(offset,632)
    offset = offset + 8

For sanity, the offset should be no bigger than (767-128)

GetServerGenuineFMSKeyDigestOffset
----------------------------------

The purpose of this function is to calculate the offset of the Server's Digest.

Input data is 4 consecutive bytes.

offset = byte[0] + byte[1] + byte[2] + byte[3]
    offset = modulo(offset,728)
    offset = offset + 776

For sanity, the offset should be no bigger than (1535-32)

GetClientDHOffset
-----------------

The purpose of this function is to calculate the offset of the client's Diffie-Hellmann key.

Input data is 4 consecutive bytes.

offset = byte[0] + byte[1] + byte[2] + byte[3]
    offset = modulo(offset,632)
    offset = offset + 772

For sanity, the offset should be no bigger than (RTMP_SIG_SIZE-128-4)

GetClientGenuineFPKeyDigestOffset
---------------------------------

The purpose of this function is to calculate the offset of the client's Digest.

Input data is 4 consecutive bytes.

offset = byte[0] + byte[1] + byte[2] + byte[3]
    offset = modulo(offset,728)
    offset = offset + 12

For sanity, the offset should be no bigger than (771-32)


Packet Format
-------------

The packets consist of a one byte command followed by a 1536 byte message

    Bytes    : Description
    -------    -----------
    0          Command
    1:1536     message of RTMP_SIG_SIZE bytes

Client First Exchange
---------------------

This is the first packet to be generated.
clientsig and clientsig2 are RTMP_SIG_SIZE bytes.
serversig and serversig2 are RTMP_SIG_SIZE bytes.

Note: Encryption is only supported on versions at least 9.0.115.0

Note: The 0x08 command-byte is not yet known. It is understood to involve further obfuscation of the Client and Server Digests,
and is understood to be implemented in Flash 10.

Command byte:

    0x06 if encrypted
    0x08 if further encrypted (undocumented)
    0x03 if unencrypted

Message:

    0:3        32-bit system time, network byte ordered (htonl)
    4:7        Client Version.  e.g. 0x09 0x0 0x7c 0x2 is 9.0.124.2
    8:11       Obfuscated pointer to "Genuine FP" key 
    12:1531    Random Data, 128-bit Diffie-Hellmann key and "Genuine FP" key.
    1532:1535  Obfuscated pointer to 128-bit Diffie-Hellmann key

Calculate location of Diffie Hellmann Public Key and create it:

dhpkl = GetClientDHoffset(clientsig[1532:1535])
    DHPrivateKeyC, DHPublicKeyC = DHKeyGenerate(128) /* 128-bit */
    clientsig[dhpkl:dhpkl+127] = DHPublicKeyC

Calculate location of Client Digest and create it:

/* Note: the SHA digest message is calculated from the bytes of
      the message, excluding the 32-bytes where the digest itself goes.
    */

    cdl = GetClientGenuineFPKeyDigestOffset(clientsig[8:11])
    msg = clientsig[0:cdl-1] + clientsig[cdl+SHA256DL:RTMP_SIG_SIZE-1]
    clientsig[cdl:cdl+SHA256DL-1] = HMACsha256(msg, GenuineFPKey)

First Exchange:

Send all 1537 bytes (command + clientsig) to the server;
    Read 1537 bytes (command + serversig) from the server.

Note that the exact circumstances under which "Message Format 1"
or "Message Format 2" are utilised is unknown. It is therefore necessary for clients to utilise the SHA verification to determine which of the two message formats is being received (!)

Command byte:

    0x06 if encrypted - same as client request
    0x03 if unencrypted - same as client request

Message Format 1:

    0:3        32-bit system time, network byte ordered (htonl)
    4:7        Server Version.  e.g. 0x09 0x0 0x7c 0x2 is 9.0.124.2
    8:11       Obfuscated pointer to "Genuine FMS" key 
    12:1531    Random Data, 128-bit Diffie-Hellmann key and "Genuine FMS" key.
    1532:1535  Obfuscated pointer to 128-bit Diffie-Hellmann key

Calculate location of Server Digest and compare it:

    sdl = GetClientGenuineFMSKeyDigestOffset(serversig[8:11])
    msg = serversig[0:sdl-1] + serversig[sdl+SHA256DL:RTMP_SIG_SIZE-1]
    Compare(serversig[sdl:sdl+SHA256DL-1], HMACsha256(msg, GenuineFMSKey))

Calculate location of Server Diffie Hellmann Public Key and get it:

   dhpkl = GetClientDHoffset(serversig[1532:1535])
    DHPublicKeyS = serversig[dhpkl:dhpkl+127]

Message Format 2:

    0:3        32-bit system time, network byte ordered (htonl)
    4:7        Server Version.  e.g. 0x09 0x0 0x7c 0x2 is 9.0.124.2
    8:767      Random Data and 128-bit Diffie-Hellmann key 
    768:771    Obfuscated pointer to 128-bit Diffie-Hellmann key 
    772:775    Obfuscated pointer to "Genuine FMS" key 
    776:1535   Random Data and "Genuine FMS" key.

Calculate location of Server Digest and compare it:

    sdl = GetServerGenuineFMSKeyDigestOffset(serversig[772:775])
    msg = serversig[0:sdl-1] + serversig[sdl+SHA256DL:RTMP_SIG_SIZE-1]
    Compare(serversig[sdl:sdl+SHA256DL-1], HMACsha256(msg, GenuineFMSKey))

Calculate location of Server Diffie Hellmann Public Key and get it:

    dhpkl = GetServerDHoffset(serversig[768:771])
    DHPublicKeyS = serversig[dhpkl:dhpkl+127]

Compute Diffie-Hellmann Shared Secret:

The key is only needed if encryption was negotiated.

    DHSharedSecret = DH(DHPrivateKeyC, DHPublicKeyS)