Ok so I've found what is going on.
It seems that the current librtmp code fails to compute the operation:
Code:
sendSecureTokenResponse(TEA.decrypt("secureToken", "token"));
For example tampering the swf file:
Code:
TEA.decrypt("5b9389e825ebcca55cf84a71ddc5a13c6ae4f5ce19cd3fdbf91e1a53b3029f383461c800", "#atd%#$ZH")
Gives:
ed08d934-41ac-4b90-80a2-d0cacd923ae1
Where librtmp doing the same operation gives:
$]m.i.eo.I#...z._hB.`
Hence the server rejects the connection