FMS live AV chat architecture

[eugene!]

Hello,
*
We have live AV chat app which assumes to have up to 50 online rooms with 500-600 people in each room. Our idea is only one people can broadcast video in the room, others will be able just see it.
*
We have quite strong server with 1Gbit/sec bandwidth. If we will get that average stream is 300kbit/sec, it means that we can support ~3,5k connections (1Gbit * 1024 * 1024 / 300 kbit). Is it correct? But actually, when we have more than 500 connections on server, time to time we get connection rejects. Any ideas?
*
Also, CPU and RAM usage is very low and I believe that server has enough resources to process more connections...
*
Another problem is when into one chat room we have more 100 people, video begins to slow down and twitch. What it may be and what should we do to avoid such behavior?
*
We use FMS 4.0 server and Centos 6 64bit architecture.
We tried to use origin/edge architecture, but it seems it quite useless with live broadcasting...
*
Please advice which architecture is the best for using such type of applications and what we must tweak in order to enhance our performance.
*
Thanks