You need to be sure you open the right kind of ports. For RTP the ports are UDP. So be sure you are opening UDP ports:
ec2-authorize default -P udp -p 10000-10003
QuickTime Broadcaster can either use RTSP/RTP (Automatic Unicast Announce) or native RTP with an SDP file (Manual Unicast). So if you want to simulate what VLC is doing you want manual unicast. You can export an SDP file from the File->Export menu.
Charlie