theplatform.com: RTMP stream works with all GUI rippers, not with RTMPDump, confused.

[Confused User]

I am trying to rip a stream. Actually, a whole bunch of streams, but I'll just provide one as an example;

The Stream URL is inside this XML file: "http://release.theplatform.com/content.select?pid=2vaZNBb_CxCP9V5QODRKpTL2vZqvVz_ _"

The streaming server seems to use some sort of time or session based authentication. Every time you refresh that XML file, the URL of the RTMP stream is slightly different.

Nevertheless, if I copy the RTMP stream URL out and paste it in any of the major GUI-based RTMP rippers, they work. I just successfully ripped a number of streams using NetTransport, it works with Replay Media Catcher, etc.

However, I'd like to use RTMPDump, both because it's free and open source, and because since it's a command line utility it'd be very easy to batch automate stuff with it.

I believe the issue is related to the unusual format of the URL; like I said above, this URL is going to expire shortly so you'll have to visit the XML file to get a new, valid URL, but here's the URL from this particular time.

rtmp://cp68811.edgefcs.net/ondemand/?auth=dbEaQdrcOauckcXdCb_b.aqaedta9awaUaj-blUJDK-4q-d9i-3qsFAnKpwE6P9&aifp=1234&slist=Canwest_Broadcast_En tertainment/<break>Canwest_Broadcast_Entertainment/Food_Glutton_S4_Ep4048.flv

What I notice that's strange is the <break> component. That's very odd for any file system. I tried various ways of switching up the URL (rtmpdump -r url, rtmpdump -r url -y Canwest_Broadcast_Entertainment/Food_Glutton_S4_Ep4048.flv, etc)

If anyone could visit the XML file, get a valid URL, and successfully figure out a combination of switches necessary to get this working with RTMPDump, that'd be great.

Thanks.

An RTMP URL looks like:
rtmp://server/app/playpath

If you have no idea how to extract the server, app and playpath components, you can find them with freeware CooJah.

[Confused User]

I installed CooJah and ran it on the initial page.

Again, keep in mind that the RTMP url uses some sort of time-sensitive auth so anything I copy/paste will be expired by the time you read it;

Here's what I've got:

url: rtmp://cp68811.edgefcs.net:1935/ondemand/?auth=dbEcAb.dgcpd2aCcvdEc5dWdLdAcpbCc_am-blUY8y-4q-d9i-7onGGpPqAC5O2&aifp=1234&slist=Canwest_Broadcast_Entertainment/Canwest_Broadcast_Entertainment/Food_Glutton_S4_Ep4052
app: ondemand/?auth=dbEcAb.dgcpd2aCcvdEc5dWdLdAcpbCc_am-blUY8y-4q-d9i-7onGGpPqAC5O2&aifp=1234&slist=Canwest_Broadcast_Entertainment/
flashVer: WIN 10,0,32,18
pageUrl: http://www.foodnetwork.ca/video/index.html
swfUrl: http://webdata.globaltv.com/global/canwestPlayer/swf/4.1/flvPlayer.swf
tcUrl: rtmp://cp68811.edgefcs.net:1935/ondemand/?auth=dbEcAb.dgcpd2aCcvdEc5dWdLdAcpbCc_am-blUY8y-4q-d9i-7onGGpPqAC5O2&aifp=1234&slist=Canwest_Broadcast_Entertainment/
playPath: Canwest_Broadcast_Entertainment/Food_Glutton_S4_Ep4052

I can verify that I can download this through Blader. Now I just need to figure out how to transfer this information to RTMPdump.

Now, as far as I can tell, assuming I need 100% of this information, rtmpdump should be called as such:
rtmpdump -r url -a app -p pageUrl -f flashVer -s swfUrl -t tcUrl -y playPath
(obviously, substituting in all of the information above for the variables.)

Here's a direct dump of the results I get running from the command line:

rtmpdump -r "rtmp://cp68811.edgefcs.net:1935/ondemand/?auth=dbEcAb.dgcpd2aCcvdEc5dWdLdAc
pbCc_am-blUY8y-4q-d9i-7onGGpPqAC5O2&aifp=1234&slist=Canwest_Broadcast_Entertainm
ent/Canwest_Broadcast_Entertainment/Food_Glutton_S4_Ep4052" -a "ondemand/?auth=d
bEcAb.dgcpd2aCcvdEc5dWdLdAcpbCc_am-blUY8y-4q-d9i-7onGGpPqAC5O2&aifp=1234&slist=C
anwest_Broadcast_Entertainment/" -p "http://www.foodnetwork.ca/video/index.html"
 -f "WIN 10,0,32,18" -s "http://webdata.globaltv.com/global/canwestPlayer/swf/4.
1/flvPlayer.swf" -t "rtmp://cp68811.edgefcs.net:1935/ondemand/?auth=dbEcAb.dgcpd
2aCcvdEc5dWdLdAcpbCc_am-blUY8y-4q-d9i-7onGGpPqAC5O2&aifp=1234&slist=Canwest_Broa
dcast_Entertainment/" -y "Canwest_Broadcast_Entertainment/Food_Glutton_S4_Ep4052
"
RTMPDump v2.2b
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
Starting download at: -0.001 kB
Metadata:
  duration              1296.32
  width                 640.00
  height                360.00
  framerate             25.00
  canSeekToEnd          TRUE
  videocodecid          4.00
  videodatarate         480.00
  audiocodecid          2.00
  audiosamplerate       44100.00
  audiosamplesize       16.00
ERROR: Download: Failed writing, exiting!

It clearly detects the duration, width, height, framerate, etc. Much better than it was doing before, which is great. But it still fails. When I throw a -z (to get debug info), it reads what seems like hundreds of packets and then dies. I'll include the last packet or so:

DEBUG2: RTMP_ReadPacket: fd=380
  0000:  c5                                                 .
  0000:  62 80 38 8e 49 ed d1 a9  97 06 69 e7 f5 51 1a e4   b.8.I.....i..Q..
  0010:  8f cb af c5 69 c7 ae f0  76 2f 19 c1 c4 22 c9 48   ....i...v/...".H
  0020:  62 cd 88 1f d9 2e 9d 64  33 76 4e 68 60 4c 2b 81   b......d3vNh`L+.
  0030:  eb 3c f7 32 53 95 8c da  be a1 41 29 94 85 c5 8d   .<.2S.....A)....
  0040:  26 4c da 90 8c 72 bc 06  d8 e2 4f 63 9b e7 ce 88   &L...r....Oc....
  0050:  c2 97 b1 4e 5d fe 6a d2  76 e0 6f 45 95 7d 06 da   ...N].j.v.oE.}..
  0060:  e4 00 9b 4c 4c 7d 4d 00  06 95 f7 df f1 9b e1 32   ...LL}M........2
  0070:  58 a8 4b 40 62 32 75 79  a0 98 7c 07 cb ec f9 b5   X.K@b2uy..|.....
  0080:  d3 00 df ca cd 60 46 46  a4 31 e0 c2 ee 38 7b d6   .....`FF.1...8{.
  0090:  5a 88 5e c7 84 c2 3d 13  19 cf bd 8a af da 3d a3   Z.^...=.......=.
  00a0:  7a ea f9 71 00 ed b5 c0  9f bb 20 61 60 7e c9 01   z..q...... a`~..
  00b0:  8b 14 04 86 52 73 5f e0  da e7 d2 66 a8 f7 7c 98   ....Rs_....f..|.
  00c0:  75 b6 b6 33 34 da 6d 84  9f 38 e8 48 1c d0 ef c1   u..34.m..8.H....
  00d0:  2b 27 f7 1f 7c b2 04 c7  ce 47 f8 59 63 4b 48 44   +'..|....G.YcKHD
  00e0:  e3 d3 c8 d1 79 ac 43 47  37 f0 44 ba 88 f6 6a 97   ....y.CG7.D...j.
  00f0:  38 a6 5b 50 28 3b c7 e8  05 43 be 53 40 5f 11 b1   8.[P(;...C.S@_..
  0100:  6a 1f da ac 57 1f 4f 28  f6 54 aa 13 ff 82 fe f7   j...W.O(.T......
  0110:  c3 6f f5 02 bb 46 44 3e  41 55 9a 77 76 95 4c ec   .o...FD>AU.wv.L.
  0120:  da 3e 66 81 ff b5 09 a2  09 37 86 bd 0b b5 1d 14   .>f......7......
  0130:  cf de 0e c7 45 d7 c2 fc  dd 92 55 f9 16 c4 67 38   ....E.....U...g8
  0140:  ae b7 7f de 51 4d a6 c7  d3 fb 2b 81 79 84 e3 d3   ....QM....+.y...
  0150:  3d 24 fd 11 22 32 d3 ca  dc da c1 bc d2 56 ca a6   =$.."2.......V..
  0160:  5d 53 c1 35 da 43 33 96  29 85 e9 3d 65 a6 0e fd   ]S.5.C3.)..=e...
  0170:  4e f5 ce 30 84 4d f5 c7  e6 5b 4e bd 4b 34 6d 2c   N..0.M...[N.K4m,
  0180:  e2 65 93 60 72 ff 03 57  58 c6 03 fb e9 52 c9 25   .e.`r..WX....R.%
  0190:  01 19 e8 7b c9 18 d7 3c  b0 69 3d 9e e8 19 c3 dd   ...{...<.i=.....
  01a0:  df 71 d0 73 92 9e a5 ff  9a 3e c4 5e 66 fc 04 11   .q.s.....>.^f...
  01b0:  02 de 69 4b 29 77 a7 e4  b9 3a 0e 62 e3 4e 27 91   ..iK)w...:.b.N'.
  01c0:  f1 29 07 8b 5f 45 a6 37  26 c1 95 94 44 ce 58 19   .).._E.7&...D.X.
  01d0:  88 47 1e 8b bb f7 af f5  71 c5 33 ea 28 fd 24 86   .G......q.3.(.$.
  01e0:  0b 6e 2e 4d da 3c 68 71  a7 5f 29 5f aa 35 67 11   .n.M.<hq._)_.5g.
  01f0:  45 1b de a7 d5 9f 60 3e  73 57 d3 b9 ba bd a3 d0   E.....`>sW......
  0200:  e4 c1 1b 91 fc ce bd 9e  a4 dc 72 b2 c5 33 56 c7   ..........r..3V.
  0210:  cf 20 22 9d 6c fc 28 15  ab cb d7 73 2e e2 3a 37   . ".l.(....s..:7
  0220:  3d 2c e9 db 68 41 9b 63  6f 07 16 f4 1f 46 24 ea   =,..hA.co....F$.
  0230:  c0 2e a2 85 6e 35 93 ed  f2 57 a3 5d 72 44 19 78   ....n5...W.]rD.x
  0240:  a7 9c f3 92 0a 08 42 e9  df ef 13 a0 94 0c e5 03   ......B.........
  0250:  b7 ee 23 fb c7 24 3a ad  c9 e4 46 a3 ba 01 4c c3   ..#..$:...F...L.
  0260:  c8 43 53 cc 32 28 21 b2  74 7a 17 55 f2 a8 7f b3   .CS.2(!.tz.U....
  0270:  3b 88 2d 24 e1 88 73 55  a3 21 5c 19 2c 77 19 c4   ;.-$..sU.!\.,w..
  0280:  b8 00 00 00 00 00                                  ......
ERROR: Download: Failed writing, exiting!
DEBUG: Closing connection.

DEBUG2: RTMP_SendPacket: fd=380, size=34
  0000:  43 00 00 00 00 00 22 14                            C.....".
  0000:  02 00 0c 64 65 6c 65 74  65 53 74 72 65 61 6d 00   ...deleteStream.
  0010:  40 14 00 00 00 00 00 00  05 00 3f f0 00 00 00 00   @.........?.....
  0020:  00 00                                              ..
DEBUG: Invoking deleteStream

Instead of throwing a -z, I'll throw a -v to try to get verbosity; If I do that, it seems to just hang:

RTMPDump v2.2b
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
ERROR: rtmp server sent error
Starting Live Stream

... and it sits there until I ctrl+c the process.

Any further advice? Like I said, downloading the stream clearly works in every GUI tool, I'd just really like to get it working in RTMPdump so I can write external scripts to batch automate this process for me.

Originally Posted by Confused User

ERROR: Download: Failed writing, exiting!

-o parameter?

-W - for SWF URL (if it is required)

[Confused User]

The issue ended up being not explicitly setting an output file and the download worked fine, thanks for the help.

In fact, I was able to remove the -W, -p, and -f switches. It works fine as -r -a -t -y -o.

But on the other hand, I don't think it was a case of "not reading what was in front of me". I really have no idea why I needed to explicitly set an output file. Without -o, the functionality should cause the stream data to be "written to stdout." according to the manual. There's nothing in the manual or elsewhere suggesting that this would trigger an error state, and the error message doesn't provide any useful diagnostic information that would suggest why it would fail one way and not the other.

Obviously I can still accomplish the same thing writing to a file and feeding that file into any postprocessing rather than dealing directly with the data being pumped into STDOUT, but again there's no indication that I should have to do it that way.

[dooglus]

I found I only have to specify --host, --app, and --playpath to download Canwest_Broadcast_Entertainment stuff.

Here's a shell script that fetches the URL including the auth token, splits it up and runs rtmpdump:

#!/bin/sh

pid='C43FtRVCkqcF6nDGMPGudYYK1sa_giDI' # Cesar Millan

start="http://release.theplatform.com/content.select?pid=$pid&UserName=Unknown&Site=global_prg.com&k4=pr&Tracking=True&v4=globalnews&k7=page&v5=programs&v6=16x9&k2=kw&k0=kw&Embedded=True&k8=cnt&Portal=GlobalTV&k3=nk&v3=cbrdcst&k1=kw&v0=ron&v1=programs&v8=news&k6=sck&v2=16x9&TrackBrowser=True&v7=video&k5=ck&TrackLocation=True&k9=cnt&v9=social_issues&Zone=16x9thebiggerpicture&show=16x9_the_bigger_picture&episode=&network=&season=&clipLength=long&format=SMIL"

url=$(wget -o/dev/null -O- "$start" | grep '[.]flv' | cut -d'"' -f2)

host="$(echo "$url" | cut -d/ -f3)"
app="$(echo "$url" | cut -d/ -f4-5 | sed 's/&/\&/g')"
playpath="$(echo "$url" | cut -d/ -f6- | sed -e 's/<break>//' -e 's/[.]flv$//')"
file="$(echo "$playpath" | cut -d/ -f2).flv"

rtmpdump --host "$host" --port 1935 --app "$app/" --playpath "$playpath" -o "$file"

It gets this URL:

rtmp://cp68811.edgefcs.net/ondemand/?auth=dbEd6ddbdc8aTcOblddb1dYdnaRata3aGby-bm9n.u-4q-d9i-9qoEDpKnBEXN2&aifp=1234&slist=Canwest_Broadcast_Entertainment/<break>Canwest_Broadcast_Entertainment/16X9Nov29-1a.flv

breaks it up as follows:

host: cp68811.edgefcs.net
app: ondemand/?auth=dbEd6ddbdc8aTcOblddb1dYdnaRata3aGby-bm9n.u-4q-d9i-9qoEDpKnBEXN2&aifp=1234&slist=Canwest_Broadcast_Entertainment
playpath: Canwest_Broadcast_Entertainment/16X9Nov29-1a
file: 16X9Nov29-1a.flv2

and ends up running this command:

rtmpdump --host cp68811.edgefcs.net --port 1935 --app ondemand/?auth=dbEd6ddbdc8aTcOblddb1dYdnaRata3aGby-bm9n.u-4q-d9i-9qoEDpKnBEXN2&aifp=1234&slist=Canwest_Broadcast_Entertainment/ --playpath Canwest_Broadcast_Entertainment/16X9Nov29-1a -o 16X9Nov29-1a.flv2