[tutorial] Using RTMPDUMP to download from BBC iPlayer

[Ed999]

RTMPdump.exe issues a request to the RTMP server for a connection, by sending a command including the requisit elements, thereby initiating the process of negotiating a connection. Such a command may be constructed using connection and authentication information obtained previously from the server by client software applications such as rtmpsrv.exe

In negotiating a connection, an RTMP server sends and receives a data stream containing multiple elements, as a single command sequence. An on-demand stream, using protocol 0 (not tunnelled, not encrypted), typically includes the following elements -


A. For a Limelight server

--protocol : Protocol types include 0 (RTMP), 1 (RTMPT) [Tunnelled], or 3 (RTMPE) [Encrypted]
--host : server address (a.k.a. domain address)
-a : authentication elements (the alternative --app may be used instead)
Typically in the format -
{application}?as={data}&av={data}&te={data}&mp={da ta}&et={data}&fmta-token={data}
application: A path address. For example, a1414/e3
as : For example, as=adobe-hmac-sha256
av : For example, av=1
te : For example, te=connect
mp : Typically, two or more comma-separated URL addresses, for alternative bitrate streams (mpeg format, mp3 or mp4)
et : Typically, a ten-character number (numerical)
fmta-token : Typically, a 64-character authentication (auth) string [i.e. an authentication token] (alphanumeric)
-y : playpath (URL address of the desired bitstream, one of those specified in mp above)
Typically, in the format mp3:{path}/{filename}.mp3 or mp4:{path}/{filename}.mp4
-o : Output filename


The foregoing are typically the only elements (or "switches") that are essential to a connection, if neither Tunnelling nor Encryption are in use by the server. Although other elements may be encountered in practice, they are normally non-essential.

Hence the following elements are typically sent by RTMPdump, as a single command line -

rtmpdump --protocol 0 --host xxxxxxxx.fcod.llnwd.net -a path?as=data&av=data&te=data&mp=data&et=data&fmta-token=data -y mp4:URL/filename.mp4 -o file_mp4.flv


The parts comprising the -a (or --app) element must be incorporated in it in the order shown above, as the sequence in which its parts are received by the RTMP server is critical.

The authentication strings (et= and fmta-token=) contain session information, so will change on each fresh connection made to the server (which in practice typically means they will expire if a new session is begun, not literally on every attempt to resume a connection), but the other elements will not usually vary from session to session.


B. For an Akamai server

The command sequence is typically as above, except that the -a (authentication) string contains the following elements instead -
auth : Typically, a 62-character authentication (auth) string [i.e. an authentication token] (alphanumeric)
aifp : For example, aifp=v001
slist : Typically, the URL address of the stream

Hence the following elements are typically sent by RTMPdump.exe, as a single command line -

rtmpdump --protocol 0 --host xxxxxxx.edgefcs.net -a ondemand?auth=data&aifp=data&slist=data -y mp3:URL/filename -o file_mp3.flv


The parts comprising the -a (or --app) element must be incorporated in it in the order shown above, as the sequence in which its parts are received by the RTMP server is critical.

The authentication string (auth=) contains session information, so will change on each fresh connection made to the server (typically, if a new session is begun, e.g. the computer is restarted, not literally on every attempt to resume a connection). The other elements will not usually vary from session to session.


Note - The above describes the simplified form, whereby the stream is first saved to the user's hard disk, to be played back thereafter in a media player capable of playing an FLV encoded file (H.263 or H.264 encoding), such as GOM Player. If it's desired, instead, to play the stream directly from the RTMP server, thus giving immediate playback, then additional elements will be needed in the command string including -

-f : This specifies the version of the Flash plugin installed on the user's computer. For example, -f "WIN 9,0,260,0" [which indicates the user has the Windows version of Flash Player 9, release 260]

-W : The capital W command. This is the URL address of the SWF player used to play the stream, as indicated by the web page from which the stream is derived. For example -

http://www.bbc.co.uk/emp/9player.swf...on=18269_21576



EXAMPLES from the RADIO iPLAYER

Here are two examples, from the BBC's Radio iPlayer. Both of these are for an on-demand ("Listen Again") file. These examples can't be used with a Live stream.

(NB: In principle these would be similar with a TV show, but in practice are rather more complicated because there can be up to six bitstreams to deal with at each step. Also, a 50 minute TV show will result in a download of up to 800MB as compared with 30MB for a 30 minute radio show.)

First, open the Playlist page (replace the 8 zero's with the 8 digit ID number from the URL of the page that plays the radio show in question):-

http://www.bbc.co.uk/iplayer/playlist/00000000

Second, open the Media Selector page (replace the 8 zero's with the 8 digit Identifier from the Playlist page), as the Media Selector page contains all the information needed below:-

http://www.bbc.co.uk/mediaselector/4/mtis/stream/00000000


A. The highest quality on-demand stream: AAC in MP4A in FLV -

This is the command string suggested by rtmpsrv.exe -

rtmpdump -r "rtmp://bbcmedia.fcod.llnwd.net:1935/a1414/e3?as=adobe-hmac-sha256&av=1&te=connect&mp=bbc7/secure_auth/modem/RBN2_bbc7_-_friday_2300_b00bf9yc_2010_09_10_22_55_13.mp4,bbc7/secure_auth/RBN2_bbc7_-_friday_2300_b00bf9yc_2010_09_10_22_54_46.mp4&et=1284234325&fmta-token=65d90a5b552fdded243f9a43bc2cebd4c9dd134c7b0a8970d6 46ebb8e3cd59e0" -a "a1414/e3?as=adobe-hmac-sha256&av=1&te=connect&mp=bbc7/secure_auth/modem/RBN2_bbc7_-_friday_2300_b00bf9yc_2010_09_10_22_55_13.mp4,bbc7/secure_auth/RBN2_bbc7_-_friday_2300_b00bf9yc_2010_09_10_22_54_46.mp4&et=1284234325&fmta-token=65d90a5b552fdded243f9a43bc2cebd4c9dd134c7b0a8970d6 46ebb8e3cd59e0" -f "WIN 9,0,260,0" -W "http://www.bbc.co.uk/emp/9player.swf?revision=18269_21576" -p "http://www.bbc.co.uk/iplayer/console/b00bf9z0/Old_Harrys_Game_Series_4_Episode_2" -y "mp4:bbc7/secure_auth/RBN2_bbc7_-_friday_2300_b00bf9yc_2010_09_10_22_54_46.mp4" -o RBN2_bbc7_-_friday_2300_b00bf9yc_2010_09_10_22_54_46.mp4


The above format works. But here is the simplified (i.e. more useful) format from my .BAT batch file -

SET mp=bbc7/secure_auth/modem/RBN2_bbc7_-_wednesday_1300_b006v244_2010_08_25_13_00_33.mp4,b bc7/secure_auth/RBN2_bbc7_-_wednesday_1300_b006v244_2010_08_25_13_00_06.mp4

SET et=1282760597

SET fmta-token=7d9c1938f53e65d3a30a0d9e313a917d580ec0055ecfa5b9d4 5bb35dc1aa7593

SET identifier=mp4:bbc7/secure_auth/RBN2_bbc7_-_wednesday_1300_b006v244_2010_08_25_13_00_33.mp4


rtmpdump --protocol 0 --host bbcmedia.fcod.llnwd.net -a a1414/e3?as=adobe-hmac-sha256&av=1&te=connect&mp=%mp%&et=%et%&fmta-token=%fmta-token% -y %identifier% -o output_mp4a.flv


B. The MP3 on-demand stream: MP3 in FLV -

This is the format from my .BAT batch file -

set server=cp48188.edgefcs.net
set auth=daEdwabaccObnascHdZbCb4d7cxcRanc5bf-bmI9Jl-bWG-DnmDCpwoIBvGqzD
set aifp=v001
set slist=secure/bbc7/RBN2_bbc7_-_saturday_1000_b005yg5q_2010_09_11_12_36_00
set identifier=mp3:secure/bbc7/RBN2_bbc7_-_saturday_1000_b005yg5q_2010_09_11_12_36_00

rtmpdump --protocol 0 --host %server% -a ondemand?auth=%auth%&aifp=%aifp%&slist=%slist% -y %identifier% -o output_mp3.flv

If this is too complicated, you can try to use software that allows to download and/or capture RTMP streams like Replay Media Catcher 4, CooJah, Orbit Grab Pro. Note that you can capture RTMP streams in real time only.

[Ed999]

Yes, sure, the procedure I've outlined in this thread is a little bit complicated. But it gives you a second option, if your favourite downloader program hits a snag.

The trouble with get_iplayer is that although it's said to work great for Windows XP, it doesn't work so well for Windows 9x. A lot of the more recent changes in get_iplayer just don't work unless you're running WinXP or later, and as I use Windows 98 (and know many people who do) I wanted to offer a solution that will help them too.

I do include (in my original post in this thread) the string which rtmpsrv returns when interrogating a BBC Limelight server for radio. That command string works! So at the moment rtmpsrv works okay for that type of server. However, it doesn't work at ALL for the BBC's Akamai servers for radio. So for those, you need a different solution.


Actually, my solution (above), using rtmpdump v2.1d, is less complicated than it looks. It boils down to this -

First, open the Playlist page (replace the 8 zero's with the 8-digit number in the URL of the BBC page that plays the iPlayer show):-

http://www.bbc.co.uk/iplayer/playlist/00000000

Second, open the Media Selector page (replace the 8 zero's with the 8 digit Identifier contained in the Playlist page):

http://www.bbc.co.uk/mediaselector/4/mtis/stream/00000000



A. The highest quality on-demand stream: AAC in MP4A in FLV -

SET mp=
SET et=
SET fmta-token=
SET identifier=

rtmpdump --protocol 0 --host bbcmedia.fcod.llnwd.net -a a1414/e3?as=adobe-hmac-sha256&av=1&te=connect&mp=%mp%&et=%et%&fmta-token=%fmta-token% -y %identifier% -o output_mp4a.flv


B. The MP3 on-demand stream: MP3 in FLV -

set server=
set auth=
set aifp=
set slist=
set identifier=

rtmpdump --protocol 0 --host %server% -a ondemand?auth=%auth%&aifp=%aifp%&slist=%slist% -y %identifier% -o output_mp3.flv

[Ed999]

Things you probably need to know -

1. I am running Windows 98. I know absolutely nothing about Windows XP/Vista/7.


2. I'm using rtmpdump v2.1d which is the last version that runs on Windows 98. Changes in later versions of rtmpdump mean that my solutions might only work for you if you run v2.1d (but don't use earlier versions, as they have unresolved bugs, use v2.1d).


3. I have posted about the radio iPlayer, not the tv iPlayer. I haven't had much success with the tv version, because it offers so many bitrate streams. That causes complications. I have done successful downloads from TV, on a handful of occasions. But each time I had to bodge a solution! Haven't got the hang of it yet.

The host address bbcmedia.fcod.llnwd.net is ONLY valid for radio!

The BBC use many different host addresses. You need to look at my above posts and understand how to open the Media Selector page. That page will give you all the vital details such as host address, etc.


4. You can't run rtmpsrv by double-clicking the EXE file! Here's the contents of my batch file -

@ECHO OFF

rem *** Edit the HOSTS file ***

rem Replace line 1 with this:
rem 127.0.0.1 bbcmedia.fcod.llnwd.net

ATTRIB -R -S -H C:\WINDOWS\HOSTS
EDIT C:\WINDOWS\HOSTS
ATTRIB +R +S +H C:\WINDOWS\HOSTS

rem *** Start RTMPSRV ***

:: Start RTMPSRV.EXE and pipe the URLs it displays to a .BAT file
:: to be used with RTMPDUMP to download the iPlayer file

rtmpsrv > RTMPdump_run.bat

After you've run that batch file, with rtmpsrv running, you start the iPlayer page playing. The above batch file saves the output from rtmpsrv for you.

Then, close the iPlayer page. Close rtmpsrv.

Then run my second batch file -

@ECHO OFF

rem *** Edit the HOSTS file ***

rem Restore the normal line 1:
rem 127.0.0.1 localhost

ATTRIB -R -S -H C:\WINDOWS\HOSTS
EDIT C:\WINDOWS\HOSTS
ATTRIB +R +S +H C:\WINDOWS\HOSTS


rem *** Run RTMPDUMP ***

rem ** View the Download commands **
C:\WINDOWS\NOTEPAD.EXE RTMPdump_run.bat

rem ** Delete all but one of the Download commands **
EDIT RTMPdump_run.bat

rem ** Run RTMPDUMP to download the FLV file **
CALL RTMPdump_run.bat

rem ** Rename the downloaded file **
REN *.MP4 *.flv

[Ed999]

Originally Posted by Stream Recorder

Note that you can capture RTMP streams in real time only.

I use an 8MB broadband internet connection. I'm running Windows 98, and using RTMPDUMP.EXE v2.1d (the last version which will run on Windows 9x).

When connecting to the BBC iPlayer, which only uses RTMP (no tunnelling, no encryption), this version of RTMPDUMP typically downloads a 30 minute radio show (34MB) in 3 minutes.

Granted, a 60minute TV show of 800 MB is more of a challenge! But TV downloads can be done on the BBC's 225 MB stream instead, and I've occasionally done those, and it took about 25 minutes.

Granted, the 800 MB version would therefore take about 90 minutes. But it's too sweeping to say you can only get a real-time connection to an RTMP stream with RTMPDUMP.

It entirely depends on the speed of your internet connection. If you max out a 20 MB connection within the UK (i.e. the closest point to the BBC) you'll get an 800 MB download much quicker than I can using 8 MB.

You need to do this outside peak hours in the UK (5pm to 11pm GMT Mon to Fri, and 2pm to 11pm GMT at weekends), so that you avoid the period when the BBC's servers are stressed out. Connection speeds to the BBC iPlayer are very poor in peak hours due to heavy use.

Originally Posted by Ed999

Yes, sure, the procedure I've outlined in this thread is a little bit complicated. But it gives you a second option, if your favourite downloader program hits a snag.

Thank you for all your hard work.

Originally Posted by Ed999

When connecting to the BBC iPlayer, which only uses RTMP (no tunnelling, no encryption), this version of RTMPDUMP typically downloads a 30 minute radio show (34MB) in 3 minutes.

Sorry for the confusion. Downloading of on-demand streams can be done faster than in real-time, capturing is always done in real time (unless the stream is being buffered faster)

[Ed999]

Running RTMPDUMP v2.1d on Windows 98, the following solution will record a BBC Radio live stream (provided you are in the UK).

You do NOT need to install the Macromedia Flash Player in order to use this solution. All you need is RTMPDUMP.exe (for the dumping of the stream to disk), and GOM Player (to play the file). I have, at present, no solution for playing the file whilst recording it: you must play it back from disk once you've finished making the recording.


First, open the Media Selector page (this example is for the BBC Radio 7 live stream) -

http://www.bbc.co.uk/mediaselector/4/mtis/stream/bbc_7


Secondly, create a plain text batch (.BAT) file, using Notepad if you are running Windows. Here is the contents of my successful batch file -

set server=
set auth=
set slist=
set identifier=

rtmpdump --live --protocol 0 --host %server% -a live?auth=%auth%&aifp=v001&slist=%slist% --playpath %identifier%?auth=%auth%&aifp=v001&slist=%slist% -o BBC_live_mp4.flv


Thirdly, insert the text in the lines set server=, set auth=, set slist= and set identifier= from the corresponding parts of the Media Selector page.


Fourthly, if you are outside the UK make sure the set identifier= string (the station address) is the INT (International) version, not the UK version. The International version for Radio 7, for example, is: Radio_7_Int@6463

For International listeners, the set slist= string will also be Radio_7_Int@6463 (because the International broadcast has no lower quality stream, being - sadly - already a low quality stream).


With slight adaptations, the above solution should work for all BBC radio stations. The Media Selector pages for Radio 1, 2, 3, 4, 5live, and 6music contain the equivalent technical information:

Radio 1 -
http://www.bbc.co.uk/mediaselector/4.../bbc_radio_one

Radio 2 -
http://www.bbc.co.uk/mediaselector/4.../bbc_radio_two

Radio 3 -
http://www.bbc.co.uk/mediaselector/4...bc_radio_three

Radio 4 -
http://www.bbc.co.uk/mediaselector/4...c_radio_fourfm

Radio 5live -
http://www.bbc.co.uk/mediaselector/4...adio_five_live

Radio 6music -
http://www.bbc.co.uk/mediaselector/4...eam/bbc_6music

Radio 7 -
http://www.bbc.co.uk/mediaselector/4/mtis/stream/bbc_7


Notes :

1. This is a live stream: there is no means of resuming a live stream!

2. Most BBC radio shows can be recorded from the Listen Again pages (e.g. http://www.bbc.co.uk/radio7/programmes/schedules) for 7 days following the broadcast. But the need to record the live stream can arise, in those cases (now, thankfully, fairly few) where the show is not likely to be allocated a Listen Again link.

Live broadcasts, mainly, are not given Listen Again status (e.g. music concerts on Radio 1 or on 6Music, or sports broadcasts on Radio 5).

[Ed999]

Playing a Live Stream while simultaneously saving it

Here's my solution for playing a live stream whilst rtmpdump is saving it to disk, or for playing an on-demand stream whilst rtmpdump is downloading it, using Windows 98.

Note - The following solution will work for ANY download of a radio or television show (even non-BBC shows), using ANY download method (not just rtmpdump). Even if you use an automated downloader such as get_iplayer or iplayer_dl the following procedure will play the file while it's still downloading - a godsend for live streams!)

This uses the program FFPLAY.EXE (which is included in standard distributions of ffmpeg). You run it in a seperate (i.e. second) DOS window at the same time that rtmpdump is downloading the rtmp stream in a DOS window.

(NB: A DOS window is called a "command window" in WinXP, as Micro$oft are pretending that MS-DOS is not included in the operating system.)

Here's a version of ffmpeg which will run under Windows 98 (SVN-r16573) (13 Jan 2009): http://ffmpeg.arrozcru.org/.../ffmpeg-r16537-gpl-shared-win32.tar.bz2

(Note: The latest ffmpeg version which runs on Win9x is release r21874)

An important point is that the batch file must be in the same folder (directory) into which rtmpdump is saving the rtmp stream.

Another important point is that both RTMPDUMP and FFPLAY must be set to use .FLV as the file type, because the batch file (see below) sets FFPLAY to play any file with that file type in the current directory.


1. Playing a Radio show while downloading it

This is the content of my successful batch (.BAT) file for a radio show (using the option -vn to specify no video, as this is a radio stream) -

:: Syntax
:: ffplay [options] 'input_file'

:: Play radio stream while it's downloading
"C:\Program Files\ffmpeg\ffplay.exe" -vn *.flv

(Note: This works for both a Live stream and an On-demand stream)


Here's an alternative command line -

Play a Radio stream as it Downloads: With Buffer, No Display

"C:\Program Files\ffmpeg\ffplay.exe" -vn -nodisp -bufsize 4096 *.flv


2. Playing a TV show while downloading it

I then tried this procedure on a television stream, successfully. I was astonished at how the crappy performance of Flash Player on Windows 98 was magically replaced by a perfect video stream. The TV stream (an on-demand tv show) played perfectly, with no hesitating or dropped frames.

I used only one trick! I didn't start the show playing in FFPLAY until rtmpdump had been downloading the show for 15 seconds, thus building-in a 15 second buffer.

Note: I had set rtmpdump to save the TV show as a file with the extension .mp4 on this occasion. If you save a show as a .flv file, remember to alter the command line (below) from *.mp4 to *.flv instead.

It was a revelation, using the following command line in the above batch file instead of the radio command -

Play on-demand TV show with 4MB Buffer

"C:\Program Files\ffmpeg\ffplay.exe" -bufsize 4096 *.mp4