[nwtim]

Back in 2012, I got a number of extremely high quality videos of the London 2012 Olympics from the NBCOlympics site using ffmpeg to input the stream from a URL I got from Firefox's Web Console using the .m3u8 filter. In trying to do something similar this year for the Winter Olympics, I'm able to play the videos in Firefox, but I've had no luck coming up with a URL to try to grab.

Instead, the best I'd probably be able to do would be to get the 5-second long snippets that tick by and merge them together.

Is anybody successfully doing this or can help me find the URL that addresses the whole video stream?

Thanks!

[x-9er]

NBC used YouTube to stream two years ago. They're using AdobeHDS now. You need to acquire every .fdf file for a stream then stitch them together using the php script.

I have yet to actually make this work; my videos end up only consisting of the keyframes.

[nwtim]

NBC used YouTube to stream two years ago. They're using AdobeHDS now. You need to acquire every .fdf file for a stream then stitch them together using the php script.

I have yet to actually make this work; my videos end up only consisting of the keyframes.Oh yes, I remember there being problems with Adobe HDS for people trying to capture BBC streams of the 2012 Olympics. Adobe HDS segments are variable bit rate which makes audio re-alignment tricky. What I think I recall regarding the Adobe HDS streams from BBC is that HDS was just a default stream. If I remember correctly, any problem loading the HDS links resulted in a redirect to rtmpe streams, which presented fewer problems. I don't know if that's the way NBC is doing it, though, because I haven't had it fail to load the HDS links.

Any thoughts on the possibility of finding rtmpe streams?

[dave5654656]

any luck on this?

I've been trying to and I have the same issue...

i.e., run the adobeHDS script against the f4m manifest file and it just assembles a flash video file (i.e,. flv) that only has the keyframes seemingly

perhaps there's some extra parameter to pass/tweak in order for it to request all of the frames instead of just keyframes?

[Stream Ripper]

Your best bet will be a Screen Capture (http://all-streaming-media.com/record-video-from-screen/) tool such as Replay Video Capture (http://applian.com/replay-video-capture/)

[nwtim]

Your best bet will be a Screen Capture (http://all-streaming-media.com/record-video-from-screen/) tool such as Replay Video Capture (http://applian.com/replay-video-capture/)I have a couple screen capturing programs that I've tried this with and the resulting file is always jerky and the filesizes are very large. Joining the actual files used for the stream saves significant disk space, and also addresses the framerate match issue.

[nwtim]

any luck on this?

I've been trying to and I have the same issue...

i.e., run the adobeHDS script against the f4m manifest file and it just assembles a flash video file (i.e,. flv) that only has the keyframes seemingly

perhaps there's some extra parameter to pass/tweak in order for it to request all of the frames instead of just keyframes?Using the Firefox add-on DownloadThemAll, I've been able to grab a long sequence of Seg1-FragXXX files, but I'm not certain how to join them with AdobeHDS.

In case it helps, the files I have are named as follows:

007_Fragments(video,format=f4m-f4f)Seg1-Frag762
...
007_Fragments(video,format=f4m-f4f)Seg1-Frag1000

Of course, I can rename them to anything I want to make joining simpler, so I just need to know what tool and syntax to use to join them.

Can someone point me in the right direction for this? Thanks!

[ehurdler]

How do you get downthemall to capture the segment files? I have been trying to figure out how to capture events from nbcolympics.com all week. Stumbled upon using downthemall, but cant even figure out how to get the seg*_frag* files to download with it.

[nwtim]

How do you get downthemall to capture the segment files? I have been trying to figure out how to capture events from nbcolympics.com all week. Stumbled upon using downthemall, but cant even figure out how to get the seg*_frag* files to download with it.There are probably a number of ways to do it, but I used the Copy URL option on several of the files linked via VideoDownloadHelper. I copied these URLs to a text window and found the links for the quality I was looking for and some of the file fragment numbers. Next, I opened the DownloadThemAll Manager window (Firefox->Tools->DownThemAll! Tools->Manager) and clicked the (+) button. The resulting Download: text window was populated with the text I had happened to put in my clipboard (which ended in "Seg1-Frag762") and I just changed the 762 to [762:2000], the range I wanted to capture. I clicked on Start! and then "Download Batch" and voila!, the files proceded to be saved.

I'm not sure it's going to do any good, though, because I can't seem to join them together to play.

[ehurdler]

Thank you for replying. I did get AdobeHDS.php to actually download all the frames, stitches them together. Problem now is, it gives the full length of what I was trying to capture. But there is first no Audio, second the timing of all the keyframes/fragments is all fast. It is like you fast forward the entire Hockey game in less than 2 mins. Any ideas?

The line of code i used was:

php AdobeHDS.php --quality high --manifest "http://olystreameast.nbcolympics.com/vod/26f4794c-d9e6-4502-9b1e-88832b876db6/men-s-hockey-quarterfinals0218090107.ism/manifest(format=f4m-f4f,filtername=vodcut).f4m?token=st=1392869889~exp =1392870009~acl=/vod/26f4794c-d9e6-4502-9b1e-88832b876db6/men-s-hockey-quarterfinals0218090107.ism/*~hmac=16779d8b54b9472f43f3ccba073edd0069840955f3c c1cb4f8a805731c1eb3db" --delete


I am using a Macintosh, running this script in the Terminal. You could give it a try whether you are using a Mac or Windows.

php AdobeHDS.php --quality high --manifest "" --delete


Just paste your manifest url within the quotes and run it as soon as you can before it expires. You could also omit the --delete option if you want to keep the fragments saved.

I am at a loss on how to cature the audio and have the timing between frames set properly. If you or anyone has any ideas, please let me know.

[nwtim]

You got a lot further than I did. I haven't even been able to identify the proper manifest URL. I'm using an older version of Firefox (17.0.5) for an add-on (CacheViewer Fx21) I frequently use that doesn't work in newer versions, so I can't use KSV's HDS Link Detector add-on and I can't seem to parse it out using Web Console or Coojah.

I understand what you're describing regarding the improper playback and audio problems. I've had this happen when audio and video streams don't mux properly, so it must have something to do with the way the fragments are being joined together. Have you tried grabbing a lower quality stream to see if the problem is there as well?

[ehurdler]

NWTIM:
As for your version of Firefox I am unsure any difference than the one I am using. But just open up Tools->Web Developer->Web Console.

Once Web Console is open, on the tabs across the top of the console turn everything off but Net. In the filter/search box on the right of the console search for Manifest. I usually open the console on the page I am viewing, do the search, then refresh the page just to make sure I see everything I am searching for the start of the page loading. Give that a try, try my method I described earlier, see if you get same result. I am trying to capture the USA vs CZE Hockey game from today.

[nwtim]

I got a good manifest link for an NBCOlympics video, but when I try to run it (I basically just copied it into your string and added a --duration 60 switch) I get a "Could not open input file: AdobeHDS.php" error. I'm wondering if I'm just missing an --auth switch string or something.

Do you know anything about that?

[ehurdler]

Can I see your code you entered in?

[nwtim]

Can I see your code you entered in?Sure, I just started one of the most recent files on the NBCOlympics|Videos page and parsed out the manifest link.

php AdobeHDS.php --quality high --duration 60 --manifest "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/manifest(format=f4m-f4f,filtername=vodcut).f4m?token=st=1392875416~exp =1392875536~acl=/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/*~hmac=89fab799b8075f745646451f1c89617a1c9045c3fa9 6923a90cff00dd02ca13d" --delete

It's probably expired by now, or perhaps it's IP address dependent.

[ehurdler]

I was able to get downthemall to download the fragments. Or at least I think I got everything. I got a bunch of files what I believe should be audio, all in sequence named "Fragments(audio_en_0,format=f4m-f4f)Seg1-Frag735", etc, etc.

Also got what I would think is video, all in sequence named "Fragments(video,format=f4m-f4f)Seg1-Frag1", etc, etc.

What I guess I figured out, I had been only downloading the keyframe files ONLY when using the AdobeHDS.php script. I dont know what to do with those keyframe files. It does stitch them together, and I get what I described as a video that looks like it is being fast forwarded. Any idea what to do with the keyframe, audio fragments, and video fragments???

[nwtim]

I was able to get downthemall to download the fragments. Or at least I think I got everything. I got a bunch of files what I believe should be audio, all in sequence named "Fragments(audio_en_0,format=f4m-f4f)Seg1-Frag735", etc, etc.

Also got what I would think is video, all in sequence named "Fragments(video,format=f4m-f4f)Seg1-Frag1", etc, etc.

What I guess I figured out, I had been only downloading the keyframe files ONLY when using the AdobeHDS.php script. I dont know what to do with those keyframe files. It does stitch them together, and I get what I described as a video that looks like it is being fast forwarded. Any idea what to do with the keyframe, audio fragments, and video fragments???This is starting to add up to something, but I don't know the solutions yet.

I recall that a few months back YouTube made a change to their video delivery that caused most downloaders to not be able to get the 1080p or 4K video without a muxing operation. Several of them began to charge money for the 1080p and 4K streams because they had to do the muxing on their own servers. JDownloader did it by using the client's PC, but requiring ffmpeg on the PC.

Perhaps what we have to do with these audio and video stream fragments is very similar to what the downloaders are doing with YouTube's highest quality videos. I just came across KSV's AdobeHDS.php yesterday, so I don't know if it can do this kind of merging or not or if this is just some kind of DRM issue. I'll try to get some time later to poke around it a bit.

[nwtim]

Okay, I see what you're talking about now because I went ahead and updated my Firefox so I could use the HDS Links Detector add-on. I just finished downloading the Women's Gold Medal Hockey game and I had it keep all the fragments. The first thing I noticed was that these fragments were really very small files - in the neighborhood of 100 KB each - which is about what you would expect from a snapshot you might use in creating stop-motion video. There was no audio at all.

In contrast, when I used DownloadThemAll, I was getting fragments averaging more than 2-3 MB each.

So, using DownloadThemAll, I downloaded about 40 fragments and renamed them to 1.f4f, 2.f4f, 3.f4f etc. Then I joined them by just running "php AdobeHDS.php" in a command window of that folder. The resulting video (Joined.flv) plays and looks fantastic, but there's just no sound.

Please tell me how you got the audio fragments, because I somehow missed those.

Anyway, I'm kinda thinking we might be able to join the audio files the same way I joined the video files using AdobeHDS and then once we have those we should be able to mux them together.

[ehurdler]

Can you please give me the code, syntax you used to join the video files? Also, what were the resulting files you were able to download with downthemall called? As well as how did you download them exactly? I did what you said in the post from last night and all the fragments/files are "Fragments(video,format=f4m-f4f)Seg1-Frag1" "Fragments(video,format=f4m-f4f)Seg1-Frag2", etc, etc. Those are the names of the files, no ext at all. I have tried adding .f4f and attempting to join together and I am getting nowhere.

[nwtim]

It looks like you got exactly the same kind of files I got, and I believe we're both using the same method. Mine were named just like yours, ending in "Seg1-Frag[sequence number]" with no extension.

Based on KSV's wiki link (https://github.com/K-S-V/Scripts/wiki) under "Offline fragment processing", I decided to rename them all to the form 1.f4f, 2.f4f, 3.f4f, and so on (I use Flexible Renamer to make it a very quick and simple operation), but the wiki states that the extension is optional.

I opened up a command window in the folder with all the fragment files and simply typed "php AdobeHDS.php". That's it - the resulting file, named Joined.flv, was created almost instantaneously even though the 20-minute video was over 500 MB in size.

Now we just have to figure out how to get it with audio...

Edit:
Okay, I've got it. I'm in the process of downloading 2000 audio and 2000 video fragments right now so I can play with portions of them. The first thing I'm going to try is join some audio files with AdobeHDS.

[nwtim]

I grabbed 10 audio fragment files and renamed them to 1,2,3,etc and was able to "join" them with AdobeHDS.php. It still created a file named Joined.flv, but I wasn't able to hear anything despite VLC being able to "play" it.

MediaInfo says this about it:

General
Complete name : E:\My Documents\Downloads\dTa\Test\Joined_101-110_Audio.flv
Format : Flash Video
File size : 423 KiB
Duration : 59s 954ms
Overall bit rate : 57.8 Kbps

Audio
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : HE-AAC / LC
Codec ID : 10
Duration : 59s 954ms
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 KHz / 22.05 KHz
Compression mode : Lossy

I joined the 10 video fragments that should match the audio fragments and MediaInfo reports:

General
Complete name : E:\My Documents\Downloads\dTa\Test\Joined_101-110_Video.flv
Format : Flash Video
File size : 24.7 MiB
Duration : 1mn 0s
Overall bit rate mode : Constant
Overall bit rate : 3 454 Kbps

Video
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L3.1
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Codec ID : 7
Duration : 1mn 0s
Bit rate mode : Constant
Bit rate : 3 450 Kbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.150
Stream size : 23.7 MiB (96%)

I see there's a small difference in the duration of the files, so muxing them - if it's even possible - would likely end in a slight offset - and that may very well be where the keyframes come in.

I'm not really sure where to go from here other than seeing if there's some way to mux them together.

[nwtim]

I just realized that the fragments I had tested with were from a commercial break. Trying a different portion now...

[nwtim]

Okay, I joined the audio and video of a completely different range of fragments and am getting audio from the joined audio files and video from the joined video files. If I play the video in one player and the audio in another, it actually sounds and looks like a great video. So now to see if I can join them - I'm not sure how flash video and audio works for muxing (as opposed to mpa and mpv files).

[nwtim]

Okay, I got the audio and video to mux together using Video-to-Video Converter (http://www.videohelp.com/tools/Video-to-Video-Converter), but the synchronization is way off (the audio for fragment 770 was not even close to the video of fragment 770). I'm going to have to look into it when I have more time.

Give it a shot and share any ideas you come up with.

Edit:
I think the audio synchronization was my fault. I noticed that the audio leads the video by about 5-6 seconds throughout a 2-hour recording, so I'm guessing when I renumbered the files I missed an audio fragment at the beginning. I'm going to give it another go.

[ehurdler]

I am kind of at the same stage now as you are. I got great video, and great audio....Just unfortunately the sync time was over an hour off. So I am deleting some 728 fragments of video and re-joining. This will have the same number of audio and video fragments. Maybe it will match. If not, I think I have by having videio open in vlc and audio opened in iTunes the difference in time. May have to fiddle with delays and seconds to chop out, but I may be close. I am using a mac and have MKV Tools to mux the tracks together. That is of course after I run both thru a different program to get .acc audio, and .mkv video.

I will let you know my progress as well. I am hopeful that I am close enough to get it working with minor tweaks now.

[chap]

try simple
ffmpeg -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(56000)/Manifest(audio_en_0,format=m3u8-aapl,filtername=vodcut)" -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(3450000)/Manifest(video,format=m3u8-aapl,filtername=vodcut)" -c copy t.ts

[nwtim]

try simple
ffmpeg -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(56000)/Manifest(audio_en_0,format=m3u8-aapl,filtername=vodcut)" -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(3450000)/Manifest(video,format=m3u8-aapl,filtername=vodcut)" -c copy t.tsWhoa! - That worked!!! (at least for the first 26 minutes of video - see below) That's exactly the way I was getting all my events from the London Olympics, but in all the dozens of links I examined, I never once saw an m3u8 one!

Did you actually find that link for the audio and m3u8 - or did you configure it based on the link I provided?

After the first 26 minutes, ffmpeg reported the error:

Invalid timestamps stream=0, pts=8786, dts=8589928978, size=67389
[mpegts @ 0731c800] Application provided invalid, non monotonically increasing d
ts to muxer in stream 0: 141188399 >= -8448742593
av_interleaved_write_frame(): Invalid argument

I do have an outdated ffmpeg, so I don't know if that was the cause, but still I'm pretty amazed that you found that link.

[nwtim]

I am kind of at the same stage now as you are. I got great video, and great audio....Just unfortunately the sync time was over an hour off. So I am deleting some 728 fragments of video and re-joining. This will have the same number of audio and video fragments. Maybe it will match. If not, I think I have by having videio open in vlc and audio opened in iTunes the difference in time. May have to fiddle with delays and seconds to chop out, but I may be close. I am using a mac and have MKV Tools to mux the tracks together. That is of course after I run both thru a different program to get .acc audio, and .mkv video.

I will let you know my progress as well. I am hopeful that I am close enough to get it working with minor tweaks now.I re-ran the mux with one more audio fragment at the front and the synchronization seemed perfect all the way through the 2+ hours I did. The trick was to make sure I streamed the video from NBCOlympics and noted the first fragment number reported by Web Console. In the case of the video I grabbed and muxed, the web stream started at fragment 715, so I renamed the video fragment "...Seg1-Frag715" to "1" and the audio fragment "...Seg1-Frag714" to "1".

I'd sure like to know where "chap" came up with the m3u8 link, because that's by far the easiest method (as long as I can get past the error I mentioned above).

[nwtim]

try simple
ffmpeg -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(56000)/Manifest(audio_en_0,format=m3u8-aapl,filtername=vodcut)" -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(3450000)/Manifest(video,format=m3u8-aapl,filtername=vodcut)" -c copy t.tsOkay, I think I see what you did, but I have no idea how you knew the syntax.

It looks like you took the manifest string I provided up to the ".ism/manifest" and replaced "manifest" with "QualityLevels(56000)/Manifest(audio_en_0,format=m3u8-aapl,filtername=vodcut)" for the audio and then send another -i argument with the same string but this time replacing "manifest" with
"QualityLevels(3450000)/Manifest(video,format=m3u8-aapl,filtername=vodcut)" for the video.

That's brilliant, but how did you know to try this? Is this simply using ffmpeg to construct the .ts video out of the manifest link?

[nwtim]

Although using ffmpeg is a brilliant solution when programs can be grabbed without error, I've had no luck getting complete programs on 2 of the 4 programs I've tried it with - notably the women's gold medal hockey game and the ladies figure skating short program. Also, my trusty old workarounds, the -ss and -t switch, don't seem to work at all with this way of using ffmpeg.

I've found that using DownloadThemAll in conjuction with AdobeHDS.php I can get pretty good sized chunks of a program and join and mux the good segments with a perfect result, but I'm having a heckuva time trying to identify the problem fragments that consistently cause AdobeHDS.php to abort with a "Unknown packet type 58 (sometimes 54) encountered! Unable to proceed." error.

I would really appreciate any thoughts or ideas anyone has on this.

[ehurdler]

try simple
ffmpeg -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(56000)/Manifest(audio_en_0,format=m3u8-aapl,filtername=vodcut)" -i "http://olystreameast.nbcolympics.com/vod/8ea27d49-ddab-48cd-8d4e-89eecd346124/ladies--short-program0219054304.ism/QualityLevels(3450000)/Manifest(video,format=m3u8-aapl,filtername=vodcut)" -c copy t.ts

chap, would you be able to explain how you got the syntax for this? I am trying to capture some of the non-sports videos and the syntax you provided does not work for those. Example, this link: http://www.nbcolympics.com/video/examining-impact-space-race

If I navigate to what comes up to be the manifest file, I can see all the options for video stream index, and audio stream index. I just can't get the proper code/syntax to reproduce anything that will stream download with ffmpeg. Quick copy/paste from manifest is:
<SmoothStreamingMedia MajorVersion="2" MinorVersion="0" Duration="11395717666">
<StreamIndex Type="video" Chunks="570" QualityLevels="7" MaxWidth="1280" MaxHeight="720" DisplayWidth="1280" DisplayHeight="720" Url="QualityLevels({bitrate})/Fragments(video={start time})">
<QualityLevel Index="0" Bitrate="3450000" FourCC="H264" MaxWidth="1280" MaxHeight="720" CodecPrivateData="0000000127640029AC723014016EC04400000FA40003A98388 8001A5240001A524EF7B80F844229C0000000128FFBCB0" NominalBitrate="4485000"/>
<QualityLevel Index="1" Bitrate="2200000" FourCC="H264" MaxWidth="960" MaxHeight="540" CodecPrivateData="00000001274D401FB9181E022FDE022000007D20001D4C1C0C 000432380008647DEF701F088453800000000128FFBC80" NominalBitrate="2860000"/>
<QualityLevel Index="2" Bitrate="1400000" FourCC="H264" MaxWidth="704" MaxHeight="396" CodecPrivateData="00000001274D401FB91816067F7808800001F4800075307030 002AB9800155CF7BDC07C22114E00000000128FFBC80" NominalBitrate="1820000"/>
<QualityLevel Index="3" Bitrate="900000" FourCC="H264" MaxWidth="640" MaxHeight="360" CodecPrivateData="000000012742C01EB9101405FF2E022000007D20001D4C1C48 0036EE0006DDDDEF701F08846A0000000128DE3C80" NominalBitrate="1170000"/>
<QualityLevel Index="4" Bitrate="600000" FourCC="H264" MaxWidth="512" MaxHeight="288" CodecPrivateData="000000016742C01EB6808025B01100000303E90000EA60E060 00927C00124FE6A200F8B1750000000168CE3C80" NominalBitrate="780000"/>
<QualityLevel Index="5" Bitrate="400000" FourCC="H264" MaxWidth="340" MaxHeight="192" CodecPrivateData="000000016742C01EB681619E7F011000003E90000EA60E2800 30D40061AE6A200F8B17500000000168CE3C80" NominalBitrate="520000"/>
<QualityLevel Index="6" Bitrate="200000" FourCC="H264" MaxWidth="340" MaxHeight="192" CodecPrivateData="000000016742C01EB681619E7F011000003E90000EA60E0600 30D40061AE6A200F8B17500000000168CE3C80" NominalBitrate="260000"/>
<c n="0" d="20020000"/>
...
...
...
<c n="568" d="20020000"/>
<c n="569" d="4337666"/>
</StreamIndex>
<StreamIndex Type="audio" Name="audio_1" Chunks="570" QualityLevels="1" Language="eng" Url="QualityLevels({bitrate})/Fragments(audio_1={start time})">
<QualityLevel Index="0" Bitrate="48000" FourCC="AACH" SamplingRate="22050" Channels="2" BitsPerSample="16" PacketSize="4" AudioTag="255" CodecPrivateData="139056E5A0"/>
<c n="0" d="21362358"/>
...
...
...
<c n="568" d="19969161"/>
<c n="569" d="2786395"/>
</StreamIndex>
</SmoothStreamingMedia>



Any help would be greatly appreciated. Thank you.

[blimey]

chap, would you be able to explain how you got the syntax for this? I am trying to capture some of the non-sports videos and the syntax you provided does not work for those. Example, this link:

h..p://www.nbcolympics.com/video/examining-impact-space-race

The fragment urls still work just fine. It is almost as easy to download all the fragments with a batch downloader and then use the script to join the fragments, and ffmpeg to mux. The sync came out fine here.

For example, for the the link you reference above;

video:

h..p://olyvodwest.nbcolympics.com/vod/91e6dfa5-213a-4d90-9e82-5a6aa6fec769/216_XX_2461_SPACE_RACE.ism/QualityLevels(3450000)/Fragments(video,format=f4m-f4f)Seg1-Frag1
.
.
.
h..p://olyvodwest.nbcolympics.com/vod/91e6dfa5-213a-4d90-9e82-5a6aa6fec769/216_XX_2461_SPACE_RACE.ism/QualityLevels(3450000)/Fragments(video,format=f4m-f4f)Seg1-Frag569

audio:

h..p://olyvodwest.nbcolympics.com/vod/91e6dfa5-213a-4d90-9e82-5a6aa6fec769/216_XX_2461_SPACE_RACE.ism/QualityLevels(48000)/Fragments(audio_1,format=f4m-f4f)Seg1-Frag1
.
.
.
h..p://olyvodwest.nbcolympics.com/vod/91e6dfa5-213a-4d90-9e82-5a6aa6fec769/216_XX_2461_SPACE_RACE.ism/QualityLevels(48000)/Fragments(audio_1,format=f4m-f4f)Seg1-Frag569


After running the script seperately on each set of fragments, mux the audio and video with ffmpeg:

ffmpeg -i Joinedvideo.flv -i Joinedaudio.flv -c copy video_audio.mp4



Using IDM batch download for the fragments seems faster than ffmpeg. Also note, for videos that don't require any provider authentication, like the one you reference here, TD will download and join the fragments for both the video and the audio (all quality levels offered), you just have to mux them.

[klatwork]

one word: TUBEDIGGER

[blimey]

TD is great, but it seems to only offer a full manifest (with all the quality selections) for public videos.

For videos requiring provider authentication, it would only download one video quality (not the highest), and didn't offer to download audio at all. If you had a different experience with non-public (authentication required) videos using TD, please share.

[ehurdler]

Thank you Blimey!

I had downloaded the individual fragments originally with DownThemAll for that video link I had posted, Space Race video.... Well that had worked, but seemed to be too long of a process. I had seen what chap wrote a few posts before, and that was working for capturing the event/sports. As I stated it did not work for capturing the features/news links off the site.

I had done some more searching after your most recent post. I installed wget, made a temp url.txt list. I keep quickly can now copy/paste the correct url in the list, add or remove lines to count for more or less fragment files. I can quickly run wget with the url list for audio and video fragments, very quick to download them into separate folders. I have to run a renaming app to rename to 1.f4f, 2.f4f, etc. etc. I am on a Macintosh so I use A Better File Rename 8. I am looking into how to use wget to download fragments and rename all in the same process.

Run php AdobeHDS.php for each type, audio/video. Gives me the Joined.flv for both, rename to JoinedAudio.flv, JoinedVideo.flv. Quickly run with ffmpeg as you stated ffmpeg -i Joinedvideo.flv -i Joinedaudio.flv -c copy video_audio.mp4 I am good to go.

Process has a few steps that leads me to sit and run each one. Still looking into automating some of the steps yet, but the process even with multiple steps goes rather quick.

Thank you Very much for your posts and help!

[blimey]

Sorry, I hadn't read the thread carefully enough to realize that you had already tried dealing with the fragments and were trying to avoid it.

I wasn't able to find the m3u/m3u8 style manifest for the most recently referenced video.

I'm confused why the f4m manifest results in only "keyframes" when used as a manifest url with AdobeHDS.php. I traced/watched tubedigger deal with that same f4m manifest (and only that manifest) and it is able to extract full urls for all of the audio and video fragments. I'm going to try to edit a local copy of the f4m manifest to see if it will work with the script directly.

[nwtim]

chap, would you be able to explain how you got the syntax for this?...When I sent a PM to chap to thank him for the ffmpeg parsing syntax, I asked him how he came up with it to which he responded, "interception traffic of mobile app(NBCOlympics)".

When ffmpeg doesn't work the way I expected, I kinda liked DownloadThemAll because of the GUI and being able to easily see the progress of each fragment being downloaded. However, since I was having problems joining them in their entirety due to AdobeHDS.php aborting on certain fragments, I'm now wondering if it is due to dTa downloading some fragments incompletely or incorrectly. I suppose I could try again and again to get all the fragments in their proper and complete form, but it's a real pain to know which fragments those are without time-consuming trial and error.

Wget - wow - I used to use that probably 20 years ago in order to save entire websites for offline viewing, but it never occurred to me to try to use it for grabbing these video fragments. Is there somewhere you can point me to that explains how to do this without downloading the entire internet?

[KSV]

I have looked into the issue and pushed an update (https://raw.github.com/K-S-V/Scripts/master/AdobeHDS.php) to fix it. here is the manifest content.

http://olyvodeast.nbcolympics.com/vod/c2940c3f-3bd7-4a0c-8101-fd847ee226ba/KRII_VOD_OZ0221_AllAround_Sochi_Today_Show.ism/manifest(format=f4m-f4f).f4m


<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns="http://ns.adobe.com/f4m/1.0" version="3.0">
<streamType>recorded</streamType>
<duration>163.067</duration>
<bootstrapInfo profile="named" id="bootstrap_0">AAAAi2Fic3QAAAAAAAAAowAAAAPoAAAAAAACfPsAAAAAAAAAAA AAAAAAAQAAABlhc3J0AAAAAAAAAAABAAAAAQAAAFIBAAAARmFm cnQAAAAAAAAD6AAAAAADAAAAAQAAAAAAAAAAAAAH0AAAAFIAAA AAAAJ40AAABCsAAABTAAAAAAACfPsAAAAAAA==</bootstrapInfo>
<bootstrapInfo profile="named" id="bootstrap_1">AAABO2Fic3QAAAAAAAAAowAAAAPoAAAAAAACfOsAAAAAAAAAAA AAAAAAAQAAABlhc3J0AAAAAAAAAAABAAAAAQAAAFIBAAAA9mFm cnQAAAAAAAAD6AAAAAAOAAAAAQAAAAAAAAAAAAAIWAAAAAIAAA AAAAAIWAAAB80AAAAQAAAAAAAAdY0AAAf7AAAAEQAAAAAAAH2I AAAHzQAAAB8AAAAAAADqvQAAB/sAAAAgAAAAAAAA8rkAAAfNAAAALgAAAAAAAV/tAAAH+wAAAC8AAAAAAAFn6QAAB80AAAA9AAAAAAAB1R4AAAf7A AAAPgAAAAAAAd0ZAAAHzQAAAEwAAAAAAAJKTgAAB/sAAABNAAAAAAACUkkAAAfNAAAAUgAAAAAAAnlKAAADoQAAAFMA AAAAAAJ86wAAAAAA</bootstrapInfo>
<media label="video_3450" bitrate="3450" type="video" bootstrapInfoId="bootstrap_0" url="QualityLevels(3450000)/Fragments(video,format=f4m-f4f)"/>
<media label="video_3450_keyframes" bitrate="3450" type="video-keyframe-only" bootstrapInfoId="bootstrap_0" url="QualityLevels(3450000)/Keyframes(video,format=f4m-f4f)"/>
<media label="video_2200" bitrate="2200" type="video" bootstrapInfoId="bootstrap_0" url="QualityLevels(2200000)/Fragments(video,format=f4m-f4f)"/>
<media label="video_2200_keyframes" bitrate="2200" type="video-keyframe-only" bootstrapInfoId="bootstrap_0" url="QualityLevels(2200000)/Keyframes(video,format=f4m-f4f)"/>
<media label="video_1400" bitrate="1400" type="video" bootstrapInfoId="bootstrap_0" url="QualityLevels(1400000)/Fragments(video,format=f4m-f4f)"/>
<media label="video_1400_keyframes" bitrate="1400" type="video-keyframe-only" bootstrapInfoId="bootstrap_0" url="QualityLevels(1400000)/Keyframes(video,format=f4m-f4f)"/>
<media label="video_900" bitrate="900" type="video" bootstrapInfoId="bootstrap_0" url="QualityLevels(900000)/Fragments(video,format=f4m-f4f)"/>
<media label="video_900_keyframes" bitrate="900" type="video-keyframe-only" bootstrapInfoId="bootstrap_0" url="QualityLevels(900000)/Keyframes(video,format=f4m-f4f)"/>
<media label="video_600" bitrate="600" type="video" bootstrapInfoId="bootstrap_0" url="QualityLevels(600000)/Fragments(video,format=f4m-f4f)"/>
<media label="video_600_keyframes" bitrate="600" type="video-keyframe-only" bootstrapInfoId="bootstrap_0" url="QualityLevels(600000)/Keyframes(video,format=f4m-f4f)"/>
<media label="video_400" bitrate="400" type="video" bootstrapInfoId="bootstrap_0" url="QualityLevels(400000)/Fragments(video,format=f4m-f4f)"/>
<media label="video_400_keyframes" bitrate="400" type="video-keyframe-only" bootstrapInfoId="bootstrap_0" url="QualityLevels(400000)/Keyframes(video,format=f4m-f4f)"/>
<media label="video_200" bitrate="200" type="video" bootstrapInfoId="bootstrap_0" url="QualityLevels(200000)/Fragments(video,format=f4m-f4f)"/>
<media label="video_200_keyframes" bitrate="200" type="video-keyframe-only" bootstrapInfoId="bootstrap_0" url="QualityLevels(200000)/Keyframes(video,format=f4m-f4f)"/>
<media label="audio_1_48" bitrate="48" type="audio" bootstrapInfoId="bootstrap_1" url="QualityLevels(48000)/Fragments(audio_1,format=f4m-f4f)" lang="en" alternate="true"/>
</manifest>


it contains two entries for each bitrate one for actual video stream and one for keyframes only. fragment url for "keyframe only fragments" was overwriting the actual video entry (due to same bitrate) resulting in downloading of keyframes only stream. also audio is being served as different stream instead of being muxed into video stream. when you run the updated script with debug switch it will show the following content.
php AdobeHDS.php -manifest "http://olyvodeast.nbcolympics.com/vod/c2940c3f-3bd7-4a0c-8101-fd847ee226ba/KRII_VOD_OZ0221_AllAround_Sochi_Today_Show.ism/manifest(format=f4m-f4f).f4m" -debug
Processing manifest info....
Manifest Entries:

Bitrate URL
3451 QualityLevels(3450000)/Keyframes(video,format=f4m-f4f)
3450 QualityLevels(3450000)/Fragments(video,format=f4m-f4f)
2201 QualityLevels(2200000)/Keyframes(video,format=f4m-f4f)
2200 QualityLevels(2200000)/Fragments(video,format=f4m-f4f)
1401 QualityLevels(1400000)/Keyframes(video,format=f4m-f4f)
1400 QualityLevels(1400000)/Fragments(video,format=f4m-f4f)
901 QualityLevels(900000)/Keyframes(video,format=f4m-f4f)
900 QualityLevels(900000)/Fragments(video,format=f4m-f4f)
601 QualityLevels(600000)/Keyframes(video,format=f4m-f4f)
600 QualityLevels(600000)/Fragments(video,format=f4m-f4f)
401 QualityLevels(400000)/Keyframes(video,format=f4m-f4f)
400 QualityLevels(400000)/Fragments(video,format=f4m-f4f)
201 QualityLevels(200000)/Keyframes(video,format=f4m-f4f)
200 QualityLevels(200000)/Fragments(video,format=f4m-f4f)
48 QualityLevels(48000)/Fragments(audio_1,format=f4m-f4f)

Quality Selection:
Available: 3451 3450 2201 2200 1401 1400 901 900 601 600 401 400 201 200 48
Selected : 3450

you can select your required stream by looking at url shown in debug info.

Video:
php AdobeHDS.php -manifest "http://olyvodeast.nbcolympics.com/vod/c2940c3f-3bd7-4a0c-8101-fd847ee226ba/KRII_VOD_OZ0221_AllAround_Sochi_Today_Show.ism/manifest(format=f4m-f4f).f4m" -quality 3450

Audio:
php AdobeHDS.php -manifest "http://olyvodeast.nbcolympics.com/vod/c2940c3f-3bd7-4a0c-8101-fd847ee226ba/KRII_VOD_OZ0221_AllAround_Sochi_Today_Show.ism/manifest(format=f4m-f4f).f4m" -quality 48

Remux:
ffmpeg -i Video.flv -i Audio.flv -c copy Final.mp4

PS: regarding the problem with converting some of the fragments downloded with DTA it's actually DTA's fault. sometimes it saves the error message sent by server instead of actual fragment data. also you don't have to rename the downloaded fragments. read the wiki (https://github.com/K-S-V/Scripts/wiki#offline-fragment-processing) again to understand how to do it properly.

[ehurdler]

WOW, KSV, this is great! Thank you! Makes the process of capturing the non-event videos so much easier!!

[chap]

KSV
as always:cool:
respect for your work

[nwtim]

I have looked into the issue and pushed an update (https://raw.github.com/K-S-V/Scripts/master/AdobeHDS.php) to fix it...

PS: regarding the problem with converting some of the fragments downloded with DTA it's actually DTA's fault. sometimes it saves the error message sent by server instead of actual fragment data. also you don't have to rename the downloaded fragments. read the wiki (https://github.com/K-S-V/Scripts/wiki#offline-fragment-processing) again to understand how to do it properly.That's an awesome fix! Thanks!!

As for the renaming, the reason I was doing it was because even when I tried to use it with a base filename it seemed to require that the first fragment was "1", while the files I was working with at the time had fragments starting at about Seg1-Frag620 or so, and the base filename method didn't seem to like that.

[nwtim]

...
When you run the updated script with debug switch it will show the following content.
php AdobeHDS.php -manifest "http://olyvodeast.nbcolympics.com/vod/c2940c3f-3bd7-4a0c-8101-fd847ee226ba/KRII_VOD_OZ0221_AllAround_Sochi_Today_Show.ism/manifest(format=f4m-f4f).f4m" -debug
...This is working great with the special feature videos, but I was also hoping to download a number of event replays which have a manifest that includes a token, and I can't seem to get anything with those.

Here's an example of an event replay's manifest script as captured with the HDS Links Detector add-on for Firefox:
"http://olystreameast.nbcolympics.com/vod/a3ceaf1c-15b3-4cb0-a83a-00af8e5e08de/pairs-short-program0211061442.ism/manifest(format=f4m-f4f,filtername=vodcut).f4m?token=st=1393308775~exp =1393308895~acl=/vod/a3ceaf1c-15b3-4cb0-a83a-00af8e5e08de/pairs-short-program0211061442.ism/*~hmac=2defc70d894a7774c2fbf749cdc5a3458c9151f212b 608410b10a44df134921e"Can you show me what to parse out in order for the -debug switch to work with such a manifest?

Thanks!

[KSV]

As for the renaming, the reason I was doing it was because even when I tried to use it with a base filename it seemed to require that the first fragment was "1", while the files I was working with at the time had fragments starting at about Seg1-Frag620 or so, and the base filename method didn't seem to like that.

You can use start switch to specify the starting fragment.
php AdobeHDS.php --fragments "Fragments(video,format=f4m-f4f)Seg1-Frag" --start 620
or
php AdobeHDS.php "Fragments(video,format=f4m-f4f)Seg1-Frag" --start 620

This is working great with the special feature videos, but I was also hoping to download a number of event replays which have a manifest that includes a token, and I can't seem to get anything with those.

Here's an example of an event replay's manifest script as captured with the HDS Links Detector add-on for Firefox:
"http://olystreameast.nbcolympics.com/vod/a3ceaf1c-15b3-4cb0-a83a-00af8e5e08de/pairs-short-program0211061442.ism/manifest(format=f4m-f4f,filtername=vodcut).f4m?token=st=1393308775~exp =1393308895~acl=/vod/a3ceaf1c-15b3-4cb0-a83a-00af8e5e08de/pairs-short-program0211061442.ism/*~hmac=2defc70d894a7774c2fbf749cdc5a3458c9151f212b 608410b10a44df134921e"Can you show me what to parse out in order for the -debug switch to work with such a manifest?

i am not sure what you are asking for. you can append "--debug" switch to the command line provided by "HDS Link Detector" addon without modifying the command line in any other way.

[blimey]

This is working great with the special feature videos, but I was also hoping to download a number of event replays which have a manifest that includes a token, and I can't seem to get anything with those.
.
.
Can you show me what to parse out in order for the -debug switch to work with such a manifest?

Thanks!

The event programs use authenticated time-limited urls for the f4m manifest. The syntax would be much like the example given by ehurdler in post #10 (http://stream-recorder.com/forum/showpost.php?p=64841&postcount=10) of this thread... only adjusted to reflect your authentication particulars.

You probably don't want to waste time, or an authenticated f4m manifest url access, doing a debug command. You know the quality levels already, have separate command windows open and do one each for the audio and video before the manifest url expires. Or you could do them consecutively, but you'd have to get a new manifest url before you do the second one as the first will likely have expired.

What was the problem you had using the ios url (m3u8) w/ffmpeg (as provided by chap) for an event program? I had no trouble, although I did the audio and video in separate consecutive ffmpeg sequences, rather than as two inputs to ffmpeg as a single command as chap has it. The ios urls for the manifest need no authentication even for event programs.

[nwtim]

You can use start switch to specify the starting fragment.
php AdobeHDS.php --fragments "Fragments(video,format=f4m-f4f)Seg1-Frag" --start 620
or
php AdobeHDS.php "Fragments(video,format=f4m-f4f)Seg1-Frag" --start 620Oops - my bad. Being new to the whole "Fragment" world, I guess I wasn't confident as to the exact syntax for the start switch so I was taking the extra minute to rename the files. I won't waste my time on that anymore. Thanks!

i am not sure what you are asking for. you can append "--debug" switch to the command line provided by "HDS Link Detector" addon without modifying the command line in any other way.Wouldn't you know it - I couldn't get it to work last night on my other PC and just now when I tried it it worked. Must be that my other PC is clogged up with Windows Media Center, recording multiple shows at once off our HDHomeRun Tuners, and is only Win7x86. With that PC I was getting no quality header from the manifest or an "Unauthorized" error when omitting the "?token=" stuff from the HDS Link.

Thanks for all your help!

[nwtim]

The event programs use authenticated time-limited urls for the f4m manifest. The syntax would be much like the example given by ehurdler in post #10 (http://stream-recorder.com/forum/showpost.php?p=64841&postcount=10) of this thread... only adjusted to reflect your authentication particulars.

You probably don't want to waste time, or an authenticated f4m manifest url access, doing a debug command. You know the quality levels already, have separate command windows open and do one each for the audio and video before the manifest url expires. Or you could do them consecutively, but you'd have to get a new manifest url before you do the second one as the first will likely have expired.

What was the problem you had using the ios url (m3u8) w/ffmpeg (as provided by chap) for an event program? I had no trouble, although I did the audio and video in separate consecutive ffmpeg sequences, rather than as two inputs to ffmpeg as a single command as chap has it. The ios urls for the manifest need no authentication even for event programs.Right - I normally don't take the time to use the debug switch. I was only using it looking for possible explanations why I couldn't get a couple event replays to work last night with KSV's Adobe HDS downloader.

After giving up on that approach, I tried chap's ffmpeg method of getting the m3u8 streams and had amazing success grabbing a couple 4-hour events. Perhaps it was due to being able to bypass whatever issue I was having with the tokens I was getting (I never waited more than a minute between getting the authentication token and trying to run with it - it just wasn't working).

I'm thinking the problems I've had with these downloads could be - and probably are - tied to having spotty internet service both at home and at work over the last week or so (using Comcast in greater Seattle area). I'm not sure how tolerant these methods are of connection interruptions, but last night and this morning seem to have been far better in that regard, and correspondingly any download attempts I've made have been successful.

For example, one I had great difficulty with a couple days ago - the Women's Moguls Qualification 1 - now seems to be coming down without a hitch using KSV's Adobe HDS downloader. Since the connection is throttled down a bit, I imagine I'll need to get a new authentication token for the audio, but so far so good.

Thanks!

[nwtim]

Okay, I've got a problem like the one I was getting last night. It's for the "Figure Skating: Team Event: Men's SP, Pairs SP". The HDS Link I get is php AdobeHDS.php --manifest "http://olystreameast.nbcolympics.com/vod/724e2f71-42bf-43c2-a878-ff6c305e318b/team--men-s-sp--pairs-sp0206063549.ism/manifest(format=f4m-f4f).f4m?token=st=1393370184~exp=1393370304~acl=/vod/724e2f71-42bf-43c2-a878-ff6c305e318b/team--men-s-sp--pairs-sp0206063549.ism/*~hmac=f96ea87266758164ee32a2a220d6f3f6ac3773ee108 bdd6da1279c86199f9c88" --delete and when I replace --delete with --debug I get "Access Denied! Unable to download the manifest." And this is while I have it playing in the browser and keep refreshing the HDS link.

I tried chap's ffmpeg method to get the m3u8 streams and it isn't working either using the string http://olystreameast.nbcolympics.com/vod/724e2f71-42bf-43c2-a878-ff6c305e318b/team--men-s-sp--pairs-sp0206063549.ism in the syntax.

[chap]

nwtim
try
ffmpeg -i "http://olystreameast.nbcolympics.com/vod/724e2f71-42bf-43c2-a878-ff6c305e318b/team--men-s-sp--pairs-sp0206063549.ism/QualityLevels(56000)/Manifest(audio_en_0,format=m3u8-aapl)" -i "http://olystreameast.nbcolympics.com/vod/724e2f71-42bf-43c2-a878-ff6c305e318b/team--men-s-sp--pairs-sp0206063549.ism/QualityLevels(3450000)/Manifest(video,format=m3u8-aapl)" -c copy t.ts

[nwtim]

Somehow the Manifest argument I had in my template had been changed to "(audio_en_0,format=m3u8-aapl,filtername=vodcut)" and the same ",filtername=vodcut" for the video. Now it's running just fine.

Thanks chap!

[blimey]

I get "Access Denied! Unable to download the manifest." And this is while I have it playing in the browser and keep refreshing the HDS link

I've now seen a few of these "Access Denied" errors for some authentication-required events. The f4m manifest link, or even a f4f fragment link, reports this error if used outside of the browser. But if the link is accessed inside the browser (in which the authentication is stored), it is okay. There must be browser cookies, or more likely flash/adobe cookies. Some special files are downloaded from adobe the first time authentication goes through.

[nwtim]

I don't see those errors when I'm using chap's ffmpeg method of getting the m3u8 streams, but there are several that I cannot get the entire event of and I can't seem to use the -ss and -t switches to steer around problem areas with his method. In those cases where I can't use the ffmpeg method and the AdobeHDS.php method returns the unauthorized access error, I don't know of any other solution than to try to do the best I can with DownloadThemAll.

[ehurdler]

I don't see those errors when I'm using chap's ffmpeg method of getting the m3u8 streams, but there are several that I cannot get the entire event of and I can't seem to use the -ss and -t switches to steer around problem areas with his method. In those cases where I can't use the ffmpeg method and the AdobeHDS.php method returns the unauthorized access error, I don't know of any other solution than to try to do the best I can with DownloadThemAll.

I have downloaded every event I have wanted so far with no errors at all with AdobeHDS.php since KSV updated the script. Can you give me an example of an event replay that is not working for you? As well and explaining your process?

I have a folder on my desktop with the AdobeHDS.php script in it and nothing else. I have a text document opened at all times with only the template of:
php AdobeHDS.php --manifest "" -quality 56 --delete

php AdobeHDS.php --manifest "" -quality 3450 --delete

I open for example:
http://stream.nbcolympics.com/cross-country/winter/14193/

I quickly open Tool->Web Developer->Web Console. I only have the "Net" tab opened. As the event reply is starting, I will click on the first line I see that has ".ism/manifest....etc.etc. in it to open the "Inspect Network Request" window. Copy the full "Request URL:" ASAP. That that and paste it in between the " "s on my text document template for the audio first.

Example:
php AdobeHDS.php --manifest "http://olystreameast.nbcolympics.com/vod/a9b40645-ee55-47e0-8e6c-fe848ead2493/cc--4x5km-relay0215001906.ism/manifest(format=f4m-f4f,filtername=vodcut).f4m?token=st=1393447534~exp =1393447654~acl=/vod/a9b40645-ee55-47e0-8e6c-fe848ead2493/cc--4x5km-relay0215001906.ism/*~hmac=a2c9229ca75d27d306db55cfe9376ce0af3e961c99b 0eed6eef6f8cb8ecfe340" -quality 56 --delete

I immediately run this command in Terminal (as I am using a Macintosh). Run the command ASAP as it does have an expiration as you know, this way I guess it won't time out, give errors, access denied, etc. I start to see the captured fragments number go up, up, etc. I just pause the event replay, and keep the website open. (not sure if this makes a difference, but it is not open long, plus I will refresh it to capture video segments in a few minutes.)

I let that capture all the fragments into the specified folder that I am running AdobeHDS.php to/from. Once that is done, I rename the .flv file that was created to something like "4x5kmRelayAudio.flv" I manually delete all the fragments now, as for some reason the --delete code doesn't work, but I don't care I just manually delete them.

Now refresh the webpage, so to grab the ".ism/manifest...etc...etc." url again just as done for the audio portion above. Copy url into my template, paste whole command into Terminal again, let it capture all the fragments. Keep page open, pause video, wait till complete. Once it captures them all I rename the .flv file again to "4x5kmRelayVideo.flv". Now I use an app to change audio to aac format and the video to .mkv format. I use another app to mux the track together. I have gotten a flawless video everytime this way. Using FFMPEG I kept getting audio issues. I couldn't edit the resulting videos at all. This way I am able to edit out portions with no issue. Look great on USB plugged to Flatscreen, or streaming to/through AppleTV.

Sorry if I went into too much detail. It took me a long time to get MY process down based on KSV, blimey, and chaps posts to get it working perfectly for me! Thanks everyone BTW! NWTIM, if you have any other issues let us know.

[nwtim]

Thanks for all that detail, ehurdler. I went over it carefully to see if there was anything you were doing that was fundamentally different than what I'm doing. The biggest difference may be that I'm getting the manifest file from KSV's HDS Link Detector Add-on or filtering the Web Console feedback for .f4m or Seg (when specific frag numbered files are of interest to me) files.

In your experience, just how quickly do these authentication tokens typically expire? I run everything from batch files, but usually never have a delay of more than about 30 seconds from the time I paste the text string in to the time I run it.

I don't have time at the moment, but tonight I'll take another stab at one of the videos I've had trouble with (they're all event replays, of which I'm having about a 60-70% success rate with).

[nwtim]

I've solved my problem - all having to do with using expired authentication tokens. I kept re-grabbing the HDS Link Detector script to plug it into my batch file and at some point I "detected" that it hadn't changed from the last time I grabbed it, so I made the mistaken assumption that it just wasn't expiring that fast.

For several days it didn't occur to me that the HDS Link Detector might not be getting its last value flushed as long as the browser was open and/or the cache not cleared. It may be even simpler than this, but ever since I began clearing the browser's cache and closing it and then re-opening it to re-grab the manifest script from HDS Link Detector, the AdobeHDS.php method has been working flawlessly - even on the videos I was having the toughest time with.

I was/am using a batch file with just these lines in it:

"" -quality 56 -outfile Audio.flv

pause

"" -quality 3450 -outfile Video.flv

pause

ffmpeg -i Audio.flv -i Video.flv -c copy FINAL.mp4

pause

The double quotes get replaced with the script that populates the clipboard when I click on the HDS Link Detector pop-up. The pauses are so I can monitor any feedback before continuing on to the next step. If something went wrong I can simply abort the batch file (Ctrl-C). I also noticed that the ffmpeg mux didn't work properly unless the pause was there and I waited a couple seconds to continue. BTW, the HDS Link Detector pop-up script includes the --delete switch and it seems to actually delete all the fragments when I run the full batch file but not when I run only one part of it (remark out the other operations).
...
Now refresh the webpage, so to grab the ".ism/manifest...etc...etc." url again just as done for the audio portion above.This is where I think I was having my problem. I WAS refreshing the page on a consistent basis (many times, actually), but the script that HDS Link Detector was putting on my clipboard was NOT changing - which is probably why I was lulled into thinking the expired tokens were still current.
...Now I use an app to change audio to aac format and the video to .mkv format. I use another app to mux the track together. I have gotten a flawless video everytime this way. Using FFMPEG I kept getting audio issues. I couldn't edit the resulting videos at all. This way I am able to edit out portions with no issue. Look great on USB plugged to Flatscreen, or streaming to/through AppleTV...I'm not getting audio issues with ffmpeg at all as long as I'm using AdobeHDS.php to merge the fragment files. I do note however, that sometimes the number of audio fragments has been one more than the number of video fragments, which may or may not be related to my finding that my audio sync was normally spot on with DownloadThemAll fragments when I forced "audio...Seg1-Frag[x]" to align with "video...Seg1-Frag[x+1]".

Anyway, all is good now with AdobeHDS.php doing the downloading and joining and ffmpeg doing the muxing and converting to MP4.

Thanks for everyone's contributions here.

[james667]

Has anyone had success downloading any Olympic streams from cbc.ca? I've been at this for a while. I want this "http://olympics.cbc.ca/videos/live/video/speed-skating-women-1500m-finals-webcast.html" I have tried Orbit downloader and ended up with a lot of unplayable mov files? I'm using Ismdownloader right now and need some help. I believe I filtered out the ism/manifest link with firefox console, now how would I input that into ismdownloader? When I open the ismdownloader.exe it closes right away. I'm missing a few steps. If anyone can help or has a better way to grab from cbc please let me know.
Thanks in advance.

[x-9er]

Has anyone figured out a way to capture the closed captioning from these broadcasts?