Stream capture from NBCOlympics

Stream capture from NBCOlympics

 

Back in 2012, I got a number of extremely high quality videos of the London 2012 Olympics from the NBCOlympics site using ffmpeg to input the stream from a URL I got from Firefox's Web Console using the .m3u8 filter. In trying to do something similar this year for the Winter Olympics, I'm able to play the videos in Firefox, but I've had no luck coming up with a URL to try to grab.

Instead, the best I'd probably be able to do would be to get the 5-second long snippets that tick by and merge them together.

Is anybody successfully doing this or can help me find the URL that addresses the whole video stream?

Thanks!

Re: Stream capture from NBCOlympics

 

NBC used YouTube to stream two years ago. They're using AdobeHDS now. You need to acquire every .fdf file for a stream then stitch them together using the php script.

I have yet to actually make this work; my videos end up only consisting of the keyframes.

Re: Stream capture from NBCOlympics

 

Oh yes, I remember there being problems with Adobe HDS for people trying to capture BBC streams of the 2012 Olympics. Adobe HDS segments are variable bit rate which makes audio re-alignment tricky. What I think I recall regarding the Adobe HDS streams from BBC is that HDS was just a default stream. If I remember correctly, any problem loading the HDS links resulted in a redirect to rtmpe streams, which presented fewer problems. I don't know if that's the way NBC is doing it, though, because I haven't had it fail to load the HDS links.

Any thoughts on the possibility of finding rtmpe streams?

Re: Stream capture from NBCOlympics

 

any luck on this?

I've been trying to and I have the same issue...

i.e., run the adobeHDS script against the f4m manifest file and it just assembles a flash video file (i.e,. flv) that only has the keyframes seemingly

perhaps there's some extra parameter to pass/tweak in order for it to request all of the frames instead of just keyframes?

Re: Stream capture from NBCOlympics

 

Your best bet will be a Screen Capture tool such as Replay Video Capture

Re: Stream capture from NBCOlympics

 

I have a couple screen capturing programs that I've tried this with and the resulting file is always jerky and the filesizes are very large. Joining the actual files used for the stream saves significant disk space, and also addresses the framerate match issue.

Re: Stream capture from NBCOlympics

 

Using the Firefox add-on DownloadThemAll, I've been able to grab a long sequence of Seg1-FragXXX files, but I'm not certain how to join them with AdobeHDS.

In case it helps, the files I have are named as follows:

007_Fragments(video,format=f4m-f4f)Seg1-Frag762
...
007_Fragments(video,format=f4m-f4f)Seg1-Frag1000

Of course, I can rename them to anything I want to make joining simpler, so I just need to know what tool and syntax to use to join them.

Can someone point me in the right direction for this? Thanks!

Re: Stream capture from NBCOlympics

 

How do you get downthemall to capture the segment files? I have been trying to figure out how to capture events from nbcolympics.com all week. Stumbled upon using downthemall, but cant even figure out how to get the seg*_frag* files to download with it.

Re: Stream capture from NBCOlympics

 

There are probably a number of ways to do it, but I used the Copy URL option on several of the files linked via VideoDownloadHelper. I copied these URLs to a text window and found the links for the quality I was looking for and some of the file fragment numbers. Next, I opened the DownloadThemAll Manager window (Firefox->Tools->DownThemAll! Tools->Manager) and clicked the (+) button. The resulting Download: text window was populated with the text I had happened to put in my clipboard (which ended in "Seg1-Frag762") and I just changed the 762 to [762:2000], the range I wanted to capture. I clicked on Start! and then "Download Batch" and voila!, the files proceded to be saved.

I'm not sure it's going to do any good, though, because I can't seem to join them together to play.

Re: Stream capture from NBCOlympics

 

Thank you for replying. I did get AdobeHDS.php to actually download all the frames, stitches them together. Problem now is, it gives the full length of what I was trying to capture. But there is first no Audio, second the timing of all the keyframes/fragments is all fast. It is like you fast forward the entire Hockey game in less than 2 mins. Any ideas?

The line of code i used was:

I am using a Macintosh, running this script in the Terminal. You could give it a try whether you are using a Mac or Windows.

Just paste your manifest url within the quotes and run it as soon as you can before it expires. You could also omit the --delete option if you want to keep the fragments saved.

I am at a loss on how to cature the audio and have the timing between frames set properly. If you or anyone has any ideas, please let me know.

Re: Stream capture from NBCOlympics

 

You got a lot further than I did. I haven't even been able to identify the proper manifest URL. I'm using an older version of Firefox (17.0.5) for an add-on (CacheViewer Fx21) I frequently use that doesn't work in newer versions, so I can't use KSV's HDS Link Detector add-on and I can't seem to parse it out using Web Console or Coojah.

I understand what you're describing regarding the improper playback and audio problems. I've had this happen when audio and video streams don't mux properly, so it must have something to do with the way the fragments are being joined together. Have you tried grabbing a lower quality stream to see if the problem is there as well?

Re: Stream capture from NBCOlympics

 

NWTIM:
As for your version of Firefox I am unsure any difference than the one I am using. But just open up Tools->Web Developer->Web Console.

Once Web Console is open, on the tabs across the top of the console turn everything off but Net. In the filter/search box on the right of the console search for Manifest. I usually open the console on the page I am viewing, do the search, then refresh the page just to make sure I see everything I am searching for the start of the page loading. Give that a try, try my method I described earlier, see if you get same result. I am trying to capture the USA vs CZE Hockey game from today.

Re: Stream capture from NBCOlympics

 

I got a good manifest link for an NBCOlympics video, but when I try to run it (I basically just copied it into your string and added a --duration 60 switch) I get a "Could not open input file: AdobeHDS.php" error. I'm wondering if I'm just missing an --auth switch string or something.

Do you know anything about that?

Re: Stream capture from NBCOlympics

 

Can I see your code you entered in?

Re: Stream capture from NBCOlympics

 

Sure, I just started one of the most recent files on the NBCOlympics|Videos page and parsed out the manifest link.

It's probably expired by now, or perhaps it's IP address dependent.

Re: Stream capture from NBCOlympics

 

I was able to get downthemall to download the fragments. Or at least I think I got everything. I got a bunch of files what I believe should be audio, all in sequence named "Fragments(audio_en_0,format=f4m-f4f)Seg1-Frag735", etc, etc.

Also got what I would think is video, all in sequence named "Fragments(video,format=f4m-f4f)Seg1-Frag1", etc, etc.

What I guess I figured out, I had been only downloading the keyframe files ONLY when using the AdobeHDS.php script. I dont know what to do with those keyframe files. It does stitch them together, and I get what I described as a video that looks like it is being fast forwarded. Any idea what to do with the keyframe, audio fragments, and video fragments???

Re: Stream capture from NBCOlympics

 

This is starting to add up to something, but I don't know the solutions yet.

I recall that a few months back YouTube made a change to their video delivery that caused most downloaders to not be able to get the 1080p or 4K video without a muxing operation. Several of them began to charge money for the 1080p and 4K streams because they had to do the muxing on their own servers. JDownloader did it by using the client's PC, but requiring ffmpeg on the PC.

Perhaps what we have to do with these audio and video stream fragments is very similar to what the downloaders are doing with YouTube's highest quality videos. I just came across KSV's AdobeHDS.php yesterday, so I don't know if it can do this kind of merging or not or if this is just some kind of DRM issue. I'll try to get some time later to poke around it a bit.

Re: Stream capture from NBCOlympics

 

Okay, I see what you're talking about now because I went ahead and updated my Firefox so I could use the HDS Links Detector add-on. I just finished downloading the Women's Gold Medal Hockey game and I had it keep all the fragments. The first thing I noticed was that these fragments were really very small files - in the neighborhood of 100 KB each - which is about what you would expect from a snapshot you might use in creating stop-motion video. There was no audio at all.

In contrast, when I used DownloadThemAll, I was getting fragments averaging more than 2-3 MB each.

So, using DownloadThemAll, I downloaded about 40 fragments and renamed them to 1.f4f, 2.f4f, 3.f4f etc. Then I joined them by just running "php AdobeHDS.php" in a command window of that folder. The resulting video (Joined.flv) plays and looks fantastic, but there's just no sound.

Please tell me how you got the audio fragments, because I somehow missed those.

Anyway, I'm kinda thinking we might be able to join the audio files the same way I joined the video files using AdobeHDS and then once we have those we should be able to mux them together.

Re: Stream capture from NBCOlympics

 

Can you please give me the code, syntax you used to join the video files? Also, what were the resulting files you were able to download with downthemall called? As well as how did you download them exactly? I did what you said in the post from last night and all the fragments/files are "Fragments(video,format=f4m-f4f)Seg1-Frag1" "Fragments(video,format=f4m-f4f)Seg1-Frag2", etc, etc. Those are the names of the files, no ext at all. I have tried adding .f4f and attempting to join together and I am getting nowhere.

Re: Stream capture from NBCOlympics

 

It looks like you got exactly the same kind of files I got, and I believe we're both using the same method. Mine were named just like yours, ending in "Seg1-Frag[sequence number]" with no extension.

Based on KSV's wiki link (https://github.com/K-S-V/Scripts/wiki) under "Offline fragment processing", I decided to rename them all to the form 1.f4f, 2.f4f, 3.f4f, and so on (I use Flexible Renamer to make it a very quick and simple operation), but the wiki states that the extension is optional.

I opened up a command window in the folder with all the fragment files and simply typed "php AdobeHDS.php". That's it - the resulting file, named Joined.flv, was created almost instantaneously even though the 20-minute video was over 500 MB in size.

Now we just have to figure out how to get it with audio...

Edit:
Okay, I've got it. I'm in the process of downloading 2000 audio and 2000 video fragments right now so I can play with portions of them. The first thing I'm going to try is join some audio files with AdobeHDS.

Re: Stream capture from NBCOlympics

 

I grabbed 10 audio fragment files and renamed them to 1,2,3,etc and was able to "join" them with AdobeHDS.php. It still created a file named Joined.flv, but I wasn't able to hear anything despite VLC being able to "play" it.

MediaInfo says this about it:

General
Complete name : E:\My Documents\Downloads\dTa\Test\Joined_101-110_Audio.flv
Format : Flash Video
File size : 423 KiB
Duration : 59s 954ms
Overall bit rate : 57.8 Kbps

Audio
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : HE-AAC / LC
Codec ID : 10
Duration : 59s 954ms
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 KHz / 22.05 KHz
Compression mode : Lossy

I joined the 10 video fragments that should match the audio fragments and MediaInfo reports:

General
Complete name : E:\My Documents\Downloads\dTa\Test\Joined_101-110_Video.flv
Format : Flash Video
File size : 24.7 MiB
Duration : 1mn 0s
Overall bit rate mode : Constant
Overall bit rate : 3 454 Kbps

Video
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L3.1
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Codec ID : 7
Duration : 1mn 0s
Bit rate mode : Constant
Bit rate : 3 450 Kbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.150
Stream size : 23.7 MiB (96%)

I see there's a small difference in the duration of the files, so muxing them - if it's even possible - would likely end in a slight offset - and that may very well be where the keyframes come in.

I'm not really sure where to go from here other than seeing if there's some way to mux them together.

Re: Stream capture from NBCOlympics

 

I just realized that the fragments I had tested with were from a commercial break. Trying a different portion now...

Re: Stream capture from NBCOlympics

 

Okay, I joined the audio and video of a completely different range of fragments and am getting audio from the joined audio files and video from the joined video files. If I play the video in one player and the audio in another, it actually sounds and looks like a great video. So now to see if I can join them - I'm not sure how flash video and audio works for muxing (as opposed to mpa and mpv files).

Re: Stream capture from NBCOlympics

 

Okay, I got the audio and video to mux together using Video-to-Video Converter (http://www.videohelp.com/tools/Video...deo-Converter), but the synchronization is way off (the audio for fragment 770 was not even close to the video of fragment 770). I'm going to have to look into it when I have more time.

Give it a shot and share any ideas you come up with.

Edit:
I think the audio synchronization was my fault. I noticed that the audio leads the video by about 5-6 seconds throughout a 2-hour recording, so I'm guessing when I renumbered the files I missed an audio fragment at the beginning. I'm going to give it another go.

Re: Stream capture from NBCOlympics

 

I am kind of at the same stage now as you are. I got great video, and great audio....Just unfortunately the sync time was over an hour off. So I am deleting some 728 fragments of video and re-joining. This will have the same number of audio and video fragments. Maybe it will match. If not, I think I have by having videio open in vlc and audio opened in iTunes the difference in time. May have to fiddle with delays and seconds to chop out, but I may be close. I am using a mac and have MKV Tools to mux the tracks together. That is of course after I run both thru a different program to get .acc audio, and .mkv video.

I will let you know my progress as well. I am hopeful that I am close enough to get it working with minor tweaks now.

Re: Stream capture from NBCOlympics

 

try simple

Re: Stream capture from NBCOlympics

 

Whoa! - That worked!!! (at least for the first 26 minutes of video - see below) That's exactly the way I was getting all my events from the London Olympics, but in all the dozens of links I examined, I never once saw an m3u8 one!

Did you actually find that link for the audio and m3u8 - or did you configure it based on the link I provided?

After the first 26 minutes, ffmpeg reported the error:

Invalid timestamps stream=0, pts=8786, dts=8589928978, size=67389
[mpegts @ 0731c800] Application provided invalid, non monotonically increasing d
ts to muxer in stream 0: 141188399 >= -8448742593
av_interleaved_write_frame(): Invalid argument

I do have an outdated ffmpeg, so I don't know if that was the cause, but still I'm pretty amazed that you found that link.

Re: Stream capture from NBCOlympics

 

I re-ran the mux with one more audio fragment at the front and the synchronization seemed perfect all the way through the 2+ hours I did. The trick was to make sure I streamed the video from NBCOlympics and noted the first fragment number reported by Web Console. In the case of the video I grabbed and muxed, the web stream started at fragment 715, so I renamed the video fragment "...Seg1-Frag715" to "1" and the audio fragment "...Seg1-Frag714" to "1".

I'd sure like to know where "chap" came up with the m3u8 link, because that's by far the easiest method (as long as I can get past the error I mentioned above).

Re: Stream capture from NBCOlympics

 

Okay, I think I see what you did, but I have no idea how you knew the syntax.

It looks like you took the manifest string I provided up to the ".ism/manifest" and replaced "manifest" with "QualityLevels(56000)/Manifest(audio_en_0,format=m3u8-aapl,filtername=vodcut)" for the audio and then send another -i argument with the same string but this time replacing "manifest" with
"QualityLevels(3450000)/Manifest(video,format=m3u8-aapl,filtername=vodcut)" for the video.

That's brilliant, but how did you know to try this? Is this simply using ffmpeg to construct the .ts video out of the manifest link?

Re: Stream capture from NBCOlympics

 

Although using ffmpeg is a brilliant solution when programs can be grabbed without error, I've had no luck getting complete programs on 2 of the 4 programs I've tried it with - notably the women's gold medal hockey game and the ladies figure skating short program. Also, my trusty old workarounds, the -ss and -t switch, don't seem to work at all with this way of using ffmpeg.

I've found that using DownloadThemAll in conjuction with AdobeHDS.php I can get pretty good sized chunks of a program and join and mux the good segments with a perfect result, but I'm having a heckuva time trying to identify the problem fragments that consistently cause AdobeHDS.php to abort with a "Unknown packet type 58 (sometimes 54) encountered! Unable to proceed." error.

I would really appreciate any thoughts or ideas anyone has on this.

Re: Stream capture from NBCOlympics

 

chap, would you be able to explain how you got the syntax for this? I am trying to capture some of the non-sports videos and the syntax you provided does not work for those. Example, this link: http://www.nbcolympics.com/video/exa...act-space-race

If I navigate to what comes up to be the manifest file, I can see all the options for video stream index, and audio stream index. I just can't get the proper code/syntax to reproduce anything that will stream download with ffmpeg. Quick copy/paste from manifest is:
Any help would be greatly appreciated. Thank you.

Re: Stream capture from NBCOlympics

 

The fragment urls still work just fine. It is almost as easy to download all the fragments with a batch downloader and then use the script to join the fragments, and ffmpeg to mux. The sync came out fine here.

For example, for the the link you reference above;
After running the script seperately on each set of fragments, mux the audio and video with ffmpeg:

Using IDM batch download for the fragments seems faster than ffmpeg. Also note, for videos that don't require any provider authentication, like the one you reference here, TD will download and join the fragments for both the video and the audio (all quality levels offered), you just have to mux them.

Re: Stream capture from NBCOlympics

 

one word: TUBEDIGGER

Re: Stream capture from NBCOlympics

 

TD is great, but it seems to only offer a full manifest (with all the quality selections) for public videos.

For videos requiring provider authentication, it would only download one video quality (not the highest), and didn't offer to download audio at all. If you had a different experience with non-public (authentication required) videos using TD, please share.

Re: Stream capture from NBCOlympics

 

Thank you Blimey!

I had downloaded the individual fragments originally with DownThemAll for that video link I had posted, Space Race video.... Well that had worked, but seemed to be too long of a process. I had seen what chap wrote a few posts before, and that was working for capturing the event/sports. As I stated it did not work for capturing the features/news links off the site.

I had done some more searching after your most recent post. I installed wget, made a temp url.txt list. I keep quickly can now copy/paste the correct url in the list, add or remove lines to count for more or less fragment files. I can quickly run wget with the url list for audio and video fragments, very quick to download them into separate folders. I have to run a renaming app to rename to 1.f4f, 2.f4f, etc. etc. I am on a Macintosh so I use A Better File Rename 8. I am looking into how to use wget to download fragments and rename all in the same process.

Run for each type, audio/video. Gives me the Joined.flv for both, rename to JoinedAudio.flv, JoinedVideo.flv. Quickly run with ffmpeg as you stated I am good to go.

Process has a few steps that leads me to sit and run each one. Still looking into automating some of the steps yet, but the process even with multiple steps goes rather quick.

Thank you Very much for your posts and help!

Re: Stream capture from NBCOlympics

 

Sorry, I hadn't read the thread carefully enough to realize that you had already tried dealing with the fragments and were trying to avoid it.

I wasn't able to find the m3u/m3u8 style manifest for the most recently referenced video.

I'm confused why the f4m manifest results in only "keyframes" when used as a manifest url with AdobeHDS.php. I traced/watched tubedigger deal with that same f4m manifest (and only that manifest) and it is able to extract full urls for all of the audio and video fragments. I'm going to try to edit a local copy of the f4m manifest to see if it will work with the script directly.

Re: Stream capture from NBCOlympics

 

When I sent a PM to chap to thank him for the ffmpeg parsing syntax, I asked him how he came up with it to which he responded, "interception traffic of mobile app(NBCOlympics)".

When ffmpeg doesn't work the way I expected, I kinda liked DownloadThemAll because of the GUI and being able to easily see the progress of each fragment being downloaded. However, since I was having problems joining them in their entirety due to AdobeHDS.php aborting on certain fragments, I'm now wondering if it is due to dTa downloading some fragments incompletely or incorrectly. I suppose I could try again and again to get all the fragments in their proper and complete form, but it's a real pain to know which fragments those are without time-consuming trial and error.

Wget - wow - I used to use that probably 20 years ago in order to save entire websites for offline viewing, but it never occurred to me to try to use it for grabbing these video fragments. Is there somewhere you can point me to that explains how to do this without downloading the entire internet?

Re: Stream capture from NBCOlympics

 

I have looked into the issue and pushed an update to fix it. here is the manifest content.

it contains two entries for each bitrate one for actual video stream and one for keyframes only. fragment url for "keyframe only fragments" was overwriting the actual video entry (due to same bitrate) resulting in downloading of keyframes only stream. also audio is being served as different stream instead of being muxed into video stream. when you run the updated script with debug switch it will show the following content.
you can select your required stream by looking at url shown in debug info.

Video:
Audio:
Remux:
PS: regarding the problem with converting some of the fragments downloded with DTA it's actually DTA's fault. sometimes it saves the error message sent by server instead of actual fragment data. also you don't have to rename the downloaded fragments. read the wiki again to understand how to do it properly.

Re: Stream capture from NBCOlympics

 

WOW, KSV, this is great! Thank you! Makes the process of capturing the non-event videos so much easier!!

Re: Stream capture from NBCOlympics

 

KSV
as always:cool:
respect for your work