be-at.tv changed their internal urls - how to download?

[blimey]

Originally Posted by sebastian

If anyone also knows a command line based app to strip the bytes, please let me know, so I could do the whole process via a batch file. :-)

Swiss File Knife is an ideal choice.

Download sfk169.exe from sourceforge.

Description of the feature that would do the job.

Very crude example of batch file (has little error checking) that uses sfk169.exe to strip the bytes and copy them with the first file to joined.flv.

joinflv.bat:

rem @echo off
if "%1" gtr "9998" goto :END
if not "%1"=="" goto copy 
echo Usage is "%0 lastfilenumber" to join 0001.flv through lastfilenumber.flv
echo and append them with 0000.flv to joined.flv
goto :END 
:COPY
copy 0000.flv joined.flv
set /a first=1
set last=%1
set /a last=10000%last% %% 10000
set last=10000%last%
set last=%last:~-4%
:LOOP
set first=10000%first%
sfk169 partcopy %first:~-4%.flv -allfrom 32 joined.flv -append -yes
set /a first+=1
set first=%first:~-4%
if %first% gtr %last% goto :END
goto LOOP
:END

The batch file is hard coded to start with 0001.flv, so you need only to pass the last file number (without the file extension) as a command line parameter;

joinflv.bat usage:

ex: joinflv lastfilenumber

joinflv 462

-----
Edit:

Check your source files carefully before using this batch file on any non-AVC encoded session, or any session other than:

h..p://www.be-at.tv/brands/jaarbeurs-utrecht/timewarp-2012/marcel-fengler

as the session may be encoded with vp6, or have a different offset for the start of the 'AVC sequence header', as mentioned by KSV in the post below.

In all cases, this binary file copy method will work, but for some (possibly many) sessions, the offset may need to be changed from 32, to quite often 13, but possibly other values as well..

However, ffmpeg with work with many sessions. All my attempts to maintain sync, with this session in particular, failed with the concatenate feature of ffmpeg, although this is probably not a problem with ffmpeg itself. Note that due to the source material in these files, it can be difficult to check sync. I checked it by comparison of the waveform segment of the audio and the corresponding video frame, as well as the overall time length of the video.

[KSV]

Stripping first 32 bytes isn't gonna work reliably with different files. you are assuming that first flv packet is 'AVC sequence header' and it's of same length for different files which isn't true. a better way to join those files is explained below:

1. create a new folder (BeatConvert) and copy BeatConvert.php, FlvFixer.php, FFMpeg.exe to this folder.

2. now create a new bat file (BeatConvert.bat) with following content:

if exist "List.txt" del "List.txt"
for %%a in (*.beat) do php BeatConvert.php %%a && echo file '%%~na.flv' >> List.txt
FFMpeg.exe -y -f concat -i List.txt -c copy Final.flv
php FlvFixer.php --in Final.flv --out Final_fixed.flv --nometa
FFMpeg.exe -y -i Final_fixed.flv -c copy Final.mkv

3. now copy your beat files (0000.beat....nnnn.beat) to the same folder and run the bat file.

if everything goes as expected a new file (Final.mkv) will be created which should be in sync.

[blimey]

Originally Posted by KSV

Stripping first 32 bytes isn't gonna work reliably with different files. you are assuming that first flv packet is 'AVC sequence header' and it's of same length for different files which isn't true.

Yes, I know. For the v1 .beat files, or some others encoded in vp6, it is 13 bytes. But whatever it has been, it has always been consistent within a session for all .beat files 0001+ through the end of the session. In any case, this parameter is easily edited as needed.

Originally Posted by KSV

a better way to join those files is explained below:

I don't know what to tell you, except, as I mentioned earlier in this thread, and in the other thread, my favored method was the concatenate feature of ffmpeg. But it bombed. And of course I tried FlvFixer.php, both on the segmented .flv files, or the joined, or both, and everything it between. I also just tried direct stream copy (individually) of the segmented .flv files with ffmpeg prior to concatenation in order to "fix" the .flv from the way in which it is time coded and yet still maintain sync.

Originally Posted by KSV

1. create a new folder (BeatConvert) and copy BeatConvert.php, FlvFixer.php, FFMpeg.exe to this folder.

2. now create a new bat file (BeatConvert.bat) with following content:

if exist "List.txt" del "List.txt"
for %%a in (*.beat) do php BeatConvert.php %%a && echo file '%%~na.flv' >> List.txt
FFMpeg.exe -y -f concat -i List.txt -c copy Final.flv
php FlvFixer.php --in Final.flv --out Final_fixed.flv --nometa
FFMpeg.exe -y -i Final_fixed.flv -c copy Final.mkv

3. now copy your beat files (0000.beat....nnnn.beat) to the same folder and run the bat file.

if everything goes as expected a new file (Final.mkv) will be created which should be in sync.

Now, for the session that has most recently been in discussion in this thread;

h..p://www.be-at.tv/brands/jaarbeurs-utrecht/timewarp-2012/marcel-fengler

Which has corresponding .beat files at;

h..p://cdn.be-at.tv/001196/128/1250/0000.beat
.
.
h..p://cdn.be-at.tv/001196/128/1250/0462.beat

I just tried your batch file using the raw source .beat files.
The resulting files it produced:

Final_fixed.flv Time length: 01:35:34
Final.mkv Time length: 01:35:34

Time length of video when played in flashplayer: 01:32:28

Time length of video when joined by binary file copy method I mentioned in my previous post: 01:32:28 (it could show up as 01:32:27 on some players)

If you have any ideas, I'm all ears.

But I do agree to caution anyone before using the batch file that I just posted to check the source .flv files after processing with BeatConvert.php.

[KSV]

can you post the debug log of FlvFixer.php?

php FlvFixer.php --in Final.flv --out Final_fixed.flv --nometa --debug 2> Debug.txt

[blimey]

Sure, thanks. The log seems oddly formatted? But here is what spit out from the command line you specified.

Debug1a.zip

Edit: There was an error in the first copy of the above log.
Updated Debug.zip w/ Debug1a.zip.

It could be that something is off with their encode of just this particular session.
For comparison, here is the debug log from FlvFixer.php for the .flv file joined by binary file copy rather than ffmpeg.

Debug2.zip

[KSV]

i have analyzed your logs. strangely the file concatenated by ffmpeg is missing most of audio video data contained in 0000.flv but when i try it on my side everything seems ok. are you sure you are using the latest version of ffmpeg?

[blimey]

hmm. that's kinda odd. I'm at a loss.

I had tried several versions of ffmpeg along the way while working with this particular set of files from this session.

I just used the newest version:
ffmpeg version N-60240-ge1cb6dc
built on Jan 29 2014 22:01:42 with gcc 4.8.2. (GCC)

Are you using the same version of ffmpeg referenced above? And it produced a ffmpeg concatenated file with a length of 01:32:28 (or 01:32:27 on some players)?

For anyone else following this thread, use the ffmpeg method shown in ksv's example. If the time length of the produced file matches the web player (or is -1 second off), it is very likely synced properly. In all cases, this binary file copy method will work, but for some (possibly many) sessions, the offset may need to be changed from 32, to quite often 13, but possibly other values as well.

[sebastian]

Final_fixed.flv Time length: 01:35:34
Final.mkv Time length: 01:35:34

same here with using KSVs batch. :-(

[blimey]

Originally Posted by sebastian

Final_fixed.flv Time length: 01:35:34
Final.mkv Time length: 01:35:34

same here with using KSVs batch. :-(

Well, could you test my batch too? It only takes a few seconds. Just copy the batch file and sfk169.exe into a folder with 0000.flv - 0462.flv (the .flv files processed already with BeatConvert.php, not the .beat files) from the timewarp-2012/marcel-fengler session.

Ffmpeg will quite often work with many sessions, but I have found about a dozen sessions that it has problems with, although this is probably not a problem with ffmpeg itself. In all cases, the binary file copy method will work, but for some (possibly many) sessions, the offset may need to be changed from 32, to quite often 13, but possibly other values as well. When you do the binary file copy method, you have a copy of the server file as they muxed it - as there is no remuxing involved.

[sebastian]

@blimey: joined.flv is 01:32:27, thanks! :-)

Anyone knows (if we can't find a solution that works for all types), how I can easily see if I should use your batch, KSVs batch, or even have to alter your batch to a different size of byte to strip?