How to get complete download from this archive?

mariner · #1 07-25-2013, 05:21 AM

Greetings.

First, many thanks to KSV for his effort to improve the rtmpdump binaries.

Having some issues when downloading streams from this archive:

http://meloman.ru/webcasts/

1. Downloaded files are often incomplete with video drop-outs. Many attempts are required before two downloaded files agree in both size and hash. What are the commands to ensure a proper download?

2. The command used are generated by RTMPDumpHelper, but with the -v flag added. If the -v flag is absent, the downloading process gets interrupted after a few seconds and start over, again and again. It seems counter-intuitive to use the -v flag here as the streams are not "live". How does one get around this?

Appreciate if anyone can take a look at this. Many thanks for your kind assistance.

mariner · #2 07-27-2013, 11:54 AM

Here's an example of the command line used.

1. Without the -v parameter, "Stream EOF" occurred at around 0.6%, and downloading would stop and resume from the beginning. This would be repeated over and over again.

What's the cause of this?

Code:

rtmpdump -r "rtmp://88.212.207.11:1935/archive" -a "archive" -f "WIN 11,8,800,94" -W "http://echomsk.onlinetv.ru/hd/player.swf" -p "http://echomsk.onlinetv.ru/hd/17072013_1/" -y "mp4:filarmonia/17072013_1hd.mp4" -o "2013-07-27_05-35-44_17072013_1hd.flv"

Code:

........
25973.210 kB / 65.68 sec (0.6%)
DEBUG: HandleCtrl, received ctrl, type: 31, len: 6
DEBUG: HandleCtrl, Stream BufferEmpty 1
DEBUG: HandleCtrl, received ctrl, type: 32, len: 6
DEBUG: HandleCtrl, Stream BufferReady 1
26808.054 kB / 67.64 sec (0.6%)
DEBUG: HandleCtrl, received ctrl, type: 31, len: 6
DEBUG: HandleCtrl, Stream BufferEmpty 1
DEBUG: HandleCtrl, received ctrl, type: 1, len: 6
DEBUG: HandleCtrl, Stream EOF 1
DEBUG: RTMP_ClientPacket, received: invoke 174 bytes
DEBUG: (object begin)
DEBUG: Property: <Name:            no-name, STRING:     onStatus>
DEBUG: Property: <Name:            no-name, NUMBER:     0.00>
DEBUG: Property: NULL
DEBUG: Property: <Name:            no-name, OBJECT>
DEBUG: (object begin)
DEBUG: Property: <Name:              level, STRING:     status>
DEBUG: Property: <Name:               code, STRING:     NetStream.Pause.Notify>
DEBUG: Property: <Name:        description, STRING:     Pausing filarmonia/2005_1hd.mp4.>
DEBUG: Property: <Name:            details, STRING:     filarmonia/2005_1hd.mp4>
.......

2. If -v flag was used, the download would complete, but not without error. How does one ensure error free download?

3. It seems download is taking place at real time speed. Is there a way to speed things up?

Many thanks and best regards.

mariner · #3 08-05-2013, 09:14 AM

Appreciate if anyone could advise the correct parameters for error free download.

chap · #4 08-05-2013, 11:21 AM

use the -v flag

chap · #5 08-05-2013, 11:35 AM

try alternative

Code:

http://cdn0.onlinetv.ru:1935/archive/_definst_/filarmonia/17072013_1hd.mp4/playlist.m3u8

mariner · #6 08-06-2013, 05:36 AM

Quote:

Originally Posted by chap

use the -v flag

Quote:

Originally Posted by chap

try alternative

Code:

http://cdn0.onlinetv.ru:1935/archive/_definst_/filarmonia/17072013_1hd.mp4/playlist.m3u8

Thanks for the kind reply, chap.

1. As I pointed out in the earlier post, streams downloaded using the -v flag are prone to errors. You may have tried and found out the same stream downloaded at different time usually differ in size/file hash. Over here, I had to keep repeating the download until hopefully there's a match.

Would you know what may be the cause? How does one know if the downloaded stream is error free?

2. There's another issue with using the -v flag: streams are downloaded at real-time speed. How to speed things up?

3. Is there any guideline on the use of -v flag? Take a look a the two following streams from the same site. The first requires the -v flag, but the second does not, and is obviously downloaded much faster.

-v required
http://www.medici.tv/#!/batiashvili-...chubert-franck

-v not required
http://www.medici.tv/#!/evgeny-sudbi...l-2013-recital

4. Not sure what to do with the alternate link you provided. Any help?

Many thanks and best regards.

chap · #7 08-06-2013, 11:33 AM

Quote:

Originally Posted by mariner

http://www.medici.tv/#!/batiashvili-...chubert-franck
http://www.medici.tv/#!/evgeny-sudbi...l-2013-recital

Code:

http://usp-01.dmcloud.net/4cb48472f325e179ec000006/51ed3af694a6f605a4000016/mp4_h264_aac_hd-1374504514.mp4?e=1375820698&st=U8NCO4Rw8Jklrq2j9t1ZWw

Code:

http://cdn.dmcloud.net/route/4cb48472f325e179ec000006/51ebdb1a94a6f652e0000002/mp4_h264_aac_hq.mp4?auth=1375817407-0-mtr4tuzn-b0f00411bccaf124854c93f6628a066f

Quote:

Originally Posted by mariner

Not sure what to do with the alternate link you provided.

use ffmpeg,vlc

mariner · #8 08-07-2013, 09:59 AM

Thanks for the reply, chaps.

Quote:

Originally Posted by chap

Code:

http://usp-01.dmcloud.net/4cb48472f325e179ec000006/51ed3af694a6f605a4000016/mp4_h264_aac_hd-1374504514.mp4?e=1375820698&st=U8NCO4Rw8Jklrq2j9t1ZWw

Code:

http://cdn.dmcloud.net/route/4cb48472f325e179ec000006/51ebdb1a94a6f652e0000002/mp4_h264_aac_hq.mp4?auth=1375817407-0-mtr4tuzn-b0f00411bccaf124854c93f6628a066f

1. Token may have expired. Can you kindly explain how to obtain these parameters?

Quote:

use ffmpeg,vlc

2. Tried the following ffmpeg command line. What other parameters are required?

Code:

ffmpeg -i http://cdn0.onlinetv.ru:1935/archive/_definst_/filarmonia/06112012_1hd.mp4/playlist.m3u8 lieb.mp4

Code:

D:\rdump\ff>ffmpeg -i http://cdn0.onlinetv.ru:1935/archive/_definst_/filarmonia/06112012_1hd.mp4/playlist.m3u8 lieb.mp4
ffmpeg version N-55337-gb45b1d7 Copyright (c) 2000-2013 the FFmpeg developers
  built on Aug  6 2013 22:49:23 with gcc 4.7.3 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvo-aacenc --enable-libvo-
amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --
enable-libxvid --enable-zlib
  libavutil      52. 41.100 / 52. 41.100
  libavcodec     55. 22.100 / 55. 22.100
  libavformat    55. 13.102 / 55. 13.102
  libavdevice    55.  3.100 / 55.  3.100
  libavfilter     3. 82.100 /  3. 82.100
  libswscale      2.  4.100 /  2.  4.100
  libswresample   0. 17.103 /  0. 17.103
  libpostproc    52.  3.100 / 52.  3.100
Input #0, hls,applehttp, from 'http://cdn0.onlinetv.ru:1935/archive/_definst_/filarmonia/06112012_1hd.mp4/playlist.m3u8':
  Duration: 02:02:08.00, start: 0.000000, bitrate: N/A
  Program 0
    Metadata:
      variant_bitrate : 3481194
    Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1920x1080
 [SAR 1:1 DAR 16:9], 12.50 fps, 25 tbr, 90k tbn, 50 tbc
    Metadata:
      variant_bitrate : 3481194
    Stream #0:1: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 174 kb/s
    Metadata:
      variant_bitrate : 3481194
[libx264 @ 0033f420] using SAR=1/1
[libx264 @ 0033f420] using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
[libx264 @ 0033f420] profile High, level 4.0
[libx264 @ 0033f420] 264 - core 135 r2345 f0c1c53 - H.264/MPEG-4 AVC codec - Cop
yleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deb
lock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 m
e_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chro
ma_qp_offset=-2 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 i
nterlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1
b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenec
ut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=
0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'lieb.mp4':
  Metadata:
    encoder         : Lavf55.13.102
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 12800 tbn, 25 tbc
    Metadata:
      variant_bitrate : 3481194
    Stream #0:1: Audio: aac (libvo_aacenc) ([64][0][0][0] / 0x0040), 48000 Hz, stereo, s16, 128 kb/s
    Metadata:
      variant_bitrate : 3481194
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx264)
  Stream #0:1 -> #0:1 (aac -> libvo_aacenc)
Press [q] to stop, [?] for help
frame=   32 fps=0.0 q=0.0 size=       0kB time=00:00:01.33 bitrate=   0.3kbits/s
frame=   45 fps= 37 q=0.0 size=       0kB time=00:00:01.82 bitrate=   0.2kbits/s
frame=   47 fps= 26 q=0.0 size=       0kB time=00:00:01.88 bitrate=   0.2kbits/s
frame=   51 fps= 22 q=28.0 size=     284kB time=00:00:02.10 bitrate=1108.1kbits/
frame=   56 fps= 19 q=28.0 size=     355kB time=00:00:02.29 bitrate=1268.5kbits/
....
frame=  559 fps=7.8 q=28.0 size=   10963kB time=00:00:22.36 bitrate=4015.5kbits/
frame=  565 fps=7.8 q=28.0 size=   11053kB time=00:00:22.64 bitrate=3998.8kbits/
frame=  573 fps=7.8 q=28.0 size=   11176kB time=00:00:22.96 bitrate=3987.0kbits/
[mpegts @ 03365600] PES packet size mismatch
frame=  581 fps=7.9 q=28.0 size=   11309kB time=00:00:23.26 bitrate=3982.6kbits/
[h264 @ 0a399fc0] error while decoding MB 38 31, bytestream (-8)
[h264 @ 0a399fc0] Cannot use next picture in error concealment
[h264 @ 0a399fc0] concealing 4451 DC, 4451 AC, 4451 MV errors in P frame
frame=  582 fps=5.4 q=28.0 size=   11326kB time=00:00:23.26 bitrate=3988.3kbits/
frame=  582 fps=5.2 q=28.0 size=   11326kB time=00:00:23.30 bitrate=3981.0kbits/
[mpegts @ 03365600] PES packet size mismatch
[h264 @ 0a399fc0] error while decoding MB 104 55, bytestream (-15)
[h264 @ 0a399fc0] Cannot use next picture in error concealment
[h264 @ 0a399fc0] concealing 1505 DC, 1505 AC, 1505 MV errors in P frame
frame=  601 fps=5.4 q=28.0 size=   11639kB time=00:00:24.00 bitrate=3971.3kbits/
frame=  601 fps=5.3 q=28.0 size=   11639kB time=00:00:24.03 bitrate=3967.8kbits/
frame=  601 fps=5.1 q=28.0 size=   11639kB time=00:00:24.05 bitrate=3964.2kbits/
.........
frame= 2101 fps=9.0 q=28.0 size=   16809kB time=00:01:24.08 bitrate=1637.6kbits/
frame= 2111 fps=9.1 q=28.0 size=   16810kB time=00:01:24.44 bitrate=1630.7kbits/
frame= 2114 fps=9.0 q=28.0 size=   16810kB time=00:01:24.59 bitrate=1627.8kbits/
frame= 2117 fps=9.0 q=28.0 size=   16810kB time=00:01:24.70 bitrate=1625.8kbits/
frame= 2120 fps=9.0 q=28.0 size=   16811kB time=00:01:24.85 bitrate=1623.0kbits/
[mpegts @ 03365600] PES packet size mismatch
[h264 @ 0a399fc0] frame= 2123 fps=9.0 q=28.0 size=   16811kB time=00:01:24.89 bi
error while decoding MB 55 36, bytestream (-9)
[h264 @ 0a399fc0] Cannot use next picture in error concealment
[h264 @ 0a399fc0] concealing 3834 DC, 3834 AC, 3834 MV errors in P frame
frame= 2123 fps=8.9 q=28.0 size=   16811kB time=00:01:24.91 bitrate=1621.8kbits/
frame= 2123 fps=8.7 q=-1.0 Lsize=   16998kB time=00:01:36.00 bitrate=1450.3kbits
/s dup=1430 drop=0
video:16531kB audio:431kB subtitle:0 global headers:0kB muxing overhead 0.206651
%
[libx264 @ 0033f420] frame I:9     Avg QP:16.75  size:331770
[libx264 @ 0033f420] frame P:936   Avg QP:20.36  size: 13690
[libx264 @ 0033f420] frame B:1178  Avg QP:27.49  size:   957
[libx264 @ 0033f420] consecutive B-frames: 24.8%  1.5%  6.2% 67.5%
[libx264 @ 0033f420] mb I  I16..4:  4.2% 37.2% 58.6%
[libx264 @ 0033f420] mb P  I16..4:  0.2%  0.5%  0.2%  P16..4: 14.4%  1.5%  1.2% 0.0%  0.0%    skip:81.9%
[libx264 @ 0033f420] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  2.9%  0.2%  0.1%
 direct: 0.1%  skip:96.6%  L0:47.0% L1:45.9% BI: 7.1%
[libx264 @ 0033f420] 8x8 transform intra:43.9% inter:35.1%
[libx264 @ 0033f420] coded y,uvDC,uvAC intra: 78.2% 77.0% 51.2% inter: 2.9% 2.5% 0.2%
[libx264 @ 0033f420] i16 v,h,dc,p: 38% 33% 13% 16%
[libx264 @ 0033f420] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 22% 22%  4%  5%  5%  5% 5%  7%
[libx264 @ 0033f420] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 32% 24%  9%  5%  6%  6%  6% 6%  7%
[libx264 @ 0033f420] i8c dc,h,v,p: 46% 27% 21%  6%
[libx264 @ 0033f420] Weighted P-Frames: Y:0.5% UV:0.1%
[libx264 @ 0033f420] ref P L0: 83.5%  7.8%  6.3%  2.3%  0.1%
[libx264 @ 0033f420] ref B L0: 92.2%  7.4%  0.4%
[libx264 @ 0033f420] ref B L1: 93.3%  6.7%
[libx264 @ 0033f420] kb/s:1594.66
Received signal 2: terminating.
Terminate batch job (Y/N)?

3. How to get rid of the packet size mismatch error and others?
4. Download was about half real-time speed. How to speed things up?
5. Is FFMpeg doing re-encoding? How to simply copy the h264 video and aac audio to output?
6. While rtmpdump saves the output to flv, ffmpeg seems to accept only mp4. Which is correct?
7. How to save the log to a text file?
8. How to generate an error report and save it?

Many thanks and best regards.

chap · #9 08-07-2013, 12:24 PM

use

Code:

ffmpeg -i http://cdn0.onlinetv.ru:1935/archive/_definst_/filarmonia/06112012_1hd.mp4/playlist.m3u8 -c copy lieb.ts

Quote:

1. Token may have expired. Can you kindly explain how to obtain these parameters?

use user agent=ipad

mariner · #10 08-08-2013, 10:56 PM

Thanks chaps.

Quote:

Originally Posted by chap

use

Code:

ffmpeg -i http://cdn0.onlinetv.ru:1935/archive/_definst_/filarmonia/06112012_1hd.mp4/playlist.m3u8 -c copy lieb.ts

1. The 2hr long stream terminated at 12:24.60 with the following eror message:

Code:

.......
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66948480 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66948480 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66958080 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66958080 in the queue
frame=18600 fps=9.8 q=-1.0 size=  312743kB time=00:12:24.00 bitrate=3443.5kbits/s    
frame=18600 fps=9.7 q=-1.0 size=  312743kB time=00:12:24.02 bitrate=3443.4kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66963840 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66963840 in the queue
frame=18602 fps=9.7 q=-1.0 size=  313028kB time=00:12:24.08 bitrate=3446.3kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66969600 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66969600 in the queue
frame=18603 fps=9.7 q=-1.0 size=  313036kB time=00:12:24.12 bitrate=3446.2kbits/s    
frame=18605 fps=9.7 q=-1.0 size=  313047kB time=00:12:24.20 bitrate=3446.0kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66975360 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66975360 in the queue
frame=18605 fps=9.7 q=-1.0 size=  313056kB time=00:12:24.20 bitrate=3446.1kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66981120 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66981120 in the queue
frame=18606 fps=9.7 q=-1.0 size=  313064kB time=00:12:24.25 bitrate=3445.9kbits/s    
frame=18606 fps=9.7 q=-1.0 size=  313064kB time=00:12:24.27 bitrate=3445.8kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66986880 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66986880 in the queue
frame=18608 fps=9.7 q=-1.0 size=  313083kB time=00:12:24.32 bitrate=3445.8kbits/s    
frame=18610 fps=9.6 q=-1.0 size=  313095kB time=00:12:24.40 bitrate=3445.6kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66992640 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66992640 in the queue
frame=18610 fps=9.6 q=-1.0 size=  313103kB time=00:12:24.40 bitrate=3445.6kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66998400 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 66998400 in the queue
frame=18611 fps=9.6 q=-1.0 size=  313111kB time=00:12:24.44 bitrate=3445.5kbits/s    
frame=18613 fps=9.6 q=-1.0 size=  313123kB time=00:12:24.52 bitrate=3445.3kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 67004160 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 67004160 in the queue
frame=18613 fps=9.6 q=-1.0 size=  313133kB time=00:12:24.52 bitrate=3445.4kbits/s    
[mpegts @ 035e19c0] Continuity check failed for pid 256 expected 2 got 15
frame=18615 fps=9.6 q=-1.0 size=  313142kB time=00:12:24.60 bitrate=3445.1kbits/s    
[mpegts @ 035e19c0] Continuity check failed for pid 257 expected 10 got 4
[mpegts @ 035e19c0] PES packet size mismatch
frame=18615 fps=9.6 q=-1.0 size=  313142kB time=00:12:24.60 bitrate=3445.1kbits/s    
[mpegts @ 035e19c0] first_dts 0 not matching first dts 67011840 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 67011840 in the queue
[mpegts @ 035e19c0] first_dts 0 not matching first dts 67011840 in the queue
frame=18615 fps=9.6 q=-1.0 size=  313151kB time=00:12:24.60 bitrate=3445.3kbits/s    
[mpegts @ 0496f960] AAC bitstream not in ADTS format and extradata missing
av_interleaved_write_frame(): Invalid data found when processing input
[AVIOContext @ 003d3ec0] Statistics: 0 seeks, 25710 writeouts
[AVIOContext @ 003d8c20] Statistics: 172 bytes read, 0 seeks

2. Download speed was only 9fps.

3. The log file was too large to be attached. What sort of errors would it record?

Quote:

use user agent=ipad

4. Can you kindly elaborate? I'm using the firefox browser.