rtmpdump and data loss

[perazim]

Can somebody please explain to me why, when I download the same flv or mp4 file five times I receive five different file sizes and md5 sums? The files are downloaded one at a time and nothing else is running on computer. ADSL 15 Mbps line in middle of night. I looked at traffic and it is TCP not UDP.

I am experiencing a lot of dropped frames.

In the rtmp protocol, is there a feature that allows the server to drop frames in the stream? If so, is there a way to stop this or minimize it?

Thanks,

Perazim

[Ed999]

Be generous: give us a clue as to which of the million or so RTMP streams on the internet you're trying to download!

And which Operating System you're using!


In general terms, taking (as a random example) the BBC in London, television companies quite often offer a half dozen (or more!) different streams for each tv show on their website. Each of these is streamed at a different bitrate, so that a computer user can choose a stream which is slow enough for his internet connection.

This answers both of your questions at once!


1. It sounds as if you might be downloading different streams, instead of always getting the one that you want. Check that the -y option in RTMPDUMP (also known as the --playlist option), which allows you to specify one single stream, has been set. If it hasn't, just copy-and-paste the stream url you want, from the several contained in the --app string.


2. If you download a live stream, on a slow connection, you will (obviously) experience dropped frames, because the server is streaming faster than your connection can handle. Well, the same sort of thing can happen with an on-demand stream.

You might think you've got a 15 Mbps ADSL connection, but there are many bottlenecks on the internet that can slow the true connection speed way, way down. If you, in Australia (well, I don't know where you are, because you didn't say ) try to connect to the BBC in the UK (again, you didn't say ), international routing will kill the connection speed stone dead.

On the other hand, if you in the Bronx (!) connect to NBC in Manhattan, you might get a genuine 15 Mbps connection. But I'm guessing you're way out in the outback.

Amazingly, you might actually get a better result by downloading at the busiest time of the day!! The heavier load will slow the server down, so that it streams high bitrate on-demand files much slower than in the dead of night.

On the technical front, a live server (i.e. one serving a live stream) has a fixed output per second for each given bitrate stream, so the system is designed to drop frames if your connection can't keep up. An 800 Kbps stream will drop 50% of the frames if the connection is 400 Kbps, for example.

No, there's nothing you can do (except move house, closer to the server!). I don't think too many tv stations are streaming at above 15 Mbps, so the trouble is internet botlenecks, most likely.

[perazim]

Sorry, I had posted on several forums regarding this and didn't give this post all the details.

rtmpdump -V -f "LNX 10,1,85,3" -r rtmp://flash.glc.us.com/VOD -y Programmers/BillCloud/1/BillCloud_High_1.flv -o BillCloud_High_1.flv
(this was hand type from other system)

Running on Fedora 12, rtmpdump-2.3, server is FMS 3,0,1,123.

Material on server may be reused as long as it is without profit.

Purpose is to capture streaming on demand shows without frame drops. Server is in US, I am in Israel. 15Mbit ADSL. I capture stream several times and then check file sizes and if equal then run md5sum on files and check results. Likelihood that two files with dropped frames produce same md5 sums is near impossible.

Adobe server is version 3.0 so it does not have dynamic streaming so I can rule out different streams. Have tried all hours of day and night. 4am-9am local time is the only time I can capture without frame drops.

Regarding the frame drops, I see this is a tcp/ip connection (not udp) so the when the server decides to drop frames it must be happening before it leaves the server in the stream management area. I have looked at the Adobe support documents for this server and there doesn't appear to me to be anything I can do to change the server behaviour but I have come up with this idea: alter the rtmpdump code to periodically inquire of the server the dropped frame count for the current stream. This is property NetStreamInfo.droppedFrames. If this is non-zero then I can abort the current session and restart the capture until I receive it without dropped frames.

Know anything about rtmpdump internals?

[Ed999]

Unfortunately I don't use Linux, nor do I use v2.3 of rtmpdump. So I can't contribute anything useful to the solution of your problem.

Except to suggest that you contact Howard Chu.

He recently posted on the subject of rtmpdump and flvstreamer, asking that anyone with a feature request for the next version contact him. You can contact him through the project homepage.

Your problem sounds to me like a request for the program to provide additional features, not a complaint about a bug in the existing features. The program as it stands simply can't interrogate the server in the manner you describe. You need to make a 'new feature' request.

[perazim]

I have already contacted Howard regarding this and his response was to lower the size of the stream to get a good download. Unfortunately, I can't do this. I am using a brute force solution now where during times of low net activity, I repetitively download the file and then filesize check and compare md5sums on the files till I get a match. Wastes a lot of time and bandwidth but it does work.

Having a way to monitor the dropped frames from the server side would allow me to immediately abandon the current download when frames are dropped and start over.

Thanks.

[Ed999]

How are your programing skills?

Modifying rtmpdump is beyond me. But do you actually need to?

It seems to me that you might interrogate the server, using a simple Perl script. You could periodically ask it for the dropped frame count (say, every 60 seconds), to compare against the desired value of nil, and then fork.

If the result is nil, rtmpdump contains a --resume option which will enable you to resume the interrupted download at the point at which it left off to interrogate the server.

If the result is not nil, do not resume the download. Go to EXIT instead.

Sorry, my scripting days are over. But this seems like something Perl could do. All you are doing is launching the download using Perl (which is what Perl scripts such as get_iplayer.pl do), then stopping the download after 60 seconds, and then immediately interrogating the server. On receiving the response, Perl either issues a resume command (the original download command plus --resume) or it exits (and, optionally, tries again: which is just a matter of issuing the original download command again, but without adding the --resume option).

If you want to examine the Perl script called get_iplayer.pl (a popular choice for downloading rtmp from the BBC in London, under either Windows or Linux), to see how RTMP downloading is done using an ActivePerl script, search in Google for author "Phil Lewis" or for "GET_IPLAYER".

But are you overlooking an easier solution? RTMPDUMP doesn't seem to use the concept of a single stream. Because it's capable of resuming, it has no need to bother with that concept. Looking at your command line, it actually contains a potentially serious flaw, in that you don't include the standard --resume switch. Try including it. That switch is the built-in protection that's designed to prevent the loss of frames/packets.