Audio/video stream recording forums (http://stream-recorder.com/forum/index.php)
-   Video stream recording (http://stream-recorder.com/forum/forumdisplay.php?f=4)
-   -  

How to use iView (ABC Australia) downloaders

(http://stream-recorder.com/forum/showthread.php?t=7039)

Yansky 03-11-2011 11:35 PM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by gxdata (Post 26076)
The blog post from Rick Strahl indicates the same, I think. But I am using WebClient() in .NET and don't see similar switches. The stream that I read is UTF-8 encoded, and gzip.exe / ungzip.exe doesn't recognize its format. I'm having a mental block atm.

What about this: http://msdn.microsoft.com/en-us/libr...=vs.80%29.aspx

Yansky 03-11-2011 11:45 PM

Re: How to use iView (ABC Australia) downloaders


 
Slightly OT, but check this out - http://forums.whirlpool.net.au/forum....cfm?t=1656227

I have to pay now to get the gcc compiler for OSX.

Anywho 03-11-2011 11:50 PM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by gxdata (Post 26076)
They seem to be switching from GZ to text mode quite often, so the test for the "[{" characters will be needed at each new catalog fetch. I'm not sure what they are doing with the Series JSON files.

With the batch file downloader, I've been using:

Code:

wget --accept "text/html;level=1" -O index.json http://tviview.abc.net.au/iview/api2/?seriesIndex
This seems to work at forcing their server to spit back the text only. (wget allows you to mess with the headers if you need to ;))

mce 03-11-2011 11:52 PM

Re: How to use iView (ABC Australia) downloaders


 
I've had the switching during my development too.

At one point there, 3 out of 5 requests were gzip the others plain.

gxdata 03-11-2011 11:55 PM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Yansky (Post 26078)

haha - I was about to amend my last post, having just read this -

Quote:

A WebClient instance does not send optional HTTP headers by default. If your request requires an optional header, you must add the header to the Headers collection.
- and a link to a similar page.

Thanks for that!

Yansky 03-12-2011 12:00 AM

Re: How to use iView (ABC Australia) downloaders


 
Any reason you guys use wget instead of curl ? Nothing against wget, just curious.

(i love curl ) :p

Anywho 03-12-2011 12:25 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Yansky (Post 26083)
Any reason you guys use wget instead of curl ? Nothing against wget, just curious.

(i love curl ) :p

Dunno! It's always been there in the distributable and never really bothered to look at anything else. From memory, I think it was because of the DLL hell - the fluff on the side you needed to get it to work which would cause the package size to blow out. wget is only 159K and doesn't use extra fluff.

I did use it in the SBS_DL_LINUX downloader... Should probably have another look at it - pretty sure it does gzip.

[edit: Looking at a 386KB version right now...]

Anywho 03-12-2011 12:52 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Yansky (Post 26083)
Any reason you guys use wget instead of curl ?

Have tried a "simple" version of it from here. This one doesn't have the SSL fluff on it so it is a fair bit cheaper - but it seems to do the job I think.

Thank you Yansky for the reminder!!!

I'll just pack this up into an archive with the newer RTMPdump and the edited batch file.

Anywho 03-12-2011 01:16 AM

Re: How to use iView (ABC Australia) downloaders


 
Newer version of iView Downloader (batch file version) - v15.82 is here.
  • Updated batch file for latest changes (so far).
  • Newer version of RTMPdump v2.3 (r557) has been included - fixes resuming in some FLV files.

iView_GetSRT has been updated too. You can get it from here.

Updated iView_Later to v1.2. You can get it from here.

(Note that the web page download titles appear to be wrong. The files you download are updated but the page titles haven't filtered to all the servers just yet.)

gxdata 03-12-2011 04:34 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Anywho (Post 26094)
Newer version of iView Downloader (batch file version) - v15.82
  • Updated batch file for latest changes (so far).
  • Newer version of RTMPdump v2.3 (r557) has been included - fixes resuming in some FLV files.
iView_GetSRT has been updated too.
Updated iView_Later to v1.2.

Great stuff!

I have used your ungzip.exe with both gzip-format and 'proper' JSON files: it plays well with text (uncompressed) catalog files - I can save a (text) JSON file as catalog.gz and ungzip.exe catalog.gz catalog.json results in a usable file.

mce 03-12-2011 05:20 PM

Re: How to use iView (ABC Australia) downloaders


 
I'm slow to release an updated iVN this time as I changed from WebClient() to HttpWebRequest() and the bugger is timing out 10% of the time now. :(

servalan 03-12-2011 06:56 PM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Anywho (Post 26094)
Newer version of iView Downloader (batch file version) - v15.82 is here.

Many thanks - iViewDL all working fine here again.

mce 03-12-2011 07:08 PM

Re: How to use iView (ABC Australia) downloaders


 
Update to iViewNapper:

Version 20110312.

* Fix for change to gzip encoding of web pages.
* Resume working again.

Now using RTMPDUMP V2.3r557.

Thanks to all in this thread for ideas. :)

gxdata 03-12-2011 09:35 PM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by mce (Post 26114)
I'm slow to release an updated iVN this time as I changed from WebClient() to HttpWebRequest() and the bugger is timing out 10% of the time now. :(

I decided to use WebClient().DownloadFile(index.gz) and then use ungzip.exe to unpack the resulting file.
Even if it is actually text-format json, the ungzip.exe utility produces a nice json file (text to text, if you get what I mean).
Not pretty, but it works.
Besides that, I couldn't get the HttpWebRequest to work.

They seem to be using the text format consistently now (not that I have kept track since mid Saturday).

The series and episode naming stinks - almost as bad as Plus7. I think regular iView Player users will get p*ssed off pretty soon.

Anywho 03-13-2011 04:14 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by servalan (Post 26117)
Many thanks

You're welcome :)

Quote:

iViewDL all working fine here again.
Good to hear ;)

Anywho 03-13-2011 04:15 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by gxdata (Post 26103)
Great stuff!

I have used your ungzip.exe with both gzip-format and 'proper' JSON files: it plays well with text (uncompressed) catalog files - I can save a (text) JSON file as catalog.gz and ungzip.exe catalog.gz catalog.json results in a usable file.

Thanks! Glad it's useful to you :)

gxdata 03-14-2011 03:50 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by mce (Post 26118)
Update to iViewNapper:

Version 20110312.

* Fix for change to gzip encoding of web pages.
* Resume working again.

Now using RTMPDUMP V2.3r557.

Thanks to all in this thread for ideas. :)

Small error: If the show does not have a .SRT (closed caption / sub-titles) file, a messagebox pops up. The easy work-around (for users) is to turn 'Get subtitles' OFF - it is ON by default - but they would have to know where to do this. I think I requested it be ON by default, some months ago.
Perhaps best to just remove the messagebox on 404?

blood.eaglz 03-14-2011 05:58 AM

Re: How to use iView (ABC Australia) downloaders


 
Thanks Anywho and gxdata for identifying and countering the latest defense by iview, and thanks mce for the quick patch to iviewnapper.
------------------------------------------------------------------------

mce, I thought it was about time I reiterated my feature request list, I remember I was supposed to a few months ago to remind you....

1)I still would like a way of defining the default action of iviewnapper's srt download, preferably without the need for a new external file, so a parse exe name with a switch on the end eg iviewnapper.exe = srt
iviewnapper-ns.exe = no srt

2) for sbs and p7 nappers to be brought up to/to look for the same version of rmtpdump as iviewnapper so can go from 3 versions to 1.

3)to remove or at least reduce the focus lock on iviewnapper so at the very least can minimise the program without having to use "show desktop" or the hit an miss alt+tab. (you seemed to like this the first time round as you said it simplified the code for you)


hope I didn't come off too pushy :)

gxdata 03-14-2011 06:59 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by blood.eaglz (Post 26153)
1)I still would like a way of defining the default action of iviewnapper's srt download, preferably without the need for a new external file, so a parse exe name with a switch on the end
eg iviewnapper.exe = srt
iviewnapper-ns.exe = no srt

2) for sbs and p7 nappers to be brought up to/to look for the same version of rmtpdump as iviewnapper so can go from 3 versions to 1.

3)to remove or at least reduce the focus lock on iviewnapper so at the very least can minimise the program without having to use "show desktop" or the hit an miss alt+tab. (you seemed to like this the first time round as you said it simplified the code for you)

#2 - I use rtmpdump v2.3, for all 3 applications (and for the Batch Downloader for iView). I rename rtmpdumpv2.3.exe to the name expected by the program - eg, p7Napper expects v2.2c (of course, the name has to be exactly what is hard-coded into the respective application). There is a better way, of course - user settings for the applications.

#1 - There is a setting in iViewNapper for subtitles - it is ON by default. I guess a command-line switch might be used (one application, you create a shortcut with the appropriate switch), but again I would favour user settings.

#3 - Would you want to minimize the rtmpdump black box, too?

blood.eaglz 03-14-2011 07:09 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by gxdata (Post 26166)
#2 - I use rtmpdump v2.3, for all 3 applications (and for the Batch Downloader for iView). I rename rtmpdumpv2.3.exe to the name expected by the program - eg, p7Napper expects v2.2c (of course, the name has to be exactly what is hard-coded into the respective application). There is a better way, of course - user settings for the applications.

#2 what I would like is for all 3 to use a SINGLE rmtpdump exe rather than have to have a separate one for all 3, if it wasn't for the possibility of one or two being incompatible with a newer version I would suggest they simply looked for rtmpdump.exe

Quote:

Originally Posted by gxdata (Post 26166)
#1 - There is a setting in iViewNapper for subtitles - it is ON by default. I guess a command-line switch might be used (one application, you create a shortcut with the appropriate switch), but again I would favour user settings.

#1 Yep I know its there but it is not a STATIC setting it resets every time you close, and as I rarely if ever use the srts it add more clutter to the folder which although easily deleted would rather not have present in the first place. as to how I suggested be implemented I didn't mean by shortcut I meant in a style like PictureResizer does where what the exe itself is named determines how it operates.


Quote:

Originally Posted by gxdata (Post 26166)
#3 - Would you want to minimize the rtmpdump black box, too?

#3 already can (and do), unlike iviewnapper itself it doesn't have a focus lock :)

gxdata 03-14-2011 07:27 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by blood.eaglz (Post 26167)
#2 what I would like is for all 3 to use a SINGLE rmtpdump exe rather than have to have a separate one for all 3, if it wasn't for the possibility of one or two being incompatible with a newer version I would suggest they simply looked for rtmpdump.exe

Well, that would be possible of course. In effect, I use just rtmpdump v2.3 - for p7 it is renamed as rtmpdump v2.2c (not its exact name, but you get the idea), for SBS it is called rtmpdump v2.2d.
I have confidence that v2.3 works OK for all 3 nappers.

With user settings, the programmer allows some flexibility to change, over time. The exact name and location of the rtmpdump file to be used is stored in a settings file. Its location can be secured so that inexperiened users don't kill the settings, or change them to make the application unuseable. I can understand why mce might be reluctant to take this approach, but it can be made foolproof.

If all 3 nappers expected to see rtmpdump.exe in the working folder (ie, there is none of the win32-2.2c stuff in the filename), then as a new rtmpdump is released and recommended, you would download it (for arguments' sake, let us say it is v2.3a) and name it as rtmpdump.exe, replacing the older file (which might actually be v2.2c).

Is this more than obvious?

gxdata 03-14-2011 07:36 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by blood.eaglz (Post 26167)
#1 Yep I know its there but it is not a STATIC setting it resets every time you close, and as I rarely if ever use the srts it add more clutter to the folder which although easily deleted would rather not have present in the first place. as to how I suggested be implemented I didn't mean by shortcut I meant in a style like PictureResizer does where what the exe itself is named determines how it operates.

The user settings behave like (for example) the last size that the MS Word window was, which is 'remembered' when you next open it. So, if you change the options and save them, when you use the napper next week it will remember the settings that you saved.

With a commandline switch, you would make a shortcut that in effect determined the default captions option -

iVN /captions=ON
or if you preferred it,
iVN /captions=OFF

(I'm using a shorthand, not the actual stuff that would go into a shortcut - but it is super-easy, and saves mce having to compile 2 versions of each release).

Unfortunately, I can't insert or attach an image to illustrate this with a partial screenshot.

vipher 03-14-2011 08:32 AM

Re: How to use iView (ABC Australia) downloaders


 
I've been away for a while. Good news is that I've just updated the Linux Script to combat the new changes made to iView. New script is v6.9 and it's available on the wiki. Thanks to Anywho and gxdata for figuring out what was done. I'll see about an OSX version tomorrow, it should be as simple as changing the rtmpdump binary.

blood.eaglz 03-14-2011 11:22 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by gxdata (Post 26169)
saves mce having to compile 2 versions of each release).

my suggestion doesn't require 2 releases to be compiled all it requires is that at start up it parses its own name
if name = "iviewnapper.exe" then srt=on
if name = "iviewnapper-ns.exe" then srt = off

*of course "iveiwnapper" would be replaced with either the current version number or a wildcard.

blood.eaglz 03-14-2011 11:28 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by gxdata (Post 26168)
If all 3 nappers expected to see rtmpdump.exe in the working folder (ie, there is none of the win32-2.2c stuff in the filename), then as a new rtmpdump is released and recommended, you would download it (for arguments' sake, let us say it is v2.3a) and name it as rtmpdump.exe, replacing the older file (which might actually be v2.2c).

Is this more than obvious?

Well considering that I included that exact Idea in my post Yes

Quote:

Originally Posted by blood.eaglz (Post 26167)
#2 what I would like is for all 3 to use a SINGLE rmtpdump exe rather than have to have a separate one for all 3, if it wasn't for the possibility of one or two being incompatible with a newer version I would suggest they simply looked for rtmpdump.exe

but as had written there is the possibility that one or two of the nappers may NOT be compatible, and if the situation arose, like it has before, where you had to use a newer version for one it could cause all sorts of head aches, with multiple directories and debug woes.

gxdata 03-14-2011 08:45 PM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by blood.eaglz (Post 26172)
Well considering that I included that exact Idea in my post Yes

but as had written there is the possibility that one or two of the nappers may NOT be compatible, and if the situation arose, like it has before, where you had to use a newer version for one it could cause all sorts of head aches, with multiple directories and debug woes.

I wasn't being rude, sorry if it sounded so - after writing it, I thought that my explanation sounded a bit pedantic & detailed, hence the 'obvious' remark.
Actually, over the last year I have suggested a few things to mce - including the single rtmpdump.exe, probably - but his experience (and that of others here) with the changes of iView/SBS/Plus!7 over that time count for more than mine, so whatever the community can come up with is fine by me.

Others can take on board what you've written.

mce 03-15-2011 12:30 AM

Re: How to use iView (ABC Australia) downloaders


 
blood.eaglz & gxdata

Yes something has to be done, it was supposed to be a simple little program that did one thing and worked well.

Sometimes I wonder if I achieved any of that. :)

1. I'll implement cross session settings (?) soon.

2. As new versions of the Nappers come out they get the latest rtmpdump that works for them at that time. Due to the way the other Nappers don't need much updating, they can seem like they are left behind.

If *someone* can tell me that the newest rtmpdump works on all the Nappers just fine it will save me a weekend of testing and speed up the process a bit.

3. Thanks gxdata for the 404 bug report, I'll fix it tonight if possible.

4. Regarding the focus lock problem, I haven't forgot about it, indeed it is a constant worry, but I will continue to look for a way to fix it.

Ferret 03-15-2011 03:04 AM

Re: How to use iView (ABC Australia) downloaders


 
Something I have noticed is that the double hyphen crash seems to be back.
for example:
Four Corners 14/03/11 : BP: Billion Blowout
will not download correctly and the Subtitles files if they are zero bytes are not deleted.

mce 03-15-2011 03:17 AM

Re: How to use iView (ABC Australia) downloaders


 
Version 20110315.

* Fix 404 error box for no .srt file.
* Subtitles state remembered between sessions.

[Edit]

Four Corners BP file OK on the Nappers.

Anywho 03-15-2011 03:51 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by mce (Post 26199)
If *someone* can tell me that the newest rtmpdump works on all the Nappers just fine it will save me a weekend of testing and speed up the process a bit.

Currently no problems I've found with them using the latest RTMPdump (v2.3-r557). :)

Same goes for iViewFox plugin ;)

blood.eaglz 03-15-2011 03:52 AM

Re: How to use iView (ABC Australia) downloaders


 
Great mce thanks

I have done some light testing on the other nappers and so far they are working perfectly with rtmpdump-win32-2.3r557.exe

edit: testing done so far
SBSnapper - partial downloads 5, full downloads 2, resumes 0. all at 1000k quality
p7napper - partial downloads 3, full downloads 1, resumes 0.

Anywho 03-15-2011 03:54 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Ferret (Post 26200)
Four Corners 14/03/11 : BP: Billion Blowout
will not download correctly and the Subtitles files if they are zero bytes are not deleted.

Yeah thanks for that one. The command prompt would squeeze its cheeks at that one... I'll have a go at it tonight if I get some more time. Latest would be tomorrow arvo.

Ferret 03-15-2011 04:40 AM

Re: How to use iView (ABC Australia) downloaders


 
Looks like they've made further changes to the names in the XML - they've now stripped out the series names completely. :(

vipher 03-15-2011 05:36 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Ferret (Post 26207)
Looks like they've made further changes to the names in the XML - they've now stripped out the series names completely. :(

Yeah I've seen that. Thankfully it's not to hard to fix in the Linux & OSX script downloaders. I've already got a fix, but I'm also working on some other updates so it will be a few hours more before I upload an updated script.

Edit: Interestingly it also appears that none of the series indexes are gzipped anymore either.

Anywho 03-15-2011 05:51 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Ferret (Post 26200)
Something I have noticed is that the double hyphen crash seems to be back.
for example:
Four Corners 14/03/11 : BP: Billion Blowout
will not download correctly and the Subtitles files if they are zero bytes are not deleted.

Should be fixed now in v15.83 here. The minimal version (with no required support files) you can find here.

blood.eaglz 03-15-2011 07:00 AM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by mce (Post 26199)
4. Regarding the focus lock problem, I haven't forgot about it, indeed it is a constant worry, but I will continue to look for a way to fix it.

haven't used SBSnapper for quite some time so didn't realise it too has the focus lock, so if you manage to to find a fix, please also apply it to SBSnapper :)

*also have posted a guess as to why SBS new videos have the audio spike in the SBSnapper whirlpool thread

vipher 03-15-2011 08:55 AM

Re: How to use iView (ABC Australia) downloaders


 
Some more updates to the Linux & OSX command line downloaders:
- Files are now saved with the actual episode name instead of the rather meaningless server name.
- Fix for the new JSON format (Series names being moved).
- Completed files are removed from the download list.

Linux v7.0
OSX v7.0

The OSX one is untested since I don't have a Mac handy but the Linux one works for me on Fedora 14.

Anywho 03-16-2011 01:48 AM

Re: How to use iView (ABC Australia) downloaders


 
Updated iView Downloader batch file to v15.84. Full version here, minimal (batch file only) version here.

Also updated GetSRT as well.

"Adds series name to filename."

Yansky 03-16-2011 05:45 PM

Re: How to use iView (ABC Australia) downloaders


 
I've made another firefox extension to make it easier to extract the mp4 video from the flv files.

https://addons.mozilla.org/en-US/fir...lvuncontainer/

It basically is just a front end for the "ffmpeg -i filename -vcodec copy -acodec copy newfilename" command.

Lemme know how it goes. I don't have a windows machine atm, so if anyone could confirm that it works ok on windows, that'd be great.

Note: it only works on Firefox 4.

gxdata 03-16-2011 06:55 PM

Re: How to use iView (ABC Australia) downloaders


 
Quote:

Originally Posted by Anywho (Post 26234)
Updated iView Downloader batch file to v15.84. Full version here, minimal (batch file only) version here.

Also updated GetSRT as well.

"Adds series name to filename."

Excellent - thx once again.


All times are GMT -6. The time now is 06:52 PM.