You are here

Restart grab after 'Exception.Message: Out of memory'

15 posts / 0 new
Last post
Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months
Restart grab after 'Exception.Message: Out of memory'

I've noticed that when the critical error 'Exception.Message: Out of memory' occurs, WebGrab thinks its restarting the channel, but really isn't.  Can it rerun the current channel overwriting if necessary the partial grab? Or, add an option to clear the memory using a function within the siteini (perhaps a first or last step)?

 

FYI- I've attached the project tha brings me the error since the siteini has numerous of repetitive replaces.

Attachments: 
Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

i wouldnt call this a webgrab issue,its a device problem.

[Error   ] GC.GetTotalMemory 1077MB before collect

[Error   ] GC.GetTotalMemory 578MB after collect

doesnt look like you have any considerable amount of ram available to begin with,unless i am missing something?

what do you expect when your timespan is set to 60 days(which you will never get) as the max days is 31.

the obvious solutions to me is reduce the number of days your grabbing,why do you want 31,are you grabbing data once/mth?

 

 

Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months

Thanks for looking into this again.  I've tried many different scenarios and can't seem to find the answer.  I still encouter this memory problem when using only one siteini (mergexml) for one xmltv and lowering the days grabbed to 0 (same day).

Any other ideas?

 

Attachments: 
Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

[Warning ] !! SiteIni merge-xmltv_uk.ini in channelfile creation mode !!!

do i need to explain what that means?

also why are you changing the site_id=xx"

<channel update="i" site="merge-xmltv_uk" site_id="FOX" xmltv_id="4.1 Fox UK">4.1 Fox UK</channel>
<channel update="i" site="merge-xmltv_uk" site_id="Sky News HD" xmltv_id="99 Sky News HD">99 Sky News HD</channel>

<channel id="4.1 Fox UK">

<display-name lang="en">4.1 Fox UK</display-name>

<url>http://www.merge-xmltv</url>

</channel>

<channel id="99 Sky News HD">

<display-name lang="en">99 Sky News HD</display-name

><url>http://www.merge-xmltv</url>

</channel>

none of these look correct so u wont get any data..

 

Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months

I had already looked into both of these concerns extensively by trying multiple different methods (including the conventional method explained in merge-xmltv_info.howtosetup.txt).  I've somewhat ran out of ideas; hoping perhaps someone had come across a similar situation with the memory being used on the scale of 1.5GB RAM... Perhaps that is normal, since it's only being a noticeable nuissance on the largest source xml (roughly 70MB total in size).

1) ini is in creation mode since the source xml is changing very frequently and this saves me from changing the siteini and rerunning.  Once the source xml stables out, I'll comment this section.  I have tested with and without this section commented and I've concluded this is not the cause of the massive amount of RAM being used.

2) site_id has not changed.  Both xmltv_id and the display-name are changed to save me from needing to change those values again in another config running afterwards; combined with others to create one file (all of the runs use only merge-xmltv).  FYI- I am not running webgrab+ at all for grabbing from a website, I am using premade xmltv files and merging them into one.  I am getting data, and it shows on the log file.

I need to run them seperately and change the display-name since the source-id from one source has many the same source-id from other countries (like FOX).  This prevents me from combining multiple source xml files into one webgrab.config file.  I have tested this scan using the exact same output from the channels.xml file and it's not the cause of the massive amount of RAM being used.

 

Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months

Since there seems to be no solution to this situation (using a large xmltv with merge-xmltv.ini), could the feature request of restarting a channel grab after the 'out of memory' be looked at?

Or perhaps a memory release line can be added at the end of the merge-xmltv.ini file?  If it exist, I was unable to find it in the documentation.

Thanks

 

As a workaround: I rerun the grab with incremental mode set and cross my fingers that the 'out of memory' condition occurs on a different channel from the first grab.  Seems to work most of the time, but it's still possible that both grabs run out of memory on the same channel.

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

how about this..

use webgrab for its intended puprose.

just because what your trying isnt working doesnt make it a webgrab problem.

merge works fine when used correctly.

figure out what your problem is.

from the looks of it your merging huges files from all over,looks like some sort or commercial use to me.

if u need the much data use a commercial supplier and pay the price.

 

 

Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months

I'm using this program for my family and trying to save bandwidth use and server abuse by using premade xml files found here:
http://www.webgrabplus.com/content/conditional-access-support-us-button

I figured doing this would help after I noticed how it basically uses 'brute force' to grab from some websites.

 

btw- I didn't say it was a webgrab problem, that's why I posted here:

Idea box
If you have a feature request or an idea, post it here.

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

give me the link to 2 file your trying to merge that give u the out of memory error.

anythying could be wrong,hes using webgrab and may be using broken ini's for example.

his xml files are most likely made from mutliple ini's.

all this could have a effect.

 

 

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

also i must add this...

xmltv-merge.ini set the timezone in the resulting guide.xml file

so if your merging xml files with mutliple time offsets nothing will display correctly when you view the guide and all program will have the time offset from the merge-xmltv.ini and not the offset they were originally grabbed with.

 

Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months

I'm only using the GB xml file with the config uploaded earlier (1-UK.zip).  All other xml files are under 50MB and have no problem.
http://epg.koditvepg.com/GB/guide.xml

I have not quite gotten that far yet for correcting the timezones.  My final file has the times totally off, since they are run through mulitple merge-xmltv.ini files.

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

i wont say what ur trying cannot be done.

i will say this i dont think you will ever get it working in a way that reliable.

i'm not sure if i could get this working,the thing i have is knowledge on how webrab works but there so may if's and's and buts's doing this.

what happens when a grab fails at his site?

if i were you i would just setup my own,after you have it all working use incrimental update mode(use force while your setting it up).

it only update changes and new programs and use the least resources on the source sites.

 

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

i was fixing a ini today and guess what error i got?

"out of memory" and i have 16gb ram.

it was caused by a element under a certain condition in certain scope.

try changing this..

index_variable_element.modify {addstart|'config_site_id'}

to 

index_variable_element.modify {set|'config_site_id'}

 

Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months

That unfortunately doesn't seem fix it. I have debugged the problem to the 'index_showsplit' element being to large.  I wish there was another way to limit this but I can't at the moment.  It looks like 50x the size of the xmltv file when processing the merge-xmltv.ini.  Are you seeing about the same amount of demand on RAM on your end?

 

Edit: I have it working for 1 day, but the siteini has to be set for 1 day, not the config file.  Any reason for it to cause a difference between the maxdays in siteini and timespan in config file?

Kick4U
Offline
Donator
Joined: 8 months
Last seen: 5 months

OK, so I have no issues at all with Windows computers running the same configs.  Therefore I've decided to steer away from the 5W linux box and am now using a laptop a friend traded me that runs off 20W of power with the screen closed.  The runtime is obviously alot faster and since I'll place the laptop in hybrid sleep in between runs, I'll actually consume less energy!  All in all it seems that I have found the solution to my problem.

*I still think WebGrab+ should regrab a channel if the 'Out of Memory' problem occurs (atleast once).  It seems that mono requires a lot of RAM and this feature would help with low RAM devices (2GB like my situation).  Also, it would help in the case that you are running memory exhaustive programs simultaneously in the background.

Thank you Blackbear for working with me again.

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl