I've noticed that when the critical error 'Exception.Message: Out of memory' occurs, WebGrab thinks its restarting the channel, but really isn't. Can it rerun the current channel overwriting if necessary the partial grab? Or, add an option to clear the memory using a function within the siteini (perhaps a first or last step)?
FYI- I've attached the project tha brings me the error since the siteini has numerous of repetitive replaces.
Attachments:
Thanks for looking into this again. I've tried many different scenarios and can't seem to find the answer. I still encouter this memory problem when using only one siteini (mergexml) for one xmltv and lowering the days grabbed to 0 (same day).
Any other ideas?
I had already looked into both of these concerns extensively by trying multiple different methods (including the conventional method explained in merge-xmltv_info.howtosetup.txt). I've somewhat ran out of ideas; hoping perhaps someone had come across a similar situation with the memory being used on the scale of 1.5GB RAM... Perhaps that is normal, since it's only being a noticeable nuissance on the largest source xml (roughly 70MB total in size).
1) ini is in creation mode since the source xml is changing very frequently and this saves me from changing the siteini and rerunning. Once the source xml stables out, I'll comment this section. I have tested with and without this section commented and I've concluded this is not the cause of the massive amount of RAM being used.
2) site_id has not changed. Both xmltv_id and the display-name are changed to save me from needing to change those values again in another config running afterwards; combined with others to create one file (all of the runs use only merge-xmltv). FYI- I am not running webgrab+ at all for grabbing from a website, I am using premade xmltv files and merging them into one. I am getting data, and it shows on the log file.
I need to run them seperately and change the display-name since the source-id from one source has many the same source-id from other countries (like FOX). This prevents me from combining multiple source xml files into one webgrab.config file. I have tested this scan using the exact same output from the channels.xml file and it's not the cause of the massive amount of RAM being used.
Since there seems to be no solution to this situation (using a large xmltv with merge-xmltv.ini), could the feature request of restarting a channel grab after the 'out of memory' be looked at?
Or perhaps a memory release line can be added at the end of the merge-xmltv.ini file? If it exist, I was unable to find it in the documentation.
Thanks
As a workaround: I rerun the grab with incremental mode set and cross my fingers that the 'out of memory' condition occurs on a different channel from the first grab. Seems to work most of the time, but it's still possible that both grabs run out of memory on the same channel.
I'm using this program for my family and trying to save bandwidth use and server abuse by using premade xml files found here:
http://www.webgrabplus.com/content/conditional-access-support-us-button
I figured doing this would help after I noticed how it basically uses 'brute force' to grab from some websites.
btw- I didn't say it was a webgrab problem, that's why I posted here:
I'm only using the GB xml file with the config uploaded earlier (1-UK.zip). All other xml files are under 50MB and have no problem.
http://epg.koditvepg.com/GB/guide.xml
I have not quite gotten that far yet for correcting the timezones. My final file has the times totally off, since they are run through mulitple merge-xmltv.ini files.
That unfortunately doesn't seem fix it. I have debugged the problem to the 'index_showsplit' element being to large. I wish there was another way to limit this but I can't at the moment. It looks like 50x the size of the xmltv file when processing the merge-xmltv.ini. Are you seeing about the same amount of demand on RAM on your end?
Edit: I have it working for 1 day, but the siteini has to be set for 1 day, not the config file. Any reason for it to cause a difference between the maxdays in siteini and timespan in config file?
OK, so I have no issues at all with Windows computers running the same configs. Therefore I've decided to steer away from the 5W linux box and am now using a laptop a friend traded me that runs off 20W of power with the screen closed. The runtime is obviously alot faster and since I'll place the laptop in hybrid sleep in between runs, I'll actually consume less energy! All in all it seems that I have found the solution to my problem.
*I still think WebGrab+ should regrab a channel if the 'Out of Memory' problem occurs (atleast once). It seems that mono requires a lot of RAM and this feature would help with low RAM devices (2GB like my situation). Also, it would help in the case that you are running memory exhaustive programs simultaneously in the background.
Thank you Blackbear for working with me again.