You are here

error with schedules direct

39 posts / 0 new
Last post
Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
error with schedules direct

Hi!

I'm trying to get Schedules Direct implemented by following the instructions from schedulesdirect.org_info.txt.

I'm getting (in my eyes) an error that shouldn't be there.

This is the output running 'run.net.sh':

-----------------------------------------

WebGrab+Plus/w MDB & REX Postprocess -- version V5.2.0.0

Jan van Straaten
Francis De Paemeleere

thanks to Paul Weterings, Mat8861, Blackbear199 and all the contributing users
--------------------------------------------------------------------------------

Job started at 11/07/2024 11:24:55
Checking Internet ..
For License request/update data, see WGLicense.log.txt
Checking License ..
- Program benefits setting "custom"
found: /home/ber/TV/webgrabplus_data/.wg++/./siteini.user/schedulesdirect.org.ini -- Revision 03

!! SiteIni schedulesdirect.org.ini in channelfile creation mode !!!

processing /home/ber/TV/webgrabplus_data/epg/guide-SD.xml ...
Found existing channel (xmltv_id=dummy) in the config file
...

i=index .=same c=change g=gab r=replace n=new

Group (0) :
update requested for - 1 - out of - 1 - channels for 1 day(s)
( 1/1 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=dummy) -- mode Incremental
i
error downloading page: Response status code does not indicate success: 401 (Unauthorized).
Unable to update channel dummy
Generic syntax exception:
message:
no index page data received from dummy
unable to update channel, try again later
Existing guide data restored!

Job finished at 11/07/2024 11:24:58 done in 2s
Job (MDB Version: 3.3.0) started at 11/07/2024 11:24:58

Object reference not set to an instance of an object.
Unhandled Exception
For detailed info, see log file /home/ber/TV/webgrabplus_data/.wg++/./WebGrab++.log.txt
Execution stopped
------------------------------------------

and this is the .xml that is generated:

---------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<tv generator-info-name="WebGrab+Plus/w MDB &amp; REX Postprocess -- version V5.2.0.0 -- Jan van Straaten" generator-info-url="http://www.webgrabplus.com">
<channel id="dummy">
<display-name lang="en">dummy</display-name>
<url>http://www.schedulesdirect.org</url>
</channel>
</tv>
----------------------------------------

There is this line "error downloading page: Response status code does not indicate success: 401 (Unauthorized)." but I do have changed the defaults to my credentials...

Any hint how to fix this?

Kind regards,

mat8861
Offline
WG++ Team memberDonator
Joined: 9 years
Last seen: 18 hours

As info.txt says Open the SchedulesDirect.com.ini file and look for the next line and change the credentials to yours.
url_index.headers {credentials=ENTER_USERNAME,ENTER_PASSWORD}
This are meant to be credentials from schedule direct not webgrab credentials.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
mat8861 wrote:

As info.txt says Open the SchedulesDirect.com.ini file and look for the next line and change the credentials to yours.
url_index.headers {credentials=ENTER_USERNAME,ENTER_PASSWORD}
This are meant to be credentials from schedule direct not webgrab credentials.

Yes, I did this and deleted the '*' at the start of the line...

But right now - checking if my credentials are really correct - I noticed a second line with credentials for pedrobmat. I commented it out ('*') and there is no error '401 unauthorized' anymore. Do I need this user?

I'm sorry - but what do I have to do with this new .xml-file? How do I get all the channels to my config? I expected a nice long list with all the channels I selected with the lineup I created to add the ones I need to my WebGrab++-config!? :-)

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

what u did was correct by disabling the perdomat line.
it's just a example as a username,password.
did you check your wg config directory,new channel lists are created there.
if the channel list is sucessfully created copy the ones u want epg for to your wg config.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

Thank you. I found the channel list.

But running WebGrab showed the next error:

------------------------------------------
[ Debug ]
[ Info ] ( 136/148 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=ESPN) -- mode Incremental
[Error ] error downloading page : TimeOut (25 sec)
[Error ] retry 1 of 4 times
[Error ] error downloading page : TimeOut (50 sec)
[Error ] retry 2 of 4 times
[Error ] error downloading page : TimeOut (75 sec)
[Error ] retry 3 of 4 times
[Error ] error downloading page : TimeOut (100 sec)
[Error ] retry 4 of 4 times
[Error ] Unable to update channel ESPN
[Critical] Generic syntax exception:
[Critical] message:
[Error ] no index page data received from ESPN
[Error ] unable to update channel, try again later
[ Info ] No guide data to restore
-------------------------------------------------

Another step I'm missing or 'just' a link not working at schedules direct?

mat8861
Offline
WG++ Team memberDonator
Joined: 9 years
Last seen: 18 hours

Try to set timeout to 80 in your config

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
mat8861 wrote:

Try to set timeout to 80 in your config

Error changed to:
( 129/148 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=ESPN) -- mode Incremental
i./run.net.sh: line 41: 2411 killed dotnet "$DIR/bin.net/WebGrab+Plus.dll" "$DIR"

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

are your lineups new or old existing ones.
see the first msg on their site from dec/2023.
https://www.schedulesdirect.org/
the ini uses SD-DD not SD-JSON.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
Blackbear199 wrote:

are your lineups new or old existing ones.
see the first msg on their site from dec/2023.
https://www.schedulesdirect.org/
the ini uses SD-DD not SD-JSON.

I'm using the 'old' SD-lines.

There is no difference with the requested 'side_id' using a 'rebuild-line' or another line. As I looked for them yesterday it should be the actual ones, also.

According to the shown error
yesterday evening it was a timeout with retry
this morning it was the killed process again (./run.net.sh: line 41 dotnet "$DIR/bin.net/WebGrab+Plus.dll" "$DIR")

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

whatever you did after you created the channel list broke something.
the ini uses the same url(request) for epg data and channel list creation.
so if channel list creation worked then so should epg grabbing as the exact same request is made for both.

did u edit anything in the run.net.sh file?

also another thing to consider..
maybe webgrab was still running from the previous day?
if u had many channels for sd in you config and a time-out of 80.
each channel x 4 retries = 320 secs per channel or 5min 20 secs.
if you had 200 sd channels...200 x 5min 20 secs = 1066 minutes.
add with the time it takes for other channels in you wg config you getting close to a day...1440 minutes.

if wg is still running from the previous day and tries to run again,a error like this could happen.
the reason is when wg runs it opens the wg config file,it stays open until wg completes.
if its opened from a previous run and it tries to run again it will error as the second process cant open the file as its already in use.

check and see if its still running in task manager
it shows as a .NET Host
if its there kill it and try running wg again.

upload your wg config and sd ini in a zip.
remove your license info from you wg config.
remove sd login info from ini.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

I did not touch the run.net.sh and I restarted the VM several times - there is no old process running.

Attached you find the two files.

Attachments: 
Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

u didnt disable the channel list creation.
add a * to the beginning of the lines between
** @auto_xml_channel_start
and
** @auto_xml_channel_end

fyi with newer webgrab u dont have to enable this manually(remove the *)
u can leave it disabled
just use update c in wg config to create a channel list.

if still the same error i am guessing its a dotnet and PATH issue.
do you know where dotnet was installed?
if you dont you will need to find its install directory.
example i installed it in /opt/dotnet,inside this directory you will find the dotnet binary file.
no idea how u installed dotnet but i did it manually.
next edit the run.net.sh file
look for this
# check if dotnet can be found

which dotnet >/dev/null 2>&1 || { echo >&2 "DotNet required, but it's not installed."; quit 1; }

before the which dotnet line add
export PATH=/path/to/dotnet/folder:$PATH
for me it would be
export PATH=/opt/dotnet:$PATH

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

My dotnet-bin can be found at /usr/bin/dotnet which is a link to /usr/share/dotnet/dotnet - installation is to find at /usr/share/dotnet

changes to run.net.sh:

# check if dotnet can be found

export PATH=/usr/share/dotnet:$PATH
which dotnet >/dev/null 2>&1 || { echo >&2 "DotNet required, but it's not installed."; quit 1; }

error:

( 129/149 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=ESPN HD) -- mode Incremental
i./run.net.sh: line 42: 4433 killed dotnet "$DIR/bin.net/WebGrab+Plus.dll" "$DIR"

It's still the same error.

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

try running wg manually in terminal,i assume your .wg++ folder is in /home/username?
/usr/share/dotnet/dotnet "/home/username/.wg++/bin.net/WebGrab+Plus.dll" "/home/username/.wg++"
replace username with your actual username

i really dont think this is the problem as webgrab starts,its when it gets to the 1st schedulesdirect channel it crashes.
it could be the way you edited the files,what editor do you use?

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
Blackbear199 wrote:

try running wg manually in terminal,i assume your .wg++ folder is in /home/username?
/usr/share/dotnet/dotnet "/home/username/.wg++/bin.net/WebGrab+Plus.dll" "/home/username/.wg++"
replace username with your actual username
i really dont think this is the problem as webgrab starts,its when it gets to the 1st schedulesdirect channel it crashes.
it could be the way you edited the files,what editor do you use?

Nope...

this time the error is 'shorter' but nothing new...

( 129/149 ) SCHEDULESDIRECT.ORG -- chan. (xmltv_id=ESPN HD) -- mode Incremental
ikilled

I'm using notepad++

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

since ur in a VM.
how much ram does it have avail?
i see your requesting `15 days epg(timespan 14 in wg config)
try it with a timespan setting of 0(1 day)

depending on how many channels are in your linups this can be a huge amount of data.
sd returns data for all channels in your linups,not just for the channels in your wg config.
wg laters sorts the data for each channel you want epg for.

i'm think this is your problem as when you made your channel list the instructions were to set timespan to 0 and it worked.
if it works with timespan 0 reduce the number of days your grabbing to a minimum or add more ram to your VM.

if i remember right the older V3.x wg version that used mono whout output a error indicating out of memory.
no idea what dotnet outputs,you'd be the first to have it run out of memory...if thats the problem.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
Blackbear199 wrote:

since ur in a VM.
how much ram does it have avail?
i see your requesting `15 days epg(timespan 14 in wg config)
try it with a timespan setting of 0(1 day)
depending on how many channels are in your linups this can be a huge amount of data.

sd returns data for all channels in your linups,not just for the channels in your wg config.
wg laters sorts the data for each channel you want epg for.
i'm think this is your problem as when you made your channel list the instructions were to set timespan to 0 and it worked.
if it works with timespan 0 reduce the number of days your grabbing to a minimum or add more ram to your VM.

Mem usage: 355MB of 3GB
CPU usage: max. 52 % mostly around 5 to 8%
all with the 'old' settings.

with the change to a timespan of '0' there is not that much of a difference...

But it seems to work... :-) Too many days... Will work my way up with this setting...

Thank you!

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

well least finally know where the problem was.

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

btw thks
i also run a vm on my server for some media stuff.
had 2gb ram assigned and figured it should be lots.
just checked it and under 500mb avail.
i doubled avail ram to 4gb.

i run everything seen plus a few other things.
as u can see lots avail ram now.

Attachments: 
Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
Blackbear199 wrote:

btw thks
i also run a vm on my server for some media stuff.
had 2gb ram assigned and figured it should be lots.
just checked it and under 500mb avail.
i doubled avail ram to 4gb.
i run everything seen plus a few other things.
as u can see lots avail ram now.

ram to be a happy camper :-)
As English isn't my first language I hope I translated it correctly...

Is there a way to get you a nice cold beer...

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

your already a donator so you have done enough.
enjoy webgrab.
buy yourself a cold beer.
thanks

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

will do so - just a short walk to the fridge will get it done :-)

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

That's not the solution I suspect...

Having it set to timespan=2 worked yesterday and did not today...

I searched the Schedules Direct forum and got the information that there is epg-data for up to two weeks (13 days) available.

The next thing is that my CPU-load is around 1-2% for a very long time before webgrab++ stalls. While getting the epg-data from other stations it is around 45%. Is there a way to get some more information? More than is to get with the generated log?

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

do u mean different ini than schedulesdirect by other stations?
far as debugging goes there nothing dotnet has available to output additional info.
one of the other guys who has a sd account will have to look into this.
i had one years ago but its expired.

Attachments: 
Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

give this file a try.
i bought a sd account to test,was only 9 dollars for 2 mths.
i use a time-out of 30.
when i first tested file i was getting timeout and retries grabbing 14 days(timespan of 13).
i made a few changes
1. added compression.this wasnt avail years ago(when ini was made) but newer webgrab can now download and decompress compressed(gzip) data.
this made the biggest improvement for a site like this that downloads alot of data.
2. change maxdays=14,was 16.
3. fix credits(actor,director,ect)

as u can see from the log its pretty fast.
2 channels 14 days epg in 22 secs.

edit:
i found another bug where the end date was incorrectly calculated.
ini was only grabbing one day epg regardless of timespan setting.
i fixed this but now since its really grabbing alot of data(really 14 days instead of 1) it requires a very high time-out setting.i had increase it to 60 in my case.
if your only using sd then you can just set it higher in your wg config.
if you also using other ini then a high timeout is most likely not needed.i always use 30 in wg config settings.
with that said wg has a channel grouping option when u can set custom setting like time-out,proxy,ect for a group of channels.
on the downloads page their are documented configuration files available.
its all explained in the wg config one.see screenshot for a example that i used to set a higher time-out just for sd channels.
course this slowed grabbing alot as each channel gets 14 days of epg.
also a screenshot of top while its running.
as u can see its using some cpu and memory..

Edit:
files updated July 15/2024
more improvements

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

I was trying to find a way to see why the process is killed.

I used the new ini for SD and added my credentials but I cannot get any data. This is the error:

------------------------------------
SCHEDULESDIRECT.ORG -- chan. (xmltv_id=ESPNEWS HD) -- mode Incremental
i
error downloading page: An error occurred while sending the request.
Unable to update channel ESPNEWS HD
Generic syntax exception:
message:
no index page data received from ESPNEWS HD
unable to update channel, try again later
Existing guide data restored!
---------------------------------------

Do I have to do more than just adding my credentials?

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

add credentials only modification.

error downloading page: An error occurred while sending the request.
to me this indicates a typo maybe changing the credentials,check it again.

i dont know why your having so many issues.
i updated files above with some more improvements.
nothing i changed was broken,there's just better ways to do things since this ini was made.
have you ever uploaded a full wg log file? i didnt notice one above.
upload one please.
also what linux distro are u running in VM?
i changed my lineup today to a usa one and no issues with the espn channels.
i assume your using a high enough time out?

i dont run wg in VM.
but i do run it in a chroot environment.

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

regarding tracing why the process was killed.
you could try a few suggestions here..
https://stackoverflow.com/questions/726690/what-killed-my-process-and-why

one suggestion i liked what this.
dmesg -T| grep -E -i -B100 'killed process'

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

double post.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

my credentials are correct. Even the version of the ini-file I used before is giving me an error.

Right now it's old error again.

Attached you find the log.

Attachments: 
Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

linux distro?

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

debian 12

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

i really cant see why your having so many issues.
i run ubuntu 22.04,as u know its debian based.
did u read the link in the post above?
have u tried increasing ram just to test?

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months
Blackbear199 wrote:

i really cant see why your having so many issues.
i run ubuntu 22.04,as u know its debian based.
did u read the link in the post above?
have u tried increasing ram just to test?

webgrab is running for the other channels - it's just SD that will not work...

Right now I'm thinking about trying another VM or this damned docker-container (again). The container got some updates since I used it...

4 CPU threads (2 cores) and 4GB RAM should be enough. The CPU is utilized with some 45 % and the RAM with some 300 MB.

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

i ran a test to get a idea of how much data was being downloaded.
for 7 days it hit 400mb and was still climbing.
i killed it as it was hitting 75% cpu and 90% ram usage.
it slowed my laptop down big time,its running a i7 quad with 16gb ram.
you also need to remember that wg has to process this huge amount of data in ram also.
sd requires huge resources.
i would keep number of days grabbed to the minimum.

imho if i were you i wouldnt use it.
tvguide.com.ini(international folder) gets the same data(uses gracenote also).
open ini and read remarks for channel list creation.
if differs from sd in that it downloads a single channel epg at a time.
uses much less resources.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

It seems I just killed my license... was trying to get the docker container to work and now it seems that my custom license is done...

What do I have to do to get my license back?

Blackbear199
Offline
Blackbear199's picture
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 hour

docker is known for this,your hardware id most likely changed.
you can do a forced hardware update with h(like u use f for force license update) on your license line in wg config.

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

I deleted the docker-container and restarted the VM.

I seems that the VM is working fine and the License.log seems to be fine, also...

Will try the tvguide.com.ini... but not today... holy...

Thank you for your help!

Bernard
Offline
Donator
Joined: 6 months
Last seen: 3 months

I changed the epg-Service-Provider and it is running now.

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl