As of last week the tvguide.com downloads "froze" at January 29. It still successfully pulls the data up until January 29, but it is not pulling data for anything after that date. So I went to the tvguide web site, and it looks like they have totally changed the web site. (Using the web site, as of today I can look forward out to February 3.)
I've attached my config and ini files.
You are here
tvguide.com usa stopped working - no new data after Jan 29, 2021
Fri, 2021-01-22 00:59
#1
tvguide.com usa stopped working - no new data after Jan 29, 2021
Brought to you by Jan van Straaten
Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl
works fine here
Hi mat8861,
The log that you posted indicates that you ran it for one day, which is (obviously) prior to January 29. That works fine for me also. The problem is that I run my grab for 14 days, and the tvguide web site currently goes out to Feb 2, and so I should be getting data through February 2. But I don't. It stops at January 29, and each night my guide.xml file is getting a little smaller. My log file from this morning is attached.
My guess is that, with the change in their web site interface, the grabbing is broken, but they must have the old format still accessible through Jan. 29.
Thanks,
Bob
Are you saying that after February 2 the ini file will stop working? Most of my channel guide I get from that website :(
Hi Tehra,
I'm actually saying that it will stop working after January 29. At least, that's when mine stopped.
they changed the entire site.
it required a total rewrite,and its no where as good as it was before.its still ok but i see alot of stuff gone.
all of your channel line will have to be replaced with new ones as the provider id's and channel id's all changed also so existing channel lines from the old ini wont work.
channel xml generation all changed also,its less complicated than before,least i think.
usa has its own section to create a provider list by zipcode.
all other countries the provider list has every provider available in the entire country.
read all the sections,its pretty self explainatory.
for those that use a single provider(ex zipcode 10001 dishnetwork new york).
edit the ini and enable site {keepindexpage) line,this will speed up grabbing.
if your using mixed zipcodes or countries leave it disabled(default).
included is a country list and a channel list for zipcode 10001 dishnetwork new york.
Hi Blackbear199, Thank you very much for the update, I already tested the ini and it works fine, I just have one question.
Why are there many letters "i"? Can't it be reduced? I remember that the last file only came out 5 letters, from channel to channel if it takes little time to pass those letters "i".
When I enable this option site {keepindexpage} and effectively, it is faster, but you say that you only have to enable when you download epg from a single provider, but I download from several countries. What could I do there to speed up the download of the index?
the old site grabbed all the epg data with a single request(based on timespan setting).
this is why you seen a single i or index page with it.
the new site grabs the data in 3 hr chunks and by the day.
so for every day of epg requested there are 8 i's or index pages(3hr x 8 = 24hrs).
i tried many different scenarios like grabbing 2x12 hr chunks or doing the same as the old ini.
sometimes it seemed to work but many times i got no index page error.
seems to be touch and go,no idea if its a site issue or something else.
so i did the ini to mimic exactly how the site grabs the data(3 hr chunks), its slower but works everytime.
make multiple copies of the ini(one for each country or zipcode you use).
name them something so you can tell them apart like tvguide.com_argentina.ini or tvguide.com_10001.ini
NOTE: i used argentina as a example,if you created a list for it you would see there are many different providers,you would need a renamed ini for every provider you use.
then you can leave site {keepindexpage} enabled.
just use each of the ini you renamed to create your channel list.
Hi Blackbear,
Thanks so much for this rework!!
Unfortunately, there seems to be a problem with the dates somehow:
Unable to update channel WTMJ HDTV (WTMJ) [4.1]
Generic syntax exception:
message:
Current culture: en-GB
time parsing error : The string was not recognized as a valid DateTime. There is an unknown word starting at index 0.
nextstartdatetime time scrubbed : null
computer date/time format: 22/01/2021 23:11:32
Existing guide data restored!
It only happens with a couple of the channels, and not for every pull. I have it configured to pull 15 days, and 61 channels. The first channel pulls fine, but then the second channel gets this error. However, if I set the pull to 8 days, then it works. So, there is something about a date in the 9th day that causes the error for this particular channel (4.1)
Once it hits the error, it throws out the entire pull for that channel and moves on. It succeeds for 4 more channels, and then the next one gets the error again.
Just to note, I am in the USA, so I don't know why it reports the en-GB culture and date format in the error.
Another thing - the 2 channels that fail are both local affiliates of the NBC network (Milwaukee and Chicago OTA), so their programming is largely the same, and they seem to fail at about the same point in the process (as I said, somewhere about 8 or 9 days into it.) So I would wager that it is the same program causing the issue for both channels.
I've attached the ini and config file that I'm using for this grab, and the log file that shows the errors for channels 4.1 and 5.1
EDIT: if you post your config with user/email/password someone else may use it, then don't complain you license doesn't work.
fixed.
redownload ini in above post.
they added a empty show(no title/start/stop time) and this caused the error.
it happened on jan 30.
pretty sure they did this because they havnt got the schedule finalized as its a week in the future.
the gap will populate with epg one they add programming for it.
You're amazing!!!!
Now I just have one more request :)
In my previous ini file, there were lines that added an asterisk to the Title if the program is new:
* next line adds * to new shows, Title * = new shows, Title = "not new", 2 places
index_title.modify {addend('index_temp_5'=="4")| *}
This is important for my HTPC Scheduled Recordings, because that's how I set it up to only record new shows (since the title is different due to the asterisk, the scheduler sees them as different programs than the re-runs.)
Is there any way this can be done in the new version of the ini?
Thanks so much!
added.redownload ini,no revision change this time.
Blackbear199, thank you so much for your quick responses and solutions to my issues!!!!!
It is currently running, and so far the results look great! (Although, as you said, the new tvguide website is not quite as detailed as it was previously.)
Definitely worthy of another (new) donation :)
Thanks again!
if you need more details you can use imdb, it has been improved a lot.
Thank you very much Blackbear199, I am following your advice, I am creating an ini file for each country and each provider, the bad thing that I cannot pass the 15 ini files, since my license says that a maximum of 15 ini files,So I have to limit myself to using a single provider for each country, sometimes I can't find a channel in that provider and I have to look for another where it is and I add it, well, that was before, right now if I do that with the new ini , I will exceed the 15.ini files that my webgrab license allows.
if you have a bunch or providers that your only getting a few channels epg for then use a single ini for them all and leave the site {keepindexpage} line disabled.
it wont slow grabbing down for a few channel,few 100 and you will see a difference.
that or contact Jan and get your license upgraded.
Nevermind. It is working now. Not sure what changed. :)
Hi Blackbear199. It seems as though when attempting to create a provider list for CAN, I get a short channel list. How can I get a provider list?
Example attached.
Im actually getting error for all the channels today
[ Info ] ( 2/41 ) TVGUIDE.COM -- chan. (xmltv_id=Much HDTV) -- mode Force
[Warning ] no robots data found
[Warning ] skipped robots check
[Error ] Unable to update channel Much HDTV
[Critical] Generic syntax exception:
[Critical] message:
[Error ] no index page data received from Much HDTV
[Error ] unable to update channel, try again later
[ Info ] Existing guide data restored!
I also used the ini file uploaded in this conversations but no luck. Any idea?
did you create new channel lists?
the ones for the old ini wont work.
I actually tried yesterday to create a new channel list but I got the same error. The ini in the github from 4 years ago is still there. Does that still work and it will be changed?
I tried creating it using the new ini and I get this:
[ Info ] ( 1/1 ) TVGUIDE.COM -- chan. (xmltv_id=(Canada) Edmonton - Telus OPTIK TV) -- mode Force
[Warning ] no robots data found
[Warning ] skipped robots check
[Error ] Unable to update channel (Canada) Edmonton - Telus OPTIK TV
[Critical] Generic syntax exception:
[Critical] message:
[Error ] no index page data received from (Canada) Edmonton - Telus OPTIK TV
[Error ] unable to update channel, try again later
[ Info ] Existing guide data restored!
[ Debug ]
[ Debug ] 0 shows in 1 channels
[ Debug ] 0 updated shows
[ Debug ] 0 new shows added
Works perfectly japangelo, maybe you didn't change "site_id" to the new format of each line of your channels
whats the new format?
in each section of the channel xml creation sections there is instructions,follow them.
i doubt your going to manually fix each channel line.
after you create new lists u can manually copy the site_id="xx" to your old channel lines in your webgrab config to save you time if you use different xmltv_id="xxx" and display names.
For anyone having same issue that I had. use the tvguide.com.ini from this tread as the one in siteini pack is not updated yet.
Try again. I was just getting that error. Took a break and tried again. Working now. Not sure if we are getting temporarily blocked for hitting server too many times, or if it was temporary issue at tvguide.com .
yeah I saw that too and stopped. Ill wait a couple more hours and see
I was able to update everything, providers and channels lists..it works now but I get "no shows in indexpage" on some channels..I dont get why
some channels aint enough info,did u check the site to see if they have epg?
Yes I did.. For channels like CBC EDMONTON or GLOBAL EDMONTON i see the epg on the website but I get the no index error
what did u use for postal code/provider.
or
post the site_id="xx" for each.
these are canadian channels. I used Telus provider ID 9100003976
or Shaw ID 9100004035
the ini grabs epg un UTC time.
right now its 8pm and i am utc-0400 so when i run it i get no programs because its midnight utc time.
check your log,u will see it shipped all the shows for today.
i increased my timespan to 2 days and it works..
update requested for - 2 - out of - 2 - channels for 2 day(s)
( 1/2 ) TVGUIDE.COM -- chan. (xmltv_id=CBC Edmonton (CBXT) [4]) -- mode Force
iiiiiiiiiiiiiiiinnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
0.98 sec/update
( 2/2 ) TVGUIDE.COM -- chan. (xmltv_id=Global Edmonton (CITV) [8]) -- mode Force
iiiiiiiiiiiiiiiinnnnnnnnnnnnnnnnnnnnnnnnnnnnn
1.35 sec/update
My time span is set to 4 and I get the error. U recommend me to wait for tomorrow?
did u do the last channel creation section with the above provider numbers?
your site_id="xxx" should be something like this,these are the shaw ones..
site_id="9100004035##9200008798" xmltv_id="CBC Edmonton (CBXT) [4]"
site_id="9100004035##9233011217" xmltv_id="Global Edmonton (CITV) [8]"
Yes I did and those ids match mine.. Exactly the same
it should work then,did u remember to disable the channel creation sections?
it's half working. I grab epg from both usa and canada channels. Basically what I noticed is that if I put the canadian channels first in the webgrab config file they load fine but the usa channels all show the no index error. If i put the usa channels first, the canadian channels get the error message. Its kinda messy. Sometimes they load, sometimes not
your a product of your own demise.
1. your not using the ini from post #6(revision 18)
2. if you where it would have the site {keepindexpage} line disabled which i took the time to add a message that says leave disabled if using more than one provider.
I guess Im really an idiot..im so sorry..Ill try later with the new ini
Blackbear199 it looks as though tvguide.ini is in 2 locations in the current siteini zip.
/Networks
/International
I am not sure which file is the correct one.
Anyone else finding the index grabbing for each channel when it pulls data takes forever. Unfortunately I use several zip codes and different providers so I can't enable keep index page.
For now i'll have to grab each provider in a separate config else it will take hours.
I'm assuming the siteini posted here is the same one Blackbear199 provided to me a week ago. I must have been psychic to know it would need a re-write but for a different reason to what I was asking for.
If anyone needs the provider/channel lists i'll include some with this post
Thank you Blackbear199. The new ini is fantastic!
@Blackbear199
Just an idea where you could have multiple zip codes/providers and still retain index page to speed up grabbing.
Is it possible to have an error capture loop so in the config you list all the channels from one zip code retaining the index for each and then when it hits the zip code that's different and fails as it's not in the index the error loop captures the no index error and sends the siteini back to start where it re-downloads the index for the next zip code and so on.
no,it has todo with how webgrab works.
just do as i said above.
copy the ini and rename it something different for every provider you use.
then you can leave the site {keepindexpage} enabled for them all.
you wont have to redo all your channel lists(unless you dont know which provider the come from).
all you would have todo is change the site="tvguide.com" to whatever the renamed ini is for channel.
everything else on the channel line stays the same.
Already done thanks
It took a while, but I did get all my channel links reformatted and grabbing properly with the new tvgude.com.ini.
But I'm using 5 different providers, 3 different zips for Dish, and one ini for Canada. And for the life of me I can't get the grabs from tvguide to speed up using Blackbear's instructions to create duplicate tvguide.ini files and add a suffix for each provider, zip, country leaving site {keepindexpage} enabled. Here's what I renamed them to:
tvguide.com.ini
tvguide.com_Bell_Satellite_TV_Toronto.ini
tvguide.com_Broadcast_90807.ini
tvguide.com_DirecTV_10001
tvguide.com_Dish_10001.ini
tvguide.com_Dish_19102.ini
tvguide.com_Palm_Coast_33965.ini
It's taking about 18 seconds to index each 3 hr chunk, so about 1.5 minutes for each channel for a 7 day grab. I've tested with those renamed inis in the \siteini.pack\International folder as well as \siteini.user and haven't seen any reduction of time for grabbing the data. Am I reading Blackbear's instructions wrong?
In the siteini find this line
*site {keepindexpage}
And uncomment it (remove the *)
Do that as well in all your siteinis but make sure your config is only using one zip code/provider per copied siteini
If you have already done the above so the index is retained per siteini (it doesn't redownload the index every channel grab for the same siteini) then that's the fastest it will go
I think that's what I have, but indexing is still very slow. Most all the channels in my config are from providers and channels generated by the tvguide.ini. In order by zip/country here's 6 examples of each zip / country in the config:
site="tvguide.com" site_id="9100001970##9233015158" xmltv_id="WPVI (ABC Philly)" - xx-Dish_19102.ini
site="tvguide.com" site_id="9100008443##9233013462" xmltv_id="Animal Planet (East2)" - xx-Dish_10001.ini
site="tvguide.com" site_id="9133007816##9200008780" xmltv_id="HBO (East)" - xx-DirecTV_10001.ini
site="tvguide.com" site_id="9133003455##9233016893" xmltv_id="Love Nature" - xx-Bell_Satellite_TV_Toronto.ini
site="tvguide.com" site_id="9100008963##9200000213" xmltv_id="PBS WGCU FL" - xx-Palm_Coast_33965.ini
site="tvguide.com" site_id="9133002871##9233003495" xmltv_id="KCBS HDTV" - xx-Broadcast_90807.ini
And I just found one for the tvguide.com.channels.Broadcast TV New York.10001.xml in my config I forgot. I didn't make this one:
site="tvguide.com" site_id="9133006313##9233009269" xmltv_id="WNBC Cozi TV" - xx-Broadcast TV New York.10001.ini
So I have those 6 copies of the tvguide.ini. Does it throw things off if I missed one, like forgetting to make a duplicate tvguide.com_Broadcast_TV_New_York.ini. Or that I left the main tvguide.com.ini in there?
Pages