Foxtel Australia no longer working

I have the same problem

My EPG has been working fine for months from foxtel

Most of this is a bit complicated for me i have only a basic understanding of how it all works

I tried updating the INI file but no good

Would really apreciate some help with this

Cheers Glenn

how do we make that work with Webgrab

Thanks Glenn

Guys, 

There was a site update a while back and it looks like its broken Webgrabs abiltiy to parse the content. Francis is looking at this and should have us and update shortly. 

However it given the new website design it might not be possiable anymore.

Regards,

Enaz

Yes I have the same problem. Happy to donate if you create a fix.
Thanks
Nigel

for some reason my foxtel guide on channel 101 updated today

no other channels

Glenn

Come on all Australian WebGrab+ Users that want a Foxtel EPG lets give the authors some motivation to fix this problem.

** Please pledge (and honour) some donations here in this forum thread **

I will kick this off by pledging $100 if a fix is made for the Foxtel EPG feed.

Regards
Nigel

As a teaser I can give you the first draft version.
No, time is not on my side these days. So that is why it is taking so long to develop the siteini. I think the full version can be made, so that is good news.

If someone has some more time than me, and wants to help, maybe checking out what the genreCode and subGenreCode just mean. Maybe we can just map them like the old values form the current official siteini?

Currently I only have it working for Sydney (regionId=8336). For the other regions, just changing that in the siteini and it shoud work (also re-creating the .channels.xml file when you want the channels of your region).

When I have some more time, I'll work further on this one.

Thanks Francis however I will wait till you've had time to finish... unless you want me to test?
FYI I'm in the Melbourne region.

C'mon Aussie WebGrab Users pledge some donations to thank theWebGrab team for producing a fix for Foxtel!

Thanks Francis, i'll test this build later today and post the results. 

So guys,  I'm running a test using Ballarat region, (Modified the region ID) and updated Time to be +1000... Everything seams to be downloading. But as im at work at the moment i'll confirm when i get hom that EPG looks ok. 

okay

so without changing to the new ini file that Francis uploaded my guide has repopulated  maybe thus/fri last week

still downloading guide updates today

i have left Webgrab running as per normal since the guide went down

do you think Foxtell may have rolled back the update as reading on their forum there were a lot of unhappy users (glitches Ect)

Wondering if anyone else has had this happen

Thanks Glenn

So results are in.. and So far so good, Guide downloaded for 2 days in about 1 hour, not too bad. Modified the Region ID for my location and updated the channels for the site_ID to match the regional Channel's id. 

Awesome work Francis.. I'll test more tonight and let people know results. 

aanda6, I'm in Regional Vic So that ID im using probably won't work for you.. you should use Region ID 12352 for Melb. 

Here's a ini file for Melb Region.. Try this  ini file

Yes also in Melbourne and getting the same errors (see end). FYI (in case I'm doing something wrong) I have listed the steps I have done:

1) I found the Sydney region ID is THREE places in Francis's new foxtel.com.au.ini file so changed it in all three places from 8336 (Sydney) to 12352 (Melbourne).
2) Changed timezone=UTC to Australia/Melbourne.
(Note for other Users in different timezones: a full list of Australian timezone values can be found in Section B in C:\ProgramData\ServerCare\WebGrab\foxtel.com.au_info.txt).
3) I then generated a new foxtel.com.au.channels.xml again as per instructs in the foxtel.com.au_info.txt file then recommented out these lines and ran WG++ but got the errors below:

processing C:\ProgramData\ServerCare\WebGrab\guide.xml .........................
...................................
update requested for - 55 - out of - 55 - channels for 14 day(s)
update mode - set per individual channel

      i=index  .=same  c=change  g=gab  r=replace  n=new

TVH!TS updating, using site FOXTEL.COM.AU, mode incremental
i
error downloading page: The remote server returned an error: (500) Internal Server Error.
pausing 1 of 4 times for 5 seconds before re-try.

error downloading page: The remote server returned an error: (500) Internal Server Error.
pausing 2 of 4 times for 10 seconds before re-try.

error downloading page: The remote server returned an error: (500) Internal Server Error.
pausing 3 of 4 times for 15 seconds before re-try.

error downloading page: The remote server returned an error: (500) Internal Server Error.
pausing 4 of 4 times for 20 seconds before re-try.

no index page data received for TVH!TS
unable to update channel, try again later
<SNIP>

What are we doing wrong since you say Sydney is now working without a problem?

@ALL (who has the new .ini working):

Do you need to set the timezone? Normally, the site lists the times in UTC, so I would suspect that we have to use UTC as the correct timezone. (Currently the info in foxtel.com.au_info.txt is not updated to the new .ini. Just be aware, that some EPG importers don't support UTC offsets in the start/stop fields. So Just check the raw guide.xml file and see if the time is correct.

@Melbourne folks:

Could you upload your .config file, .channels.xml and the log file? Will try to see if we can find what the problem could be.

Thanks Francis that would be very much appreciated.
Here are the files you requested. The log file is truncated as I didn't let WG++ finish processing since all channels were failing with thr same error.

Nigel (a Melbourne folk :)

nigeltec wrote:

Nigel (a Melbourne folk :)

Shame on you! :-)
You didn't changed your config file with the new generated .channels.xml version. So just change the channels in your config file with the new ones from the .channels.xml file and you should be fine.

The old .channels.xml file had number for site_id. But the new .channels.xml file uses a 3 letter ID.

Old:

<channel update="i" site="foxtel.com.au" site_id="548" xmltv_id="TVH!TS">TVH!TS</channel>

New:

<channel update="i" site="foxtel.com.au" site_id="IOI" xmltv_id="TVH!TS">TVH!TS</channel>

aanda6 wrote:

Hi Francis, here are my files. I did change the channels to the new ones.

No, you didn't. Shame on you!

You changed your config to the new "Sydney" channels, but not to your "Melbourn" channels.

eg. in your config file you should change

<channel update="i" site="foxtel.com.au" site_id="NIN" xmltv_id="Channel 9 Sydney">Channel 9 Sydney</channel>

into

<channel update="i" site="foxtel.com.au" site_id="NIM" xmltv_id="Channel 9 Melbourne">Channel 9 Melbourne</channel>

Hi Francis, apologies for forgetting to update my config file however I have now done that but still get the errors (see log file attached).
I've also attached the now updated config file.
(FYI: I don't include any Foxtel channels which aren't included in my subscription or FTA channels since I don't need/use the EPG data).

@nigeltec and aanda6

Could you both add a user-agent line in the config file. When I remove that from my config, I also get Error 500.

mine is:

<user-agent>Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0</user-agent>

aanda6 by "old foxtel settings" do you mean the site.ini and config file that we were using before we started to have problems?
If so does that mean we only need to add the user agent line into our ORIGINAL config file to fix the problem?

Yes mine is also currently downloading latest EPG data thanks "<user-agent>Mozilla..." line.

Francis could give you us some direction here:
1) Was this problem fixed by adding the "<user-agent>Mozilla..." line into the config file OR
2) Do we ALSO need to continue to use the NEW site.ini you created along with the NEW channels.xml data (which is inserted into the WebGrab++.config.xml file) which uses 3 CHARACTERs for site_id as opposed to the old one which used 3 DIGITS?

Thanks
Nigel

1) When you had the error 500, the <user-agent> is the key solution (for old, or for new siteini). Apparently the site needs some dedicated <user-agent> string.

2) My feeling is that you will have to change to the new siteini. The site will probably reverted some API's (or enabled the olds ones again). So that is why the old siteini works again. But the question is for how long will it be in place. a few weeks, a few months, forever... When the site has invested in a new API, it will want to switch to that (who doesn't want to ride his new bought car?). Probably the old API will be disabled, once the new API has got rid of its starter problems.

But for the time being, you could use the good old siteini.

Thanks for your time & effort in resolving this Francis.
Very much appreciated and happy to donate (as promised) as a show of thanks.
I hope others will also show their appreciation and support for WG++ through a donation.

Thanks Francis for your work to sort this out

MC not really much without a working guide

Quite happy to make a $100 donation for your efforts (Transfer over the weekend)

Cheers Glenn

**Reminder for people using Big Screen EPG to import guide data into 7MC**
If new channels are added or existing channels have been renamed you must click the “scan metadata” tile in BSEPG's "Overview" menu to add these channels into BSEPG’s channel list else the EPG data will not get imported into 7MC.
Then after running BSEPG to import this new guide data into 7MC you will need to "Edit listings" (in 7MC) to map this new listing to the appropriate channel number.

For example I was not getting any guide data for channel "Smooth" and found that this channel is now called "Foxtel Smooth" in WG++'s guide data so I needed to do the steps above to solve this.

Donation made

Cheers Glenn

Summary of steps required to use Francis's new foxtel.com.au.ini to fix Foxtel EPG data not downloading

Thought I'd consolidate the steps required in the post to make it easier for Australian Foxtel Users.
Note: For brevity in steps below I will use "ProgramData folder" to refer to C:\ProgramData\ServerCare\WebGrab\.

1. Download Francis's new foxtel.com.au.ini file and copy to "ProgramData folder".
   (See Francis's post #9 in this thread for new foxtel.com.au.ini file).
2. Edit the new foxtel.com.au.ini as follows:
   1. If NOT in Sydney region find & replace all occurances of "8336" with your regionID (should be 3 replacements).
   2. Replace "UTC" in "timezone=UTC" to your timezone.
      (A full list of Australian regionIDs & timezone values can be found in Section B of foxtel.com.au_info.txt in ProgramData folder).
3. Generate a new foxtel.com.au.channels.xml file as per instructs in the foxtel.com.au_info.txt file (located in "ProgramData folder) but remember to recomment out these lines once generated.
4. Edit WebGrab++.config.xml & replace all channel lines with only the channels that you have in your Foxtel package\subscription from the newly generated foxtel.com.au.channels.xml file. Add this line below the "<settings>" line:
   <user-agent>Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0</user-agent>
5. Run WebGrab++ to generate the guide.xml file BUT before running BSEPG to import into media center run BSEPG and click the “scan metadata” tile in BSEPG's "Overview" menu to add any new and/or renamed channels into BSEPG’s channel list else their EPG data will not get imported into 7MC.
6. If there were any new or renamed channels then after running BSEPG to import new guide.xml data into Media Center you will need to "Edit listings" (in Media Center) to map these new listing to the appropriate channel number.
7. Now you good to go - you can resume running WG++ & BSEPG regularly in task scheduler to keep your guide updated.

Hi Francis

Many thanks for all your hard work on getting this new site working for us. On your genrecode code question, I can confirm that the old mappings work. I've shared this INI below. However, I've got an issue with timings and grabbing of shows within the UTC offset.

Like others, I can confirm that you need to set UTC=+10:00 to get the correct start/finish time. However, this means that the first ten hours of guide data is not grabbed. This means that the first time you grab data, you don't see anything until 10 hours has elasped. More importantly, it means that any incremental update to shows doesn't happen within this window either. Has anyone else seen this behaviour? Have I set something wrong in the INI, or even the config file - both attached.

Cheers!

Further to above post... Francis can you please take another look at this section of your ini?

*scope.range {(urlindex)|end}
*index_variable_element.modify {calculate(format=date,unix)|'now'} * get the 'now' time
*url_index.modify {replace|##START_TIME##|'index_variable_element'}
*index_variable_element.modify {calculate(format=D0)|'config_timespan_days' 1 +}
*index_variable_element.modify {addend|:00:00}
*index_variable_element.modify {calculate(format=date,unix)|'urldate' 'index_variable_element' +} *config_timespan_days days after today
*url_index.modify {replace|##END_TIME##|'index_variable_element'}
*end_scope

When I comment it out, the problem I explained above goes away - although you then lose the ability to set the timespan. So some adjustment is needed I think.

Let me know if I can help test in any other way. Many thanks!

Further edit: After running an update, it replaced good guide data with data from the following day - in other words, the guide data is all now 24 hours too early (Friday night football listed for Thursday night etc). Hopefully this is all part of the same problem and not something new.

[  Info  ]  CHANGED show corrected,
[  Info  ]       show with ---- start = 3/09/2015 7:30:00 PM stop = 3/09/2015 10:00:00 PM title = Nine's Live Friday Night Football
[  Info  ]       Replaces ----- start = 3/09/2015 7:30:00 PM stop = 3/09/2015 9:40:00 PM title = Nine's Live Thursday Night Football
[  Info  ]       Replaces ----- start = 3/09/2015 9:40:00 PM stop = 3/09/2015 11:40:00 PM title = The Footy Show

Hi Francis et al

I have been trying to work out why the new Foxtel API grabber misses the first 11 hours of data (as per above posts). And I think I now have an answer.

When the index_variable_element is calculated, it works out a unix formatted "now" and then adds that to the index URL. However, the index URL is in GMT and the "now" is calculated in your local timezone. Hence the way that the new grabber is written, it's always asking for guide data that is offset by your difference from GMT. 

index_variable_element.modify {calculate(format=date,unix)|'now'} * get the 'now' time
url_index.modify {replace|##START_TIME##|'index_variable_element'}
index_variable_element.modify {calculate(format=D0)|'config_timespan_days' 1 +}
index_variable_element.modify {addend|:00:00}
index_variable_element.modify {calculate(format=date,unix)|'urldate' 'index_variable_element' +} *config_timespan_days days after today
url_index.modify {replace|##END_TIME##|'index_variable_element'}

If you comment out the "index_variable_element.modify {calculate(format=date,unix)|'now'} " line and manually enter the current GMT into the ini it correctly grabs all the data from now.

url_index{url|https://www.foxtel.com.au/webepg/ws/foxtel/channel/|channel|/events?movieHeight=110&tvShowHeight=90&startDate=1443776400000&endDate=##END_TIME##000&regionId=8336}

To fix this, it would be necessary to deduct the offset from the unix "now" before adding it to the url_index. While I'm learning as fast as I can - that is currently beyong my abilities. Can someone give me some pointers on how that calculation might be undertaken?

I also reported an error above above guide data moving one day ahead. I looked at the old API ini and saw that a similar correction had been made by inserting "firstshow=1" in the site string. While I have zero idea what that does, including that seems to have fixed the issue for the new API.

The only reason I've managed to get this far is because of the awesome debug logging that this program spits out. Thanks guys - would have no idea how the index URL builder was putting it all together without it!

mitchell.ross wrote:

To fix this, it would be necessary to deduct the offset from the unix "now" before adding it to the url_index. While I'm learning as fast as I can - that is currently beyong my abilities. Can someone give me some pointers on how that calculation might be undertaken?

In the most ugly of fixes, this seems to remove the difference:

index_variable_element.modify {calculate(format=date,unix)|39600 -}

Of course, being a rough and ready fix it will not apply to anyone outside of AEDT and will break at the end of daylight saving - replace the number with the number of hours difference from GMT and multiply by 3600. I've attached my most recent ini for reference.

Hia all. I've only just noticed this thread. I recently went back to using the foxtel feed because of the poor data from yourtv.com.au and I noticed that foxtel is now working again. It looks like I may have to move to the site.ini discussed here at some point but while the old one is worfking I'd rather not convert to something with temporary fixes.

In the meanwhile, I posted some time ago that the program descriptions in my guide always seem to be duplicated. Does anyone else see this? If so, is there a fix for it? I have also tried to use MDB to get more detailed descriptions etc for movies and series but that too seems to have some issues. Is anyone alse using MDB and if so, have you seen the issues described here? http://www.webgrabplus.com/content/mdb-error-invalid-token Any help would be appreciated.

Cheers,

Phil

philzgr8 wrote:

Hia all. I've only just noticed this thread. I recently went back to using the foxtel feed because of the poor data from yourtv.com.au and I noticed that foxtel is now working again. It looks like I may have to move to the site.ini discussed here at some point but while the old one is worfking I'd rather not convert to something with temporary fixes.

In the meanwhile, I posted some time ago that the program descriptions in my guide always seem to be duplicated. Does anyone else see this? If so, is there a fix for it? I have also tried to use MDB to get more detailed descriptions etc for movies and series but that too seems to have some issues. Is anyone alse using MDB and if so, have you seen the issues described here? http://www.webgrabplus.com/content/mdb-error-invalid-token Any help would be appreciated.

Cheers,

Phil

Hi Phil - name seems familiar - a pcmediacenter user perhaps?

This thread seems to have gone pretty quiet. I think while the old Foxtel API is working, people are just using that and only when that one dies are we going to see further development of the new Foxtel ini. I've gone back to the old one myself because in addition to the issues I've mentioned above, there are frequent 500 sever errors even with a user-agent string and sometimes this causes the grab to get out of kilter and start getting programs/dates/times wrong.

I have also experienced the double entry issue that you're talking about. However, since moving back to the old Foxtel ini, combined with the lastest update to webgrabplus I've not seen it again. So perhaps try an update and see if it that fixes it. 

A little off topic of this thread, but I've had mixed success with MDB post-processing. It does take a bit of fiddling. First thing to do is the update I mentioned above. Then you need to get the inis. I've attached my most recent config file and a couple of the inis I was using. I had limited this to just post-processing the movie channels and that was working well. But I was getting invalid token errors on some TV shows on other channels. There are other threads around on that issue I think - but I haven't investigated that issue yet.

Hope this is of some use.

Your memory is correct on pcmediacenter. I was a long time member there and with the same username. You may recall the frog in the tophat which was my avatar. Such a terrible loss of a great resource that forum!

Well for now I'll stick to the existing "official" site.ini too. I have also abandoned mdb postprocessing because it ran from 6:35 am yesterday continuously until I stopped it forcefully at around 1 am today. I'm sure that subsequent runs would be quicker than the initial one but I suspect they will still take too long and I only really ran it to see if it would fix the duplication issue.

Meanwhile, I have upgraded to the latest beta so I will see if that manages to fix the issue. Thanks for your help anyway.

Cheers,

Phil

Edit: I have just viewed my guide data and find that it is still duplicating the show description. Is there any chance you could give me a copy of your site.ini and your webgrab config? I'd like to see if I can find a difference.

Oner more Edit! I had a bit of a look at the site.ini file and found the following two lines:

description.scrub {regex||<ns2:MergedSynopsis>(.*?)</ns2:MergedSynopsis>||}
description.scrub {regex||<ns2:ExtendedSynopsis>(.*?)</ns2:ExtendedSynopsis>||}

When I commented out either line the duplication is gone. I'm not sure if there are potential issues from doing this but for now I will keep the second line commented out. I am still interested in your site.ini if you wouldn't mind uploading it.

My apologies for the delay - when you just edit a post it doesn't send a notification so I missed your request for the ini.

Now attached - and now that I look at it again - I came up with the exact same fix that you did, but had completely forgotten doing so as I had been mucking around with the new API for months. Nothing to do with the update at all it seems.

Hi Guys

I have placed your version in the siteini database http://www.webgrabplus.com/sites/default/files/download/ini/info/SiteIni.Pack/Australia/foxtel.com.au.ini

Thanks     Jan

mitchell.ross wrote:

My apologies for the delay - when you just edit a post it doesn't send a notification so I missed your request for the ini.

Now attached - and now that I look at it again - I came up with the exact same fix that you did, but had completely forgotten doing so as I had been mucking around with the new API for months. Nothing to do with the update at all it seems.

Thanks for that. I didn't find any other real differences other trhan a line to modify the title for Movies. All is good now!

hey guys,

im getting a 400 bad request error

has anyone got an updated ini file?

Cheers blackbear,

im sorry i am a complete noob at this, i was wondering if you could help me,

which line in the WebGrab++.config.xml do i need to edit?

im getting the following:

[        ]              WebGrab+Plus/w MDB & REX Postprocess -- version  V1.57             
[        ]
[        ]                                 Jan van Straaten                                
[        ]                              Francis De Paemeleere                              
[        ]
[        ]             thanks to Paul Weterings and all the contributing users             
[        ] --------------------------------------------------------------------------------
[        ]
[        ] Job started at 29/09/2016 02:37:38
[  Debug ]
[  Debug ] Running  on: Microsoft Windows NT 6.2.9200.0
[  Debug ] Environment: 4.0.30319.34209
[  Debug ]
[  Debug ] Loading timezone data
[  Debug ] Embedded timezones source: WGconsole.WG.Common.timezonesdata.txt
[  Debug ] Reading config file: C:\ProgramData\ServerCare\WebGrab\WebGrab++.config.xml
[        ] Job finished at 29/09/2016 02:37:38 done in 0s
[Critical] Config syntax exception:
[Critical]       file: C:\ProgramData\ServerCare\WebGrab\WebGrab++.config.xml
[Critical]    message:
[Error   ] WG.Common.InvalidConfigException: [Error   ] Channel group contains more than 1 <user-agent> element. Only 1 is supported.
   at WGconsole.Config..ctor(String ConfigurationFilePath)
[Critical] For detailed info, see log file C:\ProgramData\ServerCare\WebGrab\WebGrab++.log.txt
[Critical] Execution stopped

i need help with an example webgrab++.config.xml file,

i reverted back to the orginal file from the installation, but got confused as to which line i need to modify,

Could you supply me with a copy of your webgrab++.config.xml that works for you?

i need help with an example webgrab++.config.xml file,

i reverted back to the orginal file from the installation, but got confused as to which line i need to modify,

Could you supply me with a copy of your webgrab++.config.xml that works for you?

Anyone else having trouble with Foxtell movie channels not downloading guide

happenining last week or so

Cheers Glenn

Yes i have been having issues over the last couple of weeks.

Region is 8288. it is with all the Foxtel movie channels so Premiere, masterpiece, Disney Movies, Family, Action, Comedy, Romance, Thriller as well as the corresponding +2 channels.

Thanks