slideshow 1 slideshow 2

Welcome

WebGrab+Plus is a multi-site incremental xmltv epg grabber. It collects tv-program guide data from selected tvguide sites for your favourite channels.

Docs release
16/06/2014

Version 1.1.4 of the manual.
This latest edition of the manual. With a lot of small additions of new features. It now also includes documentation of the MDBIni files.

Article
01/06/2014

The first thing to realize is that there are two time-zones involved in the times in your final EPG, each with its own DST rules; the time-zone for which the times of the guide is provided (the 'source' time-zone) and the time-zone of the user (the 'target' time-zone). In the majority of cases these two are equal, but that doesn't change the fact that there are two to consider.

In principle the guide times must be corrected for UTC offset and DST changes for both these two time-zones:  The first correction, the one for the 'source' time-zone, is done by WG++ and the second, for the 'target' (should be done) by the some piece of software at the viewers side, probably the PVR xmltv importer.

Before upgrade 53   http://www.webgrabplus.com/sites/default/files/download/sw/V1.1.1/upgrade/patchexe_53.zip , WG++ was unable to identify the 'source' time-zone correctly, the only info available about it was the time-zone value in the siteini. But that value was limited to just an UTC offset, not the proper time-zone_id and the DST rules for that. (The reason for that limitation lies in the fact that timezone_id's are not standardised among the varies computer platforms, Windows, Linux, OSX, for which WG++ should work) Therefore the DST correction that WG++ applied for the 'source' time-zone was done with the DST rules of the 'target' time-zone. Consequently, the DST changeover dates applied for the source time-zone were the DST changeover dates of the target time-zone. Although wrong in principle, it is only a problem if these DST changeover dates are different, which is possible but unlikely. (no problem if the user lives in the 'source' time-zone).

When does this go wrong? Examples:

-  If a user in Europe want to use a guide from the US (the have different DST changeover dates), or any other mismatch of DST changeover dates.
-  Or if the source time-zone has no DST rules (no DST in that zone) and the user lives in a time-zone with DST rules. Or the opposite.
-  Or if the source time-zone is UTC (the guide times are not in any particular time-zone, just plain UTC or GMT). Any user living in a time-zone with DST rules will run into problems because WG++ will apply the DST rules of the viewer to the xmltv startime which was meant to be UTC without any DST rule.

This was the situation before upgrade 53. The problems explained above can be 'corrected', temporary, by either changing the time-zone value in the siteini or by using the small utility xmltv_time_correct, (available @ http://www.webgrabplus.com/sites/default/files/download/utility/xmltv_time_correct/xmltv_time_correct.zip on the download page) for the period of the mismatch of the DST rules.


All of that is changed with upgrade 53!  That includes an integrated international time-zone database that correctly identifies the source time-zone and the DST rule for that. (see the introduction of upgrade 53 on the download page and further down this page). With that, WG++ doesn't use the target time-zone any more. The two above mentioned DST corrections are completely separated, WG++ for the source, and the PVR for the target.


How to interpret the times in the xmltv file created by WG++?

By xmltv definition, the xmltv times are in UTC (GMT). The xmltv specification allows several formats for that and WG++ uses the one with the an utcoffset , yearmonthdayhourminutesecond +/-utcoffset or  yyyyMMddhhmmss +/-hhmm. The first part of that is (in most cases) the date and time presented in the tvguide of the source site, the second part is the actual offset of that with respect to UTC . E.g   start="20140602141500 -0500"  represents a local time at the source of 2014/06/02 14:15 (or 2:15pm) in a time-zone that is 5 hours behind (-) UTC. So , because the result is UTC by definition , this is 2014/06/02 19:15 UTC .

The eventual DST correction for the source time-zone is done by WG++ through changing the utcoffset with the effect that the result always remains the correct UTC time for this start-time.

What about the start-time for the viewer in his time-zone?

The calculation, to be carried out by some piece of program at the viewers side, probably the xmltv importer of the PVR program, must simply correct for the actual local UTC offset (corrected for DST if necessary). As example a few cases:

- Suppose the viewer is in the same as that of the source, which is quite normal. Then the start-time value of 2014/06/02 19:15 UTC will be corrected back to 2014/06/02 14:15.
- Suppose the viewers actual UTC offset is +8:00 (Singapore or there about), then this start-time is corrected to 2014/06/03 3:15 (next day early morning)

What if the times in your EPG appear to be wrong?

- Upgrade to V1.1.1.53 http://webgrabplus.com/sites/default/files/download/sw/V1.1.1/upgrade/patchexe_53.zip or higher.
- Check if the timezone in the siteini you are using is the correct one for the epg source site
- Check if the times in the xmltv file created by WG++ are indeed as described above (local epg source time and utcoffset)
- Make sure your computer is set to the correct timezone of your location, enable DST and check the time and date setting
- Try to figure out if the xmltv importer of the PVR you are using works as described above (correct for the actual local UTC offset). (Some importers have problems with xmltv files containing different utcoffsets , reading only the first one found and assuming this to be the only one)
- If still in problems post a question onthe forum

 

Jan

Upgrade release
26/05/2014
V1.1.1
/ 53


This Upgrade introduces an integrated TimeZones database which is derived from the public domain tzdata distributed by http://www.iana.org/time-zones. For the processing of that database it uses a customized version of the public domain ZoneInfo Api developed by  Mark Rodrigues as published @  http://www.codeproject.com/Articles/25001/ZoneInfo-tz-Database-Olson-Database-NET-API. With this added functionality it is now possible to calculate the proper daylight saving time cutoverdates for guidedata that is supplied in for timezones that have a different dst cutoverdate than that of the WG++ user.
 

Another important added feature is that by default the program uploads a small status report to this website at the end of each run. This status report contains data like program version used, siteini names and versions, the channels for which epg was requested and errors that occured during the run. With this data, a statistical database is created on the server side that gives insight in the problems of the various program versions, siteini's and channels. Absolutely no personal data about the user, his IP or location is uploaded. This option is switched on by default, users that do not want this, can opt-out by adding a # to the <mode> in the config file.
 

Other improvements , additions and bugs fixed:
 

- added    : the default timezone=UTC+00:00 or WET
- improved : cookie syntax testing now done only for relevant cookies
- improved : the config credential were not expanded in the headers
- bug fixed: lang attribute for titleoriginal was not properly passed to xmltv in case of index_titleoriginal and subdetail_titleoriginal
- added    : the option to grab multiple subdetail pages
- improved : loadcookie (accepts empty cookie values plus generally more robust due to syntax checking)
- improved : firstshow=now skips all shows before fist dayjump as intended
- improved : shows that are fully scheduled in the dst transitions period are skipped
- changed  : title check actual index_title with (detail_)title now uses titlematchlev , (previously just a simple trim and lowercase)
- changed  : allow a space at the the beginning of a valid siteini line
- added    : in class TimeZones Dst cutoverDates DateTime.Kind handling
- added    : TimeZones classes IsDst and InTransition
- fixed    : TimeZones classes GetCutoverWindows, GetOffsets, ConvertToLocal and ConvertToUtc for the Dst cutoverDates DateTime.Kind parameter
- added    : Dst Transition period handling for transitions from standard to dst
- removed  : all debug additions from 1.53.15.13
- added    : embedded tzdata handling
- added    : tzdata is accepted from a single file, or if single file not exist a tzdata folder or
    if tzdata folder not exist the embedded tzdata
- debug    : the timezones classes used are listed in the logfile to trace the linux error
- bug fixed: error in converttolocal in  timezones.cs (double adding of the offset)
- changed  : forward looking dst addapted to the timezones classes
- changed  : converttotime in utils is now independent from the system timezone
- added    : more logging of the tz addition
- changed  : location of the tzdata to the .exe loaction
- added    : error logging , try  catch in the TimeZones constructor to locate the Linux failure.
- added    : tzdata base, must be located in homefolder/tzdata
- added    : the timezone in the siteini will accept timezone names from a tzdata base.
    consequently the dst changes reflect the changes at the transmitter side
- changed  : logging and console text more standardised, mostly lower case
- added    : debug.n, n specifies the show index number in the indexshowslit result. Limits the debug logging to show[n] only.
- bug fixed: xmltv_id and channelname accept " in value
- added    : xmltv_id and channelname values cannot contain both ' and " during channelfile creation, then " is replaced by ¨.
  PostProcess version 1.4
  - improved : A channel excluded from MDB processing will now get the REX presentation update
- bug fixed: cleanup style=name fails when last char is a space
- added    : Statistic upload to the site:  # in mode of config to disable, ## to test it
- bug fixed: regex in substring, replace and remove with {} in it were interpreted as scrubstrings
- bug fixed: headers were only expanded once , therefore subsequent shows got the same headers. This of course is a problem when the headers contains variables.
- bug fixed: error if all channels from a site in the config have an empty update argument. This error was introduced with the new config class in 53.13.2
- bug fixed: urlshow.header postdata was not properly parsed
- improved : speed of the auto siteini locator

Poll
28/02/2014

To get a more general guideline where we should go with WG++ development.

Utility release
27/02/2014

WG++SiteIni.xml is a User Defined Language (UDL) file. This file can be imported into Notepad++, to make Notepad++ highlight special WebGrab++ names.

Read the howto.txt for more information how you must do that.

Below, a before/after screenshot

V008

Article
09/02/2014

With the MDB Postprocessor version 1.3, which is already part of WG++ since Beta build 1.1.1/53.2 it is possible to grab series episode data from IMDb.com. Potentially this series episode data can contain the IMDb episode_id, the episode number, episode title and basically all other EPG data for the actual series episode that is available in IMDb. Before this only the generic Series data could be optained, there was no episode matching. 
 
To get this to work, a dedicated MDB ini is used, imdb.com.imdb_series.ini,  which is available in the MDB postprocessor section of the EPG Channels page:  http://www.webgrabplus.com/sites/default/files/download/ini/info/zip/MDB%20postprocessor_imdb.com.imdb_series.zip. Since the MDB postprocessor main purpose is to add show details to the already grabbed (source) EPG, the shows in it must be matches with IMDb movies and/or series. For this new functionality to work, the matching algoritme uses the Series title in combination with the Episode title (sub-title) . Consequently, the 'source' EPG must at least contain this Series Title (like 'CSI: Miami') and Episode Title (like 'Last Stand') and preferable also some cast member.
 
The series episode data obtained by this can be mixed with- /added to - the EPG source data in the same way as is done with the MDB movies , using the REX postprocessor.
 
The existing functionality, adding movie and generic series data to the grabbed EPG data, is still available. For that the other MDB ini's can still be used. It is arranged such that both methods can be used in one run from the same EPG source file. It will automatically add movie data to movies and series episode data to series.
 
The Documentation page also contains a new section how to configure the MDB postprocessor http://www.webgrabplus.com/documentation/configuration-mdb.
 
At this moment this functionality is only available for English language EPG data, simply because IMDb doesn't provided other language data.
 
Jan

Pages

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl