slideshow 1 slideshow 2

Welcome

WebGrab+Plus is a multi-site incremental xmltv epg grabber. It collects tv-program guide data from selected tvguide sites for your favourite channels.

Utility release
05/11/2014

SiteIniIDE is a portable IDE created to simple create/debug/edit siteini file setups. With a few shortcuts in the IDE, you can quickly create new siteini's.

Check the readme.txt file for more info.

Upgrade release
07/08/2014
V1.1.1
/ 54

Changes, improvements and bugs fixed:

- bug fixed : when the config calls a non existend siteini loading empty.ini was not properly implemented
- bug fixed : argument value 'minimum=' was not properly implemented in MDBpostprocessor
- update     : new tzdatabase based on tzdata2014e
- changed  : default max value in each loop is length source element with a limit of 10000
- improved : unmatched timezone now logs al timezones sorted with coming DST date
- added      : timezone query  , (add ? to timezone name in ini), lists timezone details in log file
- bug fixed : not fully working debug switch in loops
- bug fixed : index_variable_element in 'foreach' loop didn't work
- improved : parsing and error handling maxdays
- bug fixed : replace type regex did not accept #### as new string (that was internally used)
- improved : logging and error handling loops
- added      : 'foreach' loop first implementatie
    e.g loop {(debug each 'index_temp_6' in 'index_actor' max=15)|end}
- changed  : loop condition is now executed as real precondition, condition false -- loop not executed
- improved : loop debug logging
- added      : config mode switch ~ , as in <mode>#~</mode> to switch off the 3 seconds wait at the end of a run
- added      : forgotten arguments for command sort  (descending ascending string integer) added to argument and preconditional extraction
- bug fixed : error in the new argument extraction corrected
- improved : MinSWversion check syntax check
- bug fixed : the order of the arguments in an operation created problems when the preconditional was not the first after debug
- improved : the overall internal extraction of preconditionals and other arguments in operations
- bug fixed : small typo in log  "pdate"  changed in "update"
- bug fixed : datetime globalization error in // indexpage overlaps ? (date MM/dd/yyyy cannot parse in dd/MM/yyyy)
- bug fixed : preconditionals ~~ and == were not properly implemented
- Postprocess V1.5
     - added: url headers
- bug fixed : a conditional string between [] is considered a wildcard even if it is no * or ?
- bug fixed : request header host cannot be empty, condition added
- added      : loadcookie=cookiefilename,domain    domain=* will disable domain filter
- bug fixed : request header referer was not used in method GET
- added      : request header host

Docs release
16/06/2014

Version 1.1.4 of the manual.
This latest edition of the manual. With a lot of small additions of new features. It now also includes documentation of the MDBIni files.

Article
01/06/2014

The first thing to realize is that there are two time-zones involved in the times in your final EPG, each with its own DST rules; the time-zone for which the times of the guide is provided (the 'source' time-zone) and the time-zone of the user (the 'target' time-zone). In the majority of cases these two are equal, but that doesn't change the fact that there are two to consider.

In principle the guide times must be corrected for UTC offset and DST changes for both these two time-zones:  The first correction, the one for the 'source' time-zone, is done by WG++ and the second, for the 'target' (should be done) by the some piece of software at the viewers side, probably the PVR xmltv importer.

Before upgrade 53   http://www.webgrabplus.com/sites/default/files/download/sw/V1.1.1/upgrade/patchexe_53.zip , WG++ was unable to identify the 'source' time-zone correctly, the only info available about it was the time-zone value in the siteini. But that value was limited to just an UTC offset, not the proper time-zone_id and the DST rules for that. (The reason for that limitation lies in the fact that timezone_id's are not standardised among the varies computer platforms, Windows, Linux, OSX, for which WG++ should work) Therefore the DST correction that WG++ applied for the 'source' time-zone was done with the DST rules of the 'target' time-zone. Consequently, the DST changeover dates applied for the source time-zone were the DST changeover dates of the target time-zone. Although wrong in principle, it is only a problem if these DST changeover dates are different, which is possible but unlikely. (no problem if the user lives in the 'source' time-zone).

When does this go wrong? Examples:

-  If a user in Europe want to use a guide from the US (the have different DST changeover dates), or any other mismatch of DST changeover dates.
-  Or if the source time-zone has no DST rules (no DST in that zone) and the user lives in a time-zone with DST rules. Or the opposite.
-  Or if the source time-zone is UTC (the guide times are not in any particular time-zone, just plain UTC or GMT). Any user living in a time-zone with DST rules will run into problems because WG++ will apply the DST rules of the viewer to the xmltv startime which was meant to be UTC without any DST rule.

This was the situation before upgrade 53. The problems explained above can be 'corrected', temporary, by either changing the time-zone value in the siteini or by using the small utility xmltv_time_correct, (available @ http://www.webgrabplus.com/sites/default/files/download/utility/xmltv_time_correct/xmltv_time_correct.zip on the download page) for the period of the mismatch of the DST rules.


All of that is changed with upgrade 53!  That includes an integrated international time-zone database that correctly identifies the source time-zone and the DST rule for that. (see the introduction of upgrade 53 on the download page and further down this page). With that, WG++ doesn't use the target time-zone any more. The two above mentioned DST corrections are completely separated, WG++ for the source, and the PVR for the target.


How to interpret the times in the xmltv file created by WG++?

By xmltv definition, the xmltv times are in UTC (GMT). The xmltv specification allows several formats for that and WG++ uses the one with the an utcoffset , yearmonthdayhourminutesecond +/-utcoffset or  yyyyMMddhhmmss +/-hhmm. The first part of that is (in most cases) the date and time presented in the tvguide of the source site, the second part is the actual offset of that with respect to UTC . E.g   start="20140602141500 -0500"  represents a local time at the source of 2014/06/02 14:15 (or 2:15pm) in a time-zone that is 5 hours behind (-) UTC. So , because the result is UTC by definition , this is 2014/06/02 19:15 UTC .

The eventual DST correction for the source time-zone is done by WG++ through changing the utcoffset with the effect that the result always remains the correct UTC time for this start-time.

What about the start-time for the viewer in his time-zone?

The calculation, to be carried out by some piece of program at the viewers side, probably the xmltv importer of the PVR program, must simply correct for the actual local UTC offset (corrected for DST if necessary). As example a few cases:

- Suppose the viewer is in the same as that of the source, which is quite normal. Then the start-time value of 2014/06/02 19:15 UTC will be corrected back to 2014/06/02 14:15.
- Suppose the viewers actual UTC offset is +8:00 (Singapore or there about), then this start-time is corrected to 2014/06/03 3:15 (next day early morning)

What if the times in your EPG appear to be wrong?

- Upgrade to V1.1.1.53 http://webgrabplus.com/sites/default/files/download/sw/V1.1.1/upgrade/patchexe_53.zip or higher.
- Check if the timezone in the siteini you are using is the correct one for the epg source site
- Check if the times in the xmltv file created by WG++ are indeed as described above (local epg source time and utcoffset)
- Make sure your computer is set to the correct timezone of your location, enable DST and check the time and date setting
- Try to figure out if the xmltv importer of the PVR you are using works as described above (correct for the actual local UTC offset). (Some importers have problems with xmltv files containing different utcoffsets , reading only the first one found and assuming this to be the only one)
- If still in problems post a question onthe forum

 

Jan

Upgrade release
26/05/2014
V1.1.1
/ 53


This Upgrade introduces an integrated TimeZones database which is derived from the public domain tzdata distributed by http://www.iana.org/time-zones. For the processing of that database it uses a customized version of the public domain ZoneInfo Api developed by  Mark Rodrigues as published @  http://www.codeproject.com/Articles/25001/ZoneInfo-tz-Database-Olson-Database-NET-API. With this added functionality it is now possible to calculate the proper daylight saving time cutoverdates for guidedata that is supplied in for timezones that have a different dst cutoverdate than that of the WG++ user.
 

Another important added feature is that by default the program uploads a small status report to this website at the end of each run. This status report contains data like program version used, siteini names and versions, the channels for which epg was requested and errors that occured during the run. With this data, a statistical database is created on the server side that gives insight in the problems of the various program versions, siteini's and channels. Absolutely no personal data about the user, his IP or location is uploaded. This option is switched on by default, users that do not want this, can opt-out by adding a # to the <mode> in the config file.
 

Other improvements , additions and bugs fixed:
 

- added    : the default timezone=UTC+00:00 or WET
- improved : cookie syntax testing now done only for relevant cookies
- improved : the config credential were not expanded in the headers
- bug fixed: lang attribute for titleoriginal was not properly passed to xmltv in case of index_titleoriginal and subdetail_titleoriginal
- added    : the option to grab multiple subdetail pages
- improved : loadcookie (accepts empty cookie values plus generally more robust due to syntax checking)
- improved : firstshow=now skips all shows before fist dayjump as intended
- improved : shows that are fully scheduled in the dst transitions period are skipped
- changed  : title check actual index_title with (detail_)title now uses titlematchlev , (previously just a simple trim and lowercase)
- changed  : allow a space at the the beginning of a valid siteini line
- added    : in class TimeZones Dst cutoverDates DateTime.Kind handling
- added    : TimeZones classes IsDst and InTransition
- fixed    : TimeZones classes GetCutoverWindows, GetOffsets, ConvertToLocal and ConvertToUtc for the Dst cutoverDates DateTime.Kind parameter
- added    : Dst Transition period handling for transitions from standard to dst
- removed  : all debug additions from 1.53.15.13
- added    : embedded tzdata handling
- added    : tzdata is accepted from a single file, or if single file not exist a tzdata folder or
    if tzdata folder not exist the embedded tzdata
- debug    : the timezones classes used are listed in the logfile to trace the linux error
- bug fixed: error in converttolocal in  timezones.cs (double adding of the offset)
- changed  : forward looking dst addapted to the timezones classes
- changed  : converttotime in utils is now independent from the system timezone
- added    : more logging of the tz addition
- changed  : location of the tzdata to the .exe loaction
- added    : error logging , try  catch in the TimeZones constructor to locate the Linux failure.
- added    : tzdata base, must be located in homefolder/tzdata
- added    : the timezone in the siteini will accept timezone names from a tzdata base.
    consequently the dst changes reflect the changes at the transmitter side
- changed  : logging and console text more standardised, mostly lower case
- added    : debug.n, n specifies the show index number in the indexshowslit result. Limits the debug logging to show[n] only.
- bug fixed: xmltv_id and channelname accept " in value
- added    : xmltv_id and channelname values cannot contain both ' and " during channelfile creation, then " is replaced by ¨.
  PostProcess version 1.4
  - improved : A channel excluded from MDB processing will now get the REX presentation update
- bug fixed: cleanup style=name fails when last char is a space
- added    : Statistic upload to the site:  # in mode of config to disable, ## to test it
- bug fixed: regex in substring, replace and remove with {} in it were interpreted as scrubstrings
- bug fixed: headers were only expanded once , therefore subsequent shows got the same headers. This of course is a problem when the headers contains variables.
- bug fixed: error if all channels from a site in the config have an empty update argument. This error was introduced with the new config class in 53.13.2
- bug fixed: urlshow.header postdata was not properly parsed
- improved : speed of the auto siteini locator

Poll
28/02/2014

To get a more general guideline where we should go with WG++ development.

Pages

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl