Updated package for DSMART, HURRIYET, TRT and DIGITURK

Hello,

You need a webhost for php file. If you are using Windows 7-8-10 you can use IIS Web Server it is Microsft's Built-in solution. You can install from Control Panel Add/remove Windows Feature. After finish you IIS installation put test.php file to C:\inetpub\wwwroot folder. Default IIS port is 80 I use 83 for this reason you need to change the line starts with 127.0.0.1:83 to 127.0.0.1:80 in ini file. That's it.

Alternative: You can find free php host on the internet. You can use free hosts but you should change the URL in ini file.

You're welcome 

Netuddki wrote:

You don't need php anymore, because mono was fixed on linux for https sites.

I don't want to burst your bubbles, but there are already better dsmart inis on the forum.
What you did, is a very simple thing and won't get you anywhere.

For example this one (which is still not perfect because sorting has errors, but I will fix it):

 

Hey kiddo, calm down. There is no competition in this website. You should change your attidude before you continue.. I checked all of ini files for dsmart none them was working properly on -WINDOWS- except Blackbear199 php file. What I did, I added some lines to php for generating channel list and I changed site_id. Yes this is not big changes I didn't launch a rocket for mars but I believe it was a good fix for Turkish users. I don't need your or someonelese's appreciations. Capish? Thanks to Blackbear199 ini file fixed and working properly for WINDOWS users.

Blackbear199 wrote:

all plse calm down...

this site is not as simple as it seems..

it your using this(mkaand i think ur using the other i made)

url_index {url|https://www.dsmart.com.tr/actions/schedule?channel_id=|channel|&day=|urldate|}

this is a royal pita

the times are correct but the date is not in the "start_date": "xx"

study the data and see for your self...

i still have this ini open trying to figure out whats going on...

I didn't check all channels that I grap from DSMART but I pick FOXCrime Currently 2:15 AM in Turkey and MacGyver just started and looks OK. But I didn't check every channel one by one.

FX 2:25 Twin Peaks (2017) Looks OK!

Blackbear199 wrote:

if its showing correct for you...great but i have no idea why..

add debug to your showsplit and loo at the "start_date": "xx"

what i amd seeing for today its like 3 days earlier(the date it shows with correct time),tomorrow is different again.

 

I think you're right and wrong. I mean the main core of code working (your code) working perfectly for some channels in DSMART. For example:

There are two A Spor channel in DSMART channel list one HD one SD EPG data (same channel). 

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c6062b2" xmltv_id="A Spor HD">A Spor HD</channel>

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c6062a3" xmltv_id="A Spor">A Spor</channel>

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c60627b" xmltv_id="24 Kitchen Hd">24 Kitchen Hd</channel>

A Spor channels returns NO DATA but 24 Kitchen returns correct data. Here is the link for debug if you want to check:

https://www.dsmart.com.tr/actions/schedule?channel_id=58d29bb0eefad3db9c...

https://www.dsmart.com.tr/actions/schedule?channel_id=58d29bb0eefad3db9c...

Actually source has data but code cannot grab. Maybe you need to add a poke-yoke. I have no idea. But still your PHP solution is the best solution for windows users. I need to check all channel one by one.

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c606243" xmltv_id="ULKE TV">ULKE TV</channel>
    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c60629f" xmltv_id="Ulke Tv HD">Ulke Tv HD</channel>

These two are interested also. First one returns NO DATA but second one returns DATA.

Yesi now ı understand what you say. But looks like this is not your fault. The source has wrong date for some channels. But I don't know main DSMART page:

https://www.dsmart.com.tr/yayin-akisi

is using same data or not but visit above side and find A Spor. It looks OK.

I am not make sure but I think I understood how is working DSMART's API. Please visit this JSON viewer:

http://jsonviewer.stack.hu/#https://www.dsmart.com.tr/actions/schedule?c...

All the programs that shows (Entries) for today's program. Because our query has 2017-08-27

Start and End dates just ignored but the times are valid for GMT. In Turkey we use GMT+3 so everything is clear now:

Currently broadcasting this one "Program Tekrarı" start date 23:00:00Z (GMT) main page shows 02:00 thats correct! The system ignores dates and uses times only I guess.

Can you modify latest DSMART.ini above with this logic? I mean just grab start and end time. I added Task Scheduler for my Windows 10. Every day 10 AM above config file runs. So it is not important to grab tomorrow and other days but of course if you can grab all of data it would be great. As I said before they use only time not date. But like 24 Kitchen channel sometimes day added correct day info also. That's why your code ini file working for some channels and not working for some channels I guess. Now time to go to bed 02:31 AM If you need help for testing just let me know 

Voila!! I just replaced two lines and looks like it fixed. I was testing on this channel:

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c606235" xmltv_id="FB TV">FB TV</channel>

And here is the result:

<?xml version="1.0" encoding="UTF-8"?>
<tv generator-info-name="WebGrab+Plus/w MDB &amp; REX Postprocess -- version  V2.1 -- Jan van Straaten" generator-info-url="http://www.webgrabplus.com">
  <channel id="FB TV">
    <display-name lang="tr">FB TV</display-name>
    <url>http://www.dsmart.com.tr</url>
  </channel>
  <programme start="20170826210000 +0000" stop="20170826220000 +0000" channel="FB TV">
    <title lang="tr">Haber</title>
  </programme>
  <programme start="20170827213000 +0000" stop="20170827220000 +0000" channel="FB TV">
    <title lang="tr">Haber</title>
  </programme>
  <programme start="20170827220000 +0000" stop="20170828063000 +0000" channel="FB TV">
    <title lang="tr">Fenerbahçe - Vardar</title>
  </programme>
  <programme start="20170828063000 +0000" stop="20170828073000 +0000" channel="FB TV">
    <title lang="tr">Günaydın Fenerbahçe</title>
  </programme>
  <programme start="20170828073000 +0000" stop="20170828083000 +0000" channel="FB TV">
    <title lang="tr">Haftalig</title>
  </programme>
  <programme start="20170829080000 +0000" stop="20170829083000 +0000" channel="FB TV">
    <title lang="tr">Sinema Sinema</title>
  </programme>
  <programme start="20170829083000 +0000" stop="20170829093000 +0000" channel="FB TV">
    <title lang="tr">Stüdyo Fenerium</title>
  </programme>
  <programme start="20170830090000 +0000" stop="20170830093000 +0000" channel="FB TV">
    <title lang="tr">Güncel</title>
  </programme>
  <programme start="20170830093000 +0000" stop="20170830100000 +0000" channel="FB TV">
    <title lang="tr">Sinema Sinema</title>
  </programme>
  <programme start="20170830100000 +0000" stop="20170830110000 +0000" channel="FB TV">
    <title lang="tr">Haber</title>
  </programme>
  <programme start="20170831104500 +0000" stop="20170831123000 +0000" channel="FB TV">
    <title lang="tr">Fenerbahçe - Gençlerbirliği</title>
  </programme>
  <programme start="20170901110000 +0000" stop="20170901123000 +0000" channel="FB TV">
    <title lang="tr">Zalgiris Kaunas - Fenerbahçe Ülker</title>
  </programme>
</tv>

But it stranges I can still see 26-08-2017. It's already passed but it is OK. Thank you very much. You grabbed all of data with just changing two lines. Problem solved I guess isn't it?

Maybe I should change UTC+3 I didn't check the times. But it grabs with this ini. Tested.

site {url=dsmart.com.tr|timezone=UTC|maxdays=7|cultureinfo=tr-TR|charset=UTF-8|titlematchfactor=90}
site {episodesystem=onscreen}
*
url_index{url(debug)|http://127.0.0.1:83/test.php?channel=|channel|&date=|urldate|&reqtype=1}
url_index.headers {customheader=Accept-Encoding=gzip,deflate}
urldate.format {datestring|yyyy-MM-dd}
*
index_showsplit.scrub {multi|"schedule": [{||}]}|}]}}

scope.range{(splitindex)|end}
index_showsplit.modify {replace|\}, \{|\|}
index_showsplit.modify {sort(ascending,string)}
sort_by.scrub {single(target="index_showsplit")|"start_date": "||",|",}
sort_by.modify {calculate(target="index_showsplit" format=date,unix)}
index_showsplit.modify {(debug)} *for debug
end_scope
*
index_start.scrub {single|"start_date": "|T|Z|"}
index_stop.scrub {single|"end_date": "|T|Z|"}
index_title.scrub {single|"program_name": "||",|",}
index_title.modify {cleanup(style=jsondecode)}
index_description.scrub {single|"description": "||",|",}
index_description.modify {cleanup(style=jsondecode)}
index_category.scrub {single|"genre": "||",|",}
index_category.modify {cleanup(style=jsondecode)}
*

**  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _
**      #####  CHANNEL FILE CREATION (only to create the xxx-channel.xml file)
**@auto_xml_channel_start
*url_index {url|http://127.0.0.1:83/test.php?reqtype=3}
*url_index.headers {customheader=Accept-Encoding=gzip,deflate}
*index_site_id.scrub {multi|"_id": ||,|,}
*index_site_id.modify {remove|"}
*index_site_id.modify {remove(notnull type=string)|'}'}
*index_site_id.modify {cleanup(removeduplicates=equal,100 link="index_site_channel")}
*index_site_id.modify {cleanup(style=jsondecode)}
*index_site_channel.scrub {multi|"channel_name": "||"|"}
*index_site_channel.modify {cleanup(style=unidecode)}
**@auto_xml_channel_end

I tested your latest modification above. I didn't add UTC+03:00 EPG data shows correct for spme channels (before modification these channels already working with UTC+00:00) If I add UTC+03:00 new EPG data shows on Kodi but all of them wrong. Looks like your fix is not valid fix for this web site. Because still dates are wrong on guide.xml. We need modification. It should grab only TIME not for date. 

For your other question. I have Raspberry Pi Zero. Can I run WebGrabPlus on raspian? It based on linux. But If I do that I need this feature:

After generation of guide.xml linux version should send this file to my FTP. Can I do that? Thanks.

I added this line to my config (Because I am trying to debug)

  <timespan>0</timespan>

Just I want to grab today's EPG data for UÇANKUŞ TV with your latest start end time modification:

What I grabbed:

  <programme start="20170827214500 +0000" stop="20170827220000 +0000" channel="Uçankuş TV">
    <title lang="tr">UçanKuş'ta Astroloji</title>
    <desc lang="tr">Dr. Astrolog Şenay Yangel, her gun burçları yorumluyor..(n)</desc>
  </programme>
  <programme start="20170827220000 +0000" stop="20170827224500 +0000" channel="Uçankuş TV">
    <title lang="tr">Maraba Televole</title>
  </programme>
  <programme start="20170827224500 +0000" stop="20170827233000 +0000" channel="Uçankuş TV">
    <title lang="tr">İstanbul Geceleri</title>
  </programme>
  <programme start="20170827233000 +0000" stop="20170828001500 +0000" channel="Uçankuş TV">
    <title lang="tr">Kuş Yuvası</title>
    <desc lang="tr">Enerji ve renk uzmanı Sebla Öztürk, ünlü konuklarıyla daha enerjik, daha rahat bir yaşam için neler yapılması gerektiğini konuşuyor(n)</desc>
  </programme>

Maraba Televole should start 01 AM but my guide.xml says 21:45 but KODI shows correct time wrong date (28th) KODI PVR time offset= 0

  <programme start="20170827220000 +0000" stop="20170827224500 +0000" channel="Uçankuş TV">
    <title lang="tr">Maraba Televole</title>
  </programme>

So, If I add +03:00, kodi will show wrong, Date info is wrong for some channels on main DSMART TV GUIDE page. For example Nat Geo Channel's EPG info date is OK but Uçankuş wrong it says 24 Agust but no it is today. So your ini modification should just pull time correct date is your query date. Can you modify like that?

Forget about my RaspberryPi questions. I started using linux mono on my remote RaspberryPi. What I did:

I have one Hobby Garden about 30km far away. There is an internet connection router and Raspberry there. Remotely I connected to my Raspberry and follow the instructions on this web site. I installed as HARDWARE FLOAT. I changed DSMART.ini (Just URLs) (it is working without php in linux now). I use 4 ini files for my EPG data. I transfered all of them to RaspberryPi. I installed wput package. I wrote a script like:

#!/bin/bash

/home/pi/.wg++/run.sh
cd /home/pi/.wg++/
wput guide.xml ftp://user:password@MY_FTP_SERVER_ADDRESS/WebGrab++/

I added above script to crontab for run every 6 hours. Works perfectly. But DSMART.INI still needs your final touch.

I think my mono installation is not OK or DMSART SSL is different. I tried to test tsltest.exe here is the results:

pi@raspberrypi:~ $ mono tlstest.exe https://www.nuget.org

https://www.nuget.org
pi@raspberrypi:~ $ mono tlstest.exe https://www.dsmart.com.tr

https://www.dsmart.com.tr
FAILED: #-2146232800
System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: Error while sending TLS Alert (Fatal:InternalError): System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer
  at System.Net.Sockets.Socket.EndReceive (System.IAsyncResult asyncResult) [0x00012] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
  at System.Net.Sockets.NetworkStream.EndRead (System.IAsyncResult asyncResult) [0x00057] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (System.IAsyncResult asyncResult) [0x00037] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (System.IAsyncResult ar, System.Boolean ignoreEmpty) [0x00000] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (System.IAsyncResult result) [0x00071] in <d74853bc825444959b20ae5c72884af0>:0 
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslClientStream.EndNegotiateHandshake (System.IAsyncResult result) [0x00032] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (System.IAsyncResult asyncResult) [0x0000c] in <d74853bc825444959b20ae5c72884af0>:0  ---> System.IO.IOException: Unable to write data to the transport connection: The socket is not connected. ---> System.Net.Sockets.SocketException: The socket is not connected
  at System.Net.Sockets.Socket.BeginSend (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags, System.AsyncCallback callback, System.Object state) [0x00022] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
  at System.Net.Sockets.NetworkStream.BeginWrite (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback callback, System.Object state) [0x0009b] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
   --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.BeginWrite (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback callback, System.Object state) [0x000e6] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.BeginSendRecord (Mono.Security.Protocol.Tls.ContentType contentType, System.Byte[] recordData, System.AsyncCallback callback, System.Object state) [0x00023] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.SendRecord (Mono.Security.Protocol.Tls.ContentType contentType, System.Byte[] recordData) [0x00000] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.SendAlert (Mono.Security.Protocol.Tls.Alert alert) [0x00021] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.SendAlert (System.Exception& ex) [0x0001b] in <d74853bc825444959b20ae5c72884af0>:0 
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslStreamBase.EndNegotiateHandshake (Mono.Security.Protocol.Tls.SslStreamBase+InternalAsyncResult asyncResult) [0x00022] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslStreamBase.NegotiateHandshake () [0x0002b] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslStreamBase.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00064] in <d74853bc825444959b20ae5c72884af0>:0 
  at System.IO.StreamWriter.Flush (System.Boolean flushStream, System.Boolean flushEncoder) [0x0007e] in <c109e83bf6b6427c945b56620bfa750d>:0 
  at System.IO.StreamWriter.Flush () [0x00006] in <c109e83bf6b6427c945b56620bfa750d>:0 
  at TlsTest.GetStreamPage (System.String url) [0x000c2] in <275c587a7d944cc49858fd9ac0265885>:0 
  at TlsTest.Main (System.String[] args) [0x002a6] in <275c587a7d944cc49858fd9ac0265885>:0 
pi@raspberrypi:~ $