You are here

Request Payload on Post Data

11 posts / 0 new
Last post
kvanc
Offline
WG++ Team memberDonator
Joined: 7 months
Last seen: 1 hour
Request Payload on Post Data

Hello guys;

I am not sure about location of topic and I may remove/move if it is not in correct place. I'm working on a ini file for several hours and didn't make it work(not even started).

https://web.tvplus.com.tr/tv

This is the address of website and I copied my ini(not working) for more details. Basicly, site is working with POST request but post_data(which contains urldate and channelid) is coming as "request payload" instead of common "form data" which is my main suspect. This is exp. the post data;

{"type":"2","channelid":"89","begintime":"20180710210000","endtime":"20180711205959"}

I tried to rename it as " type=2&channelid=89..etc. " or setting "Content-Type: application/x-www-form-urlencoded" with no luck. I just want to learn webgrab can handle this kind of post datas or is there any another problem that I have to deal.

Not: I tested on Linux and Win7

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

there nothing wrong with ur actual request,normally this would work as you have it.
the problem is the site,if you change your showsplit to..

index_showsplit.scrub {multi(debug)||||}

this will return the entire page data(written to ur wg log file).
if you look you will see its returning a error about a expired session key.

this is the problem and why your getting no data.

normally when i see this i try the usualy stuff like add any headers i havnt tried or creating a cookie file and when you've tried everything u can think of and still cannot get data then its usually a site that wg will not work with directly.

this is one of those sites.

kvanc
Offline
WG++ Team memberDonator
Joined: 7 months
Last seen: 1 hour

Hi Blackbear;

Thanks for reply. I tried to add some header, created cookie file but none of them work.Also I check what u explained above for debugging and saw

"retcode":"-2",
"desc":"User's session has been expired, please relogin EPG.",
"errorCode":"85983527"

So I tried to preload some adresses which seems to related with authentication and applied credentials, mainly this address

https://portal56.turkcelltvplus.com.tr:33207/EPG/JSON/Authenticate .It normally gives "retcode 0" with "username=guest" and "password=db90e7eb"(always same) on browser but didnt work as well.

Only thing that I havent tried, is php script which is a challenge that I cannot accept now. Anyway thanks again for your help.I will keep watching website.Maybe I can find a different way to make it work

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

ur thinking is good.
that is the correct address that needs to be called(in php script) then save the cookie file which will contain a good session id.

then u request the epg data(using the epg data url) using the cookie file saved in the above step and you will get the correct response.

this all has to be done with a php script(which i prefer to use).
i'm sure it could be done a few diff ways if one was knowledged in programming.

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours
kvanc wrote:

Hi Blackbear;
Thanks for reply. I tried to add some header, created cookie file but none of them work.Also I check what u explained above for debugging and saw
"retcode":"-2",
"desc":"User's session has been expired, please relogin EPG.",
"errorCode":"85983527"
So I tried to preload some adresses which seems to related with authentication and applied credentials, mainly this address
https://portal56.turkcelltvplus.com.tr:33207/EPG/JSON/Authenticate .It normally gives "retcode 0" with "username=guest" and "password=db90e7eb"(always same) on browser but didnt work as well.
Only thing that I havent tried, is php script which is a challenge that I cannot accept now. Anyway thanks again for your help.I will keep watching website.Maybe I can find a different way to make it work

what u said about using the preload is what i would've tried also but the problem is the preload is not setup like the url_index where we can use POST,it cannot do it(i know as i tried many times).i asked Jan and he said nope the preload can only send a regular request(using GET).we cannot do POST or add headers,ect.

then there the issue of the session key even if we could send the request,we need to save it from the preload.

kvanc
Offline
WG++ Team memberDonator
Joined: 7 months
Last seen: 1 hour

Hmm.. that's interesting because in documentation.pdf, there is an example for url_preload with post request(Clause 4.4.3.1).I applied actually same thing but catching some part from preloaded url is a thing that I have never succeed. As far as I know index_variable_element or any global/temp elements are not working with preload...

But I remember one of your ".sh" file for a website tring.al to grab some part of preload page which will be used on next stage.So something like this may work but I guess still that data needs to be converted cookie while post-data has no space for sessionid

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

your right,i forgot about that being added for the preload.
the problem is the cookie file then,its not being saved from the preload and used for the url index or i also notice that this site uses asp.net which wg has been known to have issues with but i dont think its the problem as u do get some response back from the site(usually u get nothing if wg cannot work with a asp.net server).

kvanc
Offline
WG++ Team memberDonator
Joined: 7 months
Last seen: 1 hour

Thanks blackbear.Then I will focus on catching and saving that cookie file. I think similar setup as I mentioned about your .sh file for another site, may work.Instead of preload, I am planning to call that authentication page from inside of wg(type=run) with a sh script via wget(maybe curl - I am not so sure about "post" with wget) and
--save-cookies cookies.txt \
--keep-session-cookies \
--post-data
options then try to request epg page with that cookie file but I'm not so sure about sequence..Loadcookie option shall be defined on site options of ini. so wg may try to check cookie file before requesting required one..(or I am wrong).Anyway I will try when I find some free time.

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

you can use curl in the sh script.

yes the cookie is loaded before the url_index or anything happens(like your element that will use type=run) so it must exist before wg runs.

what you could do is within ur sh script use curl or wget or whatever u think that will work and get the authenicate cookie then run wg from within the script also.

check you browser developer tools,thay are all pretty much the same,i use chromium and with it i can right click on the authenicate url in it and selecy copy,then from the drop down list select copy as curl.

this will give you everything you need which u can paste in a terminal window(to test) or in a sh script.all you need todo is add the part to save the actual cookie file which for curl is -c --cookie-jar [filename]

heres a example,it will save the cookiefile to my users home directory..

kvanc
Offline
WG++ Team memberDonator
Joined: 7 months
Last seen: 1 hour

Finally with curl..I was trying it with wget for 2 days but cannot catch the epg page with cookie file that I saved from auth page. Thanks blackbear, now i will write a script for wg to grab epg data

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 6 hours

this is why i prefer to use the php script method.
its fairly simple todo(i learn from know nothing about php).
i didnt invent this,i think it a a member named becks14 or something like that who first mentioned he used it for a site.
there shud be a few posts with php files here to use as a guide.

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl