You are here

url_preload.headers customheader to send base64 string fail

7 posts / 0 new
Last post
argo057
Offline
argo057's picture
Joined: 11 months
Last seen: 1 month
url_preload.headers customheader to send base64 string fail

I want to parser EPG from a website which need to login first.
I use customheader as a base64 string like this:

url_preload.headers {customheader=Authorization=Basic R1RUVihJUCk6ODY3YTFkYWMzMGQyYzYzNjk2YWM3NzZlODY4NTIwYTQ5NDE5ZTkyOQ==}

but Webgrabplus just send "Basic R1RUVihJUCk6ODY3YTFkYWMzMGQyYzYzNjk2YWM3NzZlODY4NTIwYTQ5NDE5ZTkyOQ".

How can I send "==" characters?

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 7 hours

try urlencoding the == as %3D%3D
so..
url_preload.headers {customheader=Authorization=Basic R1RUVihJUCk6ODY3YTFkYWMzMGQyYzYzNjk2YWM3NzZlODY4NTIwYTQ5NDE5ZTkyOQ%3D%3D}

if ur using a element for the base64 string then u can do..
element.modify {cleanup(style=urlencode)}

argo057
Offline
argo057's picture
Joined: 11 months
Last seen: 1 month

Thank your response.

When I use "%3D%3D" in customheader, I get it in the request packet. It cannot pass the authentication.

"Authorization: Basic R1RUVihJUCk6ODY3YTFkYWMzMGQyYzYzNjk2YWM3NzZlODY4NTIwYTQ5NDE5ZTkyOQ%3D%3D"

Accroding to "APPENDIX E",
preload cannot support modify

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 7 hours

could you tell me the site so i can do some test,pm me it if u dont want to post it.

btw the == is padding to make the string the correct lenght after encoding.
if u use a online base64decoder u get the same result with or w/out the == so whatever site it is, it should do the same.

argo057
Offline
argo057's picture
Joined: 11 months
Last seen: 1 month

Thank your help.

POST http://gtapi.wowotv.tw/gttv-api_v2/token/get?uuid=
preload request packet ==>

Host: gtapi.wowotv.tw
Accept: */*
Version: 201
Authorization: Basic R1RUVihJUCk6ODY3YTFkYWMzMGQyYzYzNjk2YWM3NzZlODY4NTIwYTQ5NDE5ZTkyOQ==
Proxy-Connection: keep-alive
Accept-Language: zh-Hant-JP;q=1, en-JP;q=0.9
Accept-Encoding: gzip, deflate
Platform: IPhone
IMEI: 9856cf780c45e7f2a9034c3d15ae39c6b8ab8760
Content-Length: 0
User-Agent: NewGttv/2.3 (iPhone; iOS 12.0; Scale/3.00)
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Model: iPhone 6s+

preload response packet <==
{
"data": {
"accessToken": "hKOgY8lWaXowWw8IxtfL3rCD4iLkjNf1",
"expiresIn": 3600000,
"scope": "default"
},
"message": "??Token??",
"status": 200
}

index request packet:
POST http://gtapi.wowotv.tw/gttv-api_v2/program/list

Host: gtapi.wowotv.tw
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Connection: keep-alive
Proxy-Connection: keep-alive
Accept: */*
User-Agent: NewGttv/2.3 (iPhone; iOS 12.0; Scale/3.00)
Accept-Language: zh-Hant-JP;q=1, en-JP;q=0.9
Authorization: Bearer hKOgY8lWaXowWw8IxtfL3rCD4iLkjNf1
Content-Length: 42
--------------------------------------------------------------------
contentId=13787&day=3&startDate=2018-10-03

index response packet <==
{"status":200,"message":"\xe5\x8f\x96\xe5\xbe\x97\xe7\xaf\x80\xe7\x9b\xae\xe8\xa1\xa8\xe6\x88\x90\xe5\x8a\x9f","data":{"dayList"
:[{"date":"2018-10-03","programList":[{"programId":8909293,"name":"\xe6\x9d\xb1\xe6\xa3\xae\xe5\xa4\x9c\xe9\x96\x93\xe6\x96\xb0\
xe8\x81\x9e","reservation":"N","startDateTime":"00:00","endDateTime":"01:00"}........

Blackbear199
Offline
WG++ Team memberDonator
Joined: 3 years
Last seen: 7 hours

there is no way you can do this directly with webgrab as u need to be able to read the preload page to get the token(which u cannot do with webgrab).

here a php script i made to do this for you.
you host this on a webserver(apache,nginx or windows equivalent,ect)
make sure u have the php curl extension installed/enabled as its required.

so all you do it put the included php file in ur www/html folder

to call it you do it from webgrab using the url_index..
example..
i use my own server and my php file in in a folder called php

url_index {url|http://192.168.2.250:89/php/wowotv_tw.php?channel=|channel|&date=|urldate|}

so i assumed the 13787(from your postdata above) is a channel id and the date is self explainatory.

i am not sure what the day=3 might be used for but if u need some help with changes i can do that.

so thats it the php file will return the page data back to webgrab just like if webgrab had requested it itself(it wont know any difference).from there its regular webgrab element scrubbing.

u can rename the php file to whatever you want(remember to change the name in the url_index also).

i didnt bother creating the ini part,i just sent a request via my web browser and it seems to work fine(see screenshot).

argo057
Offline
argo057's picture
Joined: 11 months
Last seen: 1 month

Thank you!!!

It is a good idea to use php as an webgrab agent.
And thank your example.

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl