Waking up with smart TV Sony Bravia

I am enjoying my new smart TV from Sony for a for quite a bit of time, and truly speaking I do not regret I haven’t bought plain one which I would connect to one of my Raspberry Pis and have fun with much more funtionalities. Well, I don’t loose any functionalities here, because little device can be still plugged in. But having TV which can do more than simple one has it’s limits. For some time I was being woken up with Sleep as Android app which is really good, if you expect to be woken up between REM phases. But now, when I got bored of that, I wanted to wake up and see my TV running one of my YouTube recommendation. Turns out that I don’t expect much and only thing I need is device running whole the time in the background. Well, I have two Raspberries running 24/7, so I’m good.

One of the good things is, that even if my TV is turned off, it’s actually turned on, but using much less power in standby mode and is being connected to my home WiFi all that time. Also it supports WakeOnLan action by desing, so it’s not much we need to do, to just turn it on. We need to just to install wakeonlan package on our linux distribution and run it with TV’s IP and MAC addresses. On raspbian, as we’re using Raspberry Pi device, it will be as simple as typing

sudo apt-get install wakeonlan

And usage is even much simplier than that

wakeonlan -i 192.168.0.66 "01:23:45:67:89:01"

Where the first argument is TV’s IP and the second one is MAC address. The later one can be easily retrieved from Wlan settings on the device. This is the simpliest way to launch TV, but journey doesn’t end here, as long as I still need to run the YouTube app.

Simple nmap scan of the TV reveals nginx server running on standard 80 port. Nothing unusual at all, besides that with couple minutes of googling brings me to SOAP endpoint, which is really unusual to see in product manufactured in 2016. Anyway, payload is really simple and looks like that

1
2
3
4
5
 <soap:Body>	 	 
   <u:X_SendIRCC xmlns:u="urn:schemas-sonycom:service:IRCC:1">	 	 
     <IRCCCode>AAAAAQAAAAEAAABgAw==</IRCCCode> 
   </u:X_SendIRCC>	 	 
 </soap:Body>

Tag IRCCCode seems to hold some binary command, which corresponds to every button on the remote. One can easily find those commands in Google, but there comes an easier way. GitHub repository with simple bash scripts to send codes and receive whole list from the device. The only catch is that it’s using really simple authentication code, which should be set on TV and in this case is “0000”, but this can be changed by editing .sh file.

Now, with all the tools needed, the only thing we need is to establish proper order of sending remote codes to the TV. In my case, after sending wake signal, I need to wait 5 seconds for device to fully launch. Then HOME button needs to be pressed, and after that I need to navigate into YouTube application on my applications list.

IMG_20170218_220932

The only obstacle here is, that I have no idea what will the cursor position be on the list. But I can force it to be in upper left corner, by sending proper amount of LEFT and UP buttons’ remote codes. After that, the drill is simple: four times RIGHT, once DOWN and hit CONFIRM. Then waiting 20 seconds for YT app to load and et voilà – it works!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
     #!/bin/bash
 
    IP=YOUR_TV_IP
    MAC="YOUR_TV_MAC"
    HOME="AAAAAQAAAAEAAABgAw=="
    UP="AAAAAQAAAAEAAAB0Aw=="
    DOWN="AAAAAQAAAAEAAAB1Aw=="
    LEFT="AAAAAQAAAAEAAAA0Aw=="
    RIGHT="AAAAAQAAAAEAAAAzAw=="
    CONFIRM="AAAAAQAAAAEAAABlAw=="
 
    function sendCmd {
      ./send_command.sh $IP $1
      sleep 1
    }
 
    wakeonlan -i $IP $MAC
    sleep 5
    sendCmd $HOME
    for i in {1..5}; do sendCmd $UP; done
    for i in {1..10}; do sendCmd $LEFT; done
    for i in {1..4}; do sendCmd $RIGHT; done
    sendCmd $DOWN
    sendCmd $CONFIRM
    sleep 20
    sendCmd $CONFIRM

The sleep command is necessary here, because otherwise send_command.sh script works asynchronously and may send ie. CONFIRM command, when YT application is not yet launched, which will spoil the play. Code can be also found on my GitHub repository.

Never ever throw your code away!

This is the what I have learned recently. Sometimes you write solution which seems wrong or one that you decide not to use. So you discard your changes, use git reset HEAD and prepare to new approach. This is wrong! I recently learned why.

We were rewritting functionalities from old system to the new one and needed to decide where to place new module. There were two solutions and there were also two of us, so we decided to take different approaches and meet the day after to decide which is best.

My solutions lost, so we decided that we won’t implement it. I checked out changes from my git repository and moved on with second approach. But suddenly it turned out (as it sometimes does), that the first approach was the proper one. Sad thing. I new the solution and could write the code from memory, but lost some time and nerves then. So even when you thing that your new solution is a crap and won’t work. Never ever discard your changes. It is easier to revert a commit or pop from git stash, than to write same solution from the beginning.

FB custom API: Accepting friend request on Facebook

This is third post in series of web scraping FB for creation of own API. Today I will show how to accept friend request. Previously in entry Your own Facebook API – logging in I showed how to log into your Facebook account.

Getting list of friend request is quite simple using low-end Facebook interface. You just need to go to https://mbasic.facebook.com/friends/center/requests/ and parse every “a” tag which “href” attribute starts with “/a/notifications.php?confirm=”. If you care about name of user who sent you friend request, you’d like to remember last parsed “a” tag with “href” attribute starting with “/friends/hovercard/mbasic/” and read text value from it. Of course, if your account has many friend requsts, not all of them will be available under single page. To get more of them you’d need to find and parse “a” tag which “href” attribute starts with “/friends/center/requests/”. But it is out of this post’s scope.

We need to add class field containing requests url and method which will return friend requests. Whole class should look following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  class Facebook:
    fbUrl = "https://mbasic.facebook.com"
    loggedIn = False
    receivedFriendRequestsUrl = "https://mbasic.facebook.com/friends/center/requests/"
    headers = {"User-Agent": "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0",
               "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
               "Accept - Language": "en-US,en;q=0.5",
               "Accept - Encoding": "gzip, deflate, br",
               "Referer": "https://mbasic.facebook.com/"
    }
 
    def login(self, login, password):
        response = requests.get(self.fbUrl, headers = self.headers)
        parser = LoginParser()
        parser.feed(response.text)
        form = parser.data
        form["email"] = login
        form["pass"] = password
        response = requests.post(parser.action, form, cookies = response.cookies, headers = self.headers, allow_redirects = False)
        self.cookies = response.cookies
 
    def ensureLoggedIn(self):
        if self.cookies is None:
            raise RuntimeError("Not logged in")
 
    def getFriendRequests(self):
        self.ensureLoggedIn()
        parser = FriendConfirmParser()
        response = requests.get(self.receivedFriendRequestsUrl, cookies = self.cookies)
        parser.feed(response.text)
        return [FriendRequest(self.cookies, username, path) for username, path in parser.requests.items()]

As you can see, I created new classes: FriendRequest and FriendConfirmParser. First is an object representing friend request, containing user name and approval link. The second one is another parser, to find information we’re looking for. They should look like that

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
  class FriendRequest:
    def __init__(self, cookies, username, path):
        self.url = Facebook.fbUrl + path
        self.cookies = cookies
        self.username = username
 
    def accept(self):
        print self.url
        print self.cookies
        requests.get(self.url, headers=Facebook.headers, cookies=self.cookies)
 
  class FriendConfirmParser(HTMLParser):
    requests = {}
    insideHovercardTag = False
    lastHovercardTagValue = None
 
    def handle_starttag(self, tag, attrs):
        if tag == "a":
            attrs = {k[0]: k[1] for k in attrs}
 
            if attrs["href"].find("/a/notifications.php?confirm=") == 0:
                self.requests[self.lastHovercardTagValue] = attrs["href"]
            elif attrs["href"].find("/friends/hovercard/mbasic/") == 0:
                self.insideHovercardTag = True
 
    def handle_data(self, data):
        if self.insideHovercardTag:
            self.lastHovercardTagValue = data
 
    def handle_endtag(self, tag):
        if self.insideHovercardTag:
            self.insideHovercardTag = False

Having above code we can easily get couple first friend requests, display usernames or accept their to our friend list, like on example below.

1
2
3
4
5
6
   fb = Facebok()
   fb.login("email", "pass")
   for request in fb.getFriendRequests():
      print request.username
      if request.username == "Joe Doe":
         request.accept()

Setting your Facebook developer account

This is the second part of FB related article. Checkout my previous post here.

It is a good habit to not test your FB applications on your main account. There are several reasons why. The very first one is that you do not want to bother your friends with things you like, post on your wall, send on messenger, etc. The other one is, if you’re not using public API, you can get banned. So what you want to do is create separate account for your testing, invite some friends and see how it goes.

I don’t reccomend downloading a model photos out of internet, even if they are on creative commons license. You will receive many friend invitations but also many disturbing messenger requests with many of them containing dick pics. It is also most likely that someone will notice your profile is fake and report it. The most honest way IMO is to use your own image for your bot application.

Getting new friends is really easy, as there are many groups specially created for that purpose. You can see who’s looking for new contacts and add them or post your own add. After that if not many of them add you as their friend, you can always write simple bot to add new friends from friend suggestions box. In further posts I will describe how to do that from python, but you can also easily make simple capybara script which will do the job.

Inviting friends to your dev account isn’t easy thing as you do not want to add many people the same sex as you, to not get called a fag. You do not want to invite people living near you, to avoid awkward moments when your common friend introduces you and you do not want  to invite very young persons because of obvious reasons. Obeying such restrictions might be hard using capybara, but isn’t always easy when you have our API. Many kids lie about their age and many of them never make their age and location public, so it is a good practice to write simple chatting bot which will ask your new friend couple questions. I will describe how to chat with people in further posts.

Running such account isn’t always easy thing cause you may easily send some weird message to someon or post random stuff on your wall, so remember be gentle and if someone demands to know who you are and why are you inviting them, be honest, say truth and apologize if needed.

Your own Facebook API – logging in

Facebook API is a powerfull tool which provides you an interface to create games, authorize for an application or utility application, like wall content analyser, etc. But there are couple things you cannot use API to. For example, you can easily send message to your friends, but there is no way to receive messages from them. For such and other reasons I decided to web scrape FB in order to create own API. In this case I used low-end web interface available at http://mbasic.facebook.com. In this post I will describe how to create simple logging in python.

We will need to libraries for our purpose: requests  for making calls to FB and HTMLParser to scrape HTML code and extract useful information. We will also need to know how Facebook’s low-end interface work. In this case we will go to http://mbasic.facebook.com (make sure to logout first) and use web inspector tool to preview HTML code. We should see something like below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 <form id="login_form" action="https://mbasic.facebook.com/login.php?refsrc=https%3A%2F%2Fmbasic.facebook.com%2F&amp;lwv=100&amp;refid=8">
  <input autocomplete="off" name="lsd" type="hidden" value="AVrCfvCI" /> 
  <input name="charset_test" type="hidden" value="€,´,€,´,水,Д,Є" /> 
  <input name="version" type="hidden" value="1" /> 
  <input id="ajax" name="ajax" type="hidden" value="0" /> 
  <input id="width" name="width" type="hidden" value="0" /> 
  <input id="pxr" name="pxr" type="hidden" value="0" /> 
  <input id="gps" name="gps" type="hidden" value="0" /> 
  <input id="dimensions" name="dimensions" type="hidden" value="0" /> 
  <input name="m_ts" type="hidden" value="1466018891" /> 
  <input name="li" type="hidden" value="S6xhV6uh5PgJqdeqoQg1mGd-" /> 
  <input class="bi bj bk" name="email" type="text" value="" /> 
  <input class="bi bj bl bm" name="pass" type="password" /> 
  <input class="m n bn bo bp" name="login" type="submit" value="Log In" />
 </form>

As you can see, what we need is to find a form tag with id “login_form” and extract every input field with it’s name and value attribute. To do that we will need to create parser class which derives from HTMLParser. We should end up with someting like that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class LoginParser(HTMLParser):
    isLoginForm = False
    data = {}
 
    def handle_starttag(self, tag, attrs):
        if tag == "form":
            attrs = {k[0]: k[1] for k in attrs}
            if attrs['id'] == "login_form":
                self.isLoginForm = True
                self.action = attrs['action']
        else:
            if self.isLoginForm:
                if tag == "input":
                    name = ""
                    value = ""
                    for key, val in attrs:
                        if key == "name":
                            name = val
                        if key == "value":
                            value = val
 
                    self.data[name] = value
 
    def handle_endtag(self, tag):
        if tag == "form" and self.isLoginForm:
            self.isLoginForm = False

Now we need to connect to Facebook and retrieve login form and send login request. We can do that with following code with usage of requests library

1
2
3
4
5
6
7
        response = requests.get(self.loginUrl, headers = self.headers})
        parser = LoginParser()
        parser.feed(response.text)
        form = parser.data
        form['email'] = login
        form['pass'] = password
        response = requests.post(parser.action, form, cookies = response.cookies, headers = self.headers})

Here we send initial request to get form, then we feed parser with response and extract login form data. After that we set login and password in data dictionary and send post request to Facebook with login and form data, cookies retrieved at the beginning and headers which should simulate desktop browser. Setting headers is not mandatory, but we do not want to let Facebook know, that we are using our own script to do the job. Whole application should look like following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import requests
from HTMLParser import HTMLParser
 
class Facebook:
    loginUrl = "https://mbasic.facebook.com/"
    headers = {"User-Agent": "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0",
               "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
               "Accept - Language": "en-US,en;q=0.5",
               "Accept - Encoding": "gzip, deflate, br",
               "Referer": "https://mbasic.facebook.com/"
    }
 
    def login(self, login, password):
        response = requests.get(self.loginUrl, headers = self.headers})
        parser = LoginParser()
        parser.feed(response.text)
        form = parser.data
        form['email'] = login
        form['pass'] = password
        response = requests.post(parser.action, form, cookies = response.cookies, headers = self.headers})
        print response.text
 
class LoginParser(HTMLParser):
    isLoginForm = False
    data = {}
 
    def handle_starttag(self, tag, attrs):
        if tag == "form":
            attrs = {k[0]: k[1] for k in attrs}
            if attrs['id'] == "login_form":
                self.isLoginForm = True
                self.action = attrs['action']
        else:
            if self.isLoginForm:
                if tag == "input":
                    name = ""
                    value = ""
                    for key, val in attrs:
                        if key == "name":
                            name = val
                        if key == "value":
                            value = val
 
                    self.data[name] = value
 
    def handle_endtag(self, tag):
        if tag == "form" and self.isLoginForm:
            self.isLoginForm = False
 
 
fb = Facebook()
fb.login("[email protected]", "secretpassword")