Writing Messenger bot in Java with Spring Framework


Previously I wrote about setting Messenger bot environment on own VPS. Today I’m writing about setting basic bot with Java and Spring Framework. To do so I’m needed:

Aftere setting up basic Facebook page and creation of Facebook app, I had  to select Messenger product, which will be used with my application. After that there is no much to do. Wee need to get:

  • Page access token, which is needed to send message
  • Set webhook for Messenger to receive messages
  • Set submissions for: messaging and messaging subscriptions

First submission is neeed to reply on messages, which is possible only up to 24h after receiving one. Second submission is needed to reply after 24h and reply periodically. But let’s do some coding first.

I need web controller for receiving messages

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@Controller
@RequestMapping("/webhook")
public class WebhookController {
    private final static String TOKEN = "SUPER_SECRET_TOKEN";
 
    @GetMapping
    public @ResponseBody String getHook(
            @RequestParam("hub.mode") String mode,
            @RequestParam("hub.verify_token") String token,
            @RequestParam("hub.challenge") String challenge,
            HttpServletResponse response) {
        if (TOKEN.equals(token) && "subscribe".equals(mode)) {
            response.setStatus(HttpServletResponse.SC_OK);
            return challenge;
        } else {
            response.setStatus(HttpServletResponse.SC_FORBIDDEN);
            return "";
        }
    }
}

Main class

1
2
3
4
5
6
7
8
@SpringBootApplication
public class Messenger {
 
    public static void main(String... args) throws Exception {
        SpringApplication.run(Messenger.class, args);
    }
 
}

And application.properties file in resources directory

1
2
3
4
5
6
7
server.port=4321
server.ssl.key-store=classpath:keystore.jks
server.ssl.keyStoreType=JKS
server.ssl.key-store-password=secret
server.ssl.key-password=secret
server.ssl.keyAlias=tomcat
logging.level.root=DEBUG

After building project and running it on VPS, comes time to set webhook in Facebook app configuration

And create page token to return messages

Now when hook is ready, there is time to create code for receiving messages in WebhookControlelr class

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
@PostMapping
    public @ResponseBody
    String onEvent(@RequestBody Event event, HttpServletResponse response) {
        if ("page".equals(event.getObject())) {
            messageFromEvent(event).ifPresent(m -> 
                messengerController.sendMessage(Message.of(m.getSender().getId(), reverse(m.getMessage().getText()))));
            response.setStatus(HttpStatus.OK.value());
            return "EVENT_RECEIVED";
        }
 
        response.setStatus(HttpStatus.NOT_FOUND.value());
        return "";
    }
 
 
    private static Optional<Message> messageFromEvent(Event event) {
        Message message = null;
        List<Entry> entry = event.getEntry();
 
        if (isNonEmpty(entry)) {
            List<Message> messaging = entry.get(0).getMessaging();
            if (isNonEmpty(messaging)) {
                message = messaging.get(0);
            }
        }
 
        return Optional.of(message);
    }
 
    private static boolean isNonEmpty(List list) {
        return list != null && list.size() > 0;
    }
 
    private static String reverse(String in)  {
        byte[] buf = in.getBytes();
        int pivot = buf.length / 2;
        int len = buf.length - 1;
        for (int i = 0; i < pivot; i++) {
            buf[i] ^= buf[len-i];
            buf[len-i] ^= buf[i];
            buf[i] ^= buf[len-i];
        }
        return new String(buf);
    }

Well, that escalated quickly. Let me explain code above. Facebook sends POST request to mine webhook endpoint, which is deserialized as an Event class object, which JSON representation looks like that

1
{"object":"page","entry":[{"id":"1774684282840000","time":1515003458200,"messaging":[{"sender":{"id":"6515631598500000"},"recipient":{"id":"17746742828000"},"timestamp":1515003457419,"message":{"mid":"mid.$cAAYU8xm2sYBm7oN3i1gvT4v","seq":1123008,"text":"hello bot"}}]}]}]

I needed to create POJO classes to handle all parts of message. Here what’s interesting for us, is sender->id and message->text. Rest can be easily ignored. Here I used messengerController bean to handle reply containing reversed message (test on ASCII chars only plx or it fails :<). There is also a contract with Facebook, that after properly receiving message, application should return HTTP status 200. To send message back to Facebook I used retrofit2 library again. Simple message must only contain recipient and text components.

1
{"recipient":{"id":"17746742828000"},"message":{"text":"hello human"}}

Message sender code looks like below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
public interface MessengerService {
    @POST("/v2.6/me/messages")
    Call<Map<String, String>> sendMessage(@Query("access_token") String token, @Body Message message);
}
 
@Component
public class MessengerController {
    private final static String TOKEN = "PAGE_TOKEN_HERE";
    private final static Retrofit retrofit = new Retrofit.Builder()
            .baseUrl("https://graph.facebook.com/")
            .client(new OkHttpClient())
            .addConverterFactory(JacksonConverterFactory.create())
            .build();
 
    public void sendMessage(Message message) {
        MessengerService service = retrofit.create(MessengerService.class);
 
        try {
            service.sendMessage(TOKEN, message).execute();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

And that’s it. You can test your bot, by writing private message to facebook page connected with bot application as long as you are page / app admin. After that, when everything works properly, you apply for review and describe your bot commands. Later on one of Facebook non technical (!!!) employees will send described commands and some random ones, and if everything will work properly, your bot will be accepted and available for public.

Full project is available at https://github.com/felix-catus/simple-messenger-bot
Please put your keystore in resources directory. If you don’t have one yet, you can learn how to get it here.

Setting environment for Messenger bot

Recently I decided to move my pet project further on and create messenger bot for setting hook on fully booked crossfit workouts and getting notification when vacancy occurs. It’s not rocket science, but required some effort, which I thought would be nice to document here.

To write your own Messenger bot you need to set a web server and get trusted SSL certificate. And this note could be easily ended if I used Heroku to host my application, but I decided to get my hands a little dirty and try myself with own VPS. To do so I had to create my own trusted SSL certificate. I didn’t want to pay, so decided to use Let’s Encrypt which offers full chain of trust, which is required with most browsers as well as with Facebook’s curl command call.

Getting certificate was quite easy. You just need to download Certbot application and proof on your machine, that it is you, who control the domain.

1
2
$ wget https://dl.eff.org/certbot-auto
$ chmod a+x certbot-auto

And then using standalone option I generated certificate. To do so I needed to switch off apache service for a moment, because in this mode Certbot is creating own web server to proof that domain is assigned properly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ sudo ./certbot-auto certonly --standalone  -d test.dusik.pl
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for test.dusik.pl
Waiting for verification...
Cleaning up challenges
 
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/test.dusik.pl/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/test.dusik.pl/privkey.pem
   Your cert will expire on 2018-04-11. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot-auto
   again. To non-interactively renew *all* of your certificates, run
   "certbot-auto renew"
 - If you like Certbot, please consider supporting our work by:
 
   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

After that my certificate resides in /etc/letsencrypt/archive/test.dusik.pl/

1
2
3
4
5
6
$ ll
total 16
-rw-r--r-- 1 root root 1789 01-11 21:41 cert1.pem
-rw-r--r-- 1 root root 1647 01-11 21:41 chain1.pem
-rw-r--r-- 1 root root 3436 01-11 21:41 fullchain1.pem
-rw-r--r-- 1 root root 1704 01-11 21:41 privkey1.pem

To make it usable by web server I needed to export it into p12 cert and then move it into keystore

1
2
3
$ sudo openssl pkcs12 -export -in fullchain1.pem -inkey privkey1.pem -name tomcat -out cert.p12
Enter Export Password: ****
Verifying - Enter Export Password: ****
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
$ sudo keytool -importkeystore -destkeystore keystore.jks -srckeystore cert.p12 -srcstoretype PKCS12
Enter destination keystore password: ****
Re-enter new password: ****
Enter source keystore password:  
Entry for alias tomcat successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
 
$ sudo /opt/jdk1.8.0_65/bin/keytool -list -v -keystore keystore.jks
Enter keystore password: ****
 
Keystore type: JKS
Keystore provider: SUN
 
Your keystore contains 1 entry
 
Alias name: tomcat
Creation date: Jan 11, 2018
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=test.dusik.pl
Issuer: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Serial number: 3c9934c0ddb20d712ba893c7272c35349d4
Valid from: Thu Jan 11 20:41:39 CET 2018 until: Wed Apr 11 21:41:39 CEST 2018
Certificate fingerprints:
	 MD5:  01:A2:59:B3:4A:E4:AE:D2:79:B5:BD:21:A7:63:BA:35
	 SHA1: 37:D3:8B:FB:C1:C5:7F:D1:5F:14:9A:99:7A:37:8C:1B:E3:EE:3F:DE
	 SHA256: 2F:3B:EA:30:87:9D:C6:90:53:7E:F8:2B:7B:07:46:A1:19:03:FA:7B:40:82:4C:1E:9A:8B:76:D7:30:DE:AE:F0
	 Signature algorithm name: SHA256withRSA
	 Version: 3
 
Extensions: 
 
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: ocsp
   accessLocation: URIName: http://ocsp.int-x3.letsencrypt.org
, 
   accessMethod: caIssuers
   accessLocation: URIName: http://cert.int-x3.letsencrypt.org/
]
]
 
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: A8 4A 6A 63 04 7D DD BA   E6 D1 39 B7 A6 45 65 EF  .Jjc......9..Ee.
0010: F3 A8 EC A1                                        ....
]
]
 
#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:false
  PathLen: undefined
]
 
#4: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.23.140.1.2.1]
[]  ]
  [CertificatePolicyId: [1.3.6.1.4.1.44947.1.1.1]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 1A 68 74 74 70 3A 2F   2F 63 70 73 2E 6C 65 74  ..http://cps.let
0010: 73 65 6E 63 72 79 70 74   2E 6F 72 67              sencrypt.org
 
], PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.2
  qualifier: 0000: 30 81 9E 0C 81 9B 54 68   69 73 20 43 65 72 74 69  0.....This Certi
0010: 66 69 63 61 74 65 20 6D   61 79 20 6F 6E 6C 79 20  ficate may only 
0020: 62 65 20 72 65 6C 69 65   64 20 75 70 6F 6E 20 62  be relied upon b
0030: 79 20 52 65 6C 79 69 6E   67 20 50 61 72 74 69 65  y Relying Partie
0040: 73 20 61 6E 64 20 6F 6E   6C 79 20 69 6E 20 61 63  s and only in ac
0050: 63 6F 72 64 61 6E 63 65   20 77 69 74 68 20 74 68  cordance with th
0060: 65 20 43 65 72 74 69 66   69 63 61 74 65 20 50 6F  e Certificate Po
0070: 6C 69 63 79 20 66 6F 75   6E 64 20 61 74 20 68 74  licy found at ht
0080: 74 70 73 3A 2F 2F 6C 65   74 73 65 6E 63 72 79 70  tps://letsencryp
0090: 74 2E 6F 72 67 2F 72 65   70 6F 73 69 74 6F 72 79  t.org/repository
00A0: 2F                                                 /
 
]]  ]
]
 
#5: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]
 
#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]
 
#7: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: test.dusik.pl
]
 
#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 3C FA C9 A8 07 A5 EC A3   61 C7 AA B0 3F E8 D8 4D  <.......a...?..M
0010: 04 9D 74 A6                                        ..t.
]
]
 
Certificate[2]:
Owner: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: a0141420000015385736a0b85eca708
Valid from: Thu Mar 17 17:40:46 CET 2016 until: Wed Mar 17 17:40:46 CET 2021
Certificate fingerprints:
	 MD5:  B1:54:09:27:4F:54:AD:8F:02:3D:3B:85:A5:EC:EC:5D
	 SHA1: E6:A3:B4:5B:06:2D:50:9B:33:82:28:2D:19:6E:FE:97:D5:95:6C:CB
	 SHA256: 25:84:7D:66:8E:B4:F0:4F:DD:40:B1:2B:6B:07:40:C5:67:DA:7D:02:43:08:EB:6C:2C:96:FE:41:D9:DE:21:8D
	 Signature algorithm name: SHA256withRSA
	 Version: 3
 
Extensions: 
 
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: ocsp
   accessLocation: URIName: http://isrg.trustid.ocsp.identrust.com
, 
   accessMethod: caIssuers
   accessLocation: URIName: http://apps.identrust.com/roots/dstrootcax3.p7c
]
]
 
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: C4 A7 B1 A4 7B 2C 71 FA   DB E1 4B 90 75 FF C4 15  .....,q...K.u...
0010: 60 85 89 10                                        `...
]
]
 
#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]
 
#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [URIName: http://crl.identrust.com/DSTROOTCAX3CRL.crl]
]]
 
#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.23.140.1.2.1]
[]  ]
  [CertificatePolicyId: [1.3.6.1.4.1.44947.1.1.1]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 22 68 74 74 70 3A 2F   2F 63 70 73 2E 72 6F 6F  ."http://cps.roo
0010: 74 2D 78 31 2E 6C 65 74   73 65 6E 63 72 79 70 74  t-x1.letsencrypt
0020: 2E 6F 72 67                                        .org
 
]]  ]
]
 
#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]
 
#7: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: A8 4A 6A 63 04 7D DD BA   E6 D1 39 B7 A6 45 65 EF  .Jjc......9..Ee.
0010: F3 A8 EC A1                                        ....
]
]
 
 
 
*******************************************
*******************************************

Above keystore is ready to be imported into tomcat container. All we need to do now is have open port for our application to communicate. Because I’m using Apache server to host this blog, I had two options for that. Either configure reverse proxy mod for Apache or open new port on firewall. I choosed the other option, because I didn’t want to rely on Apache server. I wanted the port to be 4321, because why not. But this port wasn’t reachable from outside, so I needed to do some iptables tweaking:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ sudo iptables-save
# Generated by iptables-save v1.4.7 on Thu Jan 11 22:43:28 2018
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1039644:427395441]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 10000 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 21 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 20 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 35000:35999 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited 
COMMIT
# Completed on Thu Jan 11 22:43:28 2018

This is standard VPS configuration. Ports like HTTP, SSH, HTTPS are opened, but all other are blocked with command

1
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

What we need here is add new ACCEPT rule before REJECT one. We use following command to print line numbers for us

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
2    ACCEPT     icmp --  anywhere             anywhere            
3    ACCEPT     all  --  anywhere             anywhere            
4    ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
5    ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http 
6    ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ndmp 
7    ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp 
8    ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ftp-data 
9    ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpts:35000:35999 flags:FIN,SYN,RST,ACK/SYN 
10   ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:https 
12   REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
 
Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         
1    REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
 
Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Than remove REJECT rule with

1
$ sudo iptables -D INPUT 12

Add new ACCEPT RULE and add REJECT rule at the end:

1
2
$ sudo iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 4321 -j ACCEPT 
$ sudo iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

And that’s it. I created trusted certificate and opened port on firewall. Now there is time to write an application, but I will cover that in another post.

Checking crossfit workout booking availability with jspoon and retrofit2


There are many pros associates with new technologies. You can easily chat with friends without leaving your bed, order things online and get them brought to your place without leaving home and even get pizza and pay for it, before it gets delivered. One of those super fancy features is possibility to register for sports activities. This way you can be sure, that there would be place waiting for you, but it also gives you convenient way to change your plans and change your booking activity. But there are also days, when your favorite workout is fully booked and you’re constantly refreshing the page waiting for someone to change their plans. This might be frustrating and it has happened to me many times. That’s why I decided to write a simple application, which will scrape booking page and notify me, when there will be place available for my favorite workout.

At first I wanted to use python with simple regex solution (you really shouldn’t use regular expressions to parse HTML. It will work for simple tag collection, but it turns out that many web pages are not so regular, as regular expressions are), but then I thought:
– hey, maybe it is possible to deserialize HTML to POJO, just like you deserialize XML. And guess what? Someone already did it, and there is already ready solution for webscraping. You just need to annotate your POJO fields with proper CSS selectors and voilà, any webpage you read, can be transformed to java objects. It’s called jspoon and uses jsoup to parse HTML code. Principle of operation is similar to jackson library.

There is also another interesting library I used in my little project. It’s called retrofit and allows you to create fast and simple HTTP client for any API. It uses OkHttp as a HTTP client and has jspoon dedicated convertor, which makes it perfect solution for our problem.

So, let’s do the job. At first we need to determine the structure of page, we want to scrape. We’re good, it turns out to be simple HTML table

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<tr>
    <td class="hour">
        18:30
    </td>
 
    <td>
        <div style="width:100%;float:left">
            <div class="event" meta:id="10894364" style="color:#050505;background-color:#6feb1d;">
                <span class="eventlength">60 min</span>
                <span class="availability">
 
                    <span class="availability-number">0</span> wolnych</span>
                <p class="event_name">WOD Gymnastics</p>
                <p class="instructor">Artur</p>
                <p class="room"></p>
            </div>
        </div>
    </td>
</tr>

This is actually one workout from page containing every workout for certain day. So we need to actually only get two parameters: an hour and availibility number. But we’ll get also workout name, just to make this solution output clearer data. Our POJO code should look like following class

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
public class Workout {
    @Selector("td.hour")
    String hour;
 
    @Selector("p.event_name")
    String name;
 
    @Selector("span.availability-number")
    Integer available;
 
    public String getHour() {
        return hour;
    }
 
    public String getName() {
        return name;
    }
 
    public Integer getAvailable() {
        return available;
    }
 
    @Override
    public String toString() {
        return "Workout{" +
                "hour=" + hour +
                ", name='" + name + '\'' +
                ", available=" + available +
                '}';
    }
}

We mapped every needed property onto java POJO fields, but now we need to make a collection container, to read all workouts from page. So we do with following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
public class WorkoutDay {
    @Selector("table.calendar_table_day tbody tr")
    List<Workout> workouts;
 
    public List<Workout> getWorkouts() {
        return workouts;
    }
 
    @Override
    public String toString() {
        return "WorkoutDay{" +
                "workouts=" + workouts +
                '}';
    }
}

The catch here is that we don’t point onto workout container, which in our case is table > tbody, but we must indicate CSS selector for workout field, which in our case is table > tbody > tr. Because every table row is mapped to single workout entry.

Having above classes we could easily use simple jspoon invocation and deserialize our workout entries:

1
2
3
Jspoon jspoon = Jspoon.create();
HtmlAdapter<WorkoutDay> htmlAdapter = jspoon.adapter(WorkoutDay.class);
WorkoutDay day = htmlAdapter.fromHtml(htmlContent);

But we’re getting a little creative here and we’ll use retrofit library to get the page. To do so, let’s create our API service interface with properly annotated method:

1
2
3
4
public interface WorkoutService {
    @GET("/kalendarz-zajec?view=DayByHour")
    Call<WorkoutDay> getDay(@Query("day") String date);
}

We declared HTTP GET operation and URL path for getting our workout entries. There is also dynamic query parameter day which is part of URL query and is being set from method parameter. With such interface, it is time for controller code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public class WorkoutController {
    private final Retrofit retrofit = new Retrofit.Builder()
            .baseUrl("https://cf-krakow.cms.efitness.com.pl/")
            .addConverterFactory(JspoonConverterFactory.create())
            .build();
    private final DateTimeFormatter formatter = DateTimeFormatter.ofPattern("dd-MM-uuuu");
 
    public WorkoutDay getDay(LocalDateTime date) throws IOException {
        WorkoutService service = retrofit.create(WorkoutService.class);
        Response<WorkoutDay> response =  service.getDay(date.format(formatter)).execute();
 
        if (response.isSuccessful()) {
            return response.body();
        } else {
            throw new RuntimeException("Something went wrong: " + response.errorBody());
        }
    }
}

As you can see, with builder pattern we’re setting retrofit engine with proper URL and data converter. And later, we use it to create API service and make the call. As simple as several lines of java code. Now let’s use streams to enhance Workout class functionality and return workout for specified hour

1
2
3
    public Optional<Workout> getWorkoutByHour(String hour) {
        return workouts.parallelStream().filter(w -> w.getHour().equals(hour)).findFirst();
    }

and maybe create entry point class, to read user params and invoke controller code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
public class CfNotify {
    private static final WorkoutController controller = new WorkoutController();
    private static final DateTimeFormatter formatter = DateTimeFormatter.ofPattern("HH:mm");
 
    public static void main(String... args) {
        LocalDateTime date = parseArgs(args);
 
        try {
            WorkoutDay day = controller.getDay(date);
 
            Workout workout =
                    day.getWorkoutByHour(date.format(formatter)).orElseThrow(
                            () -> new RuntimeException("No such workout"));
            System.out.println(workout);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
 
    private static LocalDateTime parseArgs(String... args) {
        if (args.length < 1) {
            throw new RuntimeException("Please execute with date parameter in following scheme 2011-12-03T10:15");
        } else {
            return LocalDateTime.from(ISO_LOCAL_DATE_TIME.parse(args[0]));
        }
    }
}

Above code altogether with previous classes will return workout for ISO date time format, which is smth like 2011-12-03T10:15 and will output similar to:

Workout{hour=16:00, name='WOD Beginners', available=2}

Of course you can change the code to output only availability and if it is bigger than 0 notify you in any choosen way. But this is outside of the scope of this post and will be described next time, when I show how to connect above mechanism to Facebook Messenger bot to set hook and get notified when again there is an possibility to register for the workout.

Whole solution can be downloaded / cloned from GitHub repo https://github.com/felix-catus/CfNotify

Waking up with smart TV Sony Bravia

I am enjoying my new smart TV from Sony for a for quite a bit of time, and truly speaking I do not regret I haven’t bought plain one which I would connect to one of my Raspberry Pis and have fun with much more funtionalities. Well, I don’t loose any functionalities here, because little device can be still plugged in. But having TV which can do more than simple one has it’s limits. For some time I was being woken up with Sleep as Android app which is really good, if you expect to be woken up between REM phases. But now, when I got bored of that, I wanted to wake up and see my TV running one of my YouTube recommendation. Turns out that I don’t expect much and only thing I need is device running whole the time in the background. Well, I have two Raspberries running 24/7, so I’m good.

One of the good things is, that even if my TV is turned off, it’s actually turned on, but using much less power in standby mode and is being connected to my home WiFi all that time. Also it supports WakeOnLan action by desing, so it’s not much we need to do, to just turn it on. We need to just to install wakeonlan package on our linux distribution and run it with TV’s IP and MAC addresses. On raspbian, as we’re using Raspberry Pi device, it will be as simple as typing

sudo apt-get install wakeonlan

And usage is even much simplier than that

wakeonlan -i 192.168.0.66 "01:23:45:67:89:01"

Where the first argument is TV’s IP and the second one is MAC address. The later one can be easily retrieved from Wlan settings on the device. This is the simpliest way to launch TV, but journey doesn’t end here, as long as I still need to run the YouTube app.

Simple nmap scan of the TV reveals nginx server running on standard 80 port. Nothing unusual at all, besides that with couple minutes of googling brings me to SOAP endpoint, which is really unusual to see in product manufactured in 2016. Anyway, payload is really simple and looks like that

1
2
3
4
5
 <soap:Body>	 	 
   <u:X_SendIRCC xmlns:u="urn:schemas-sonycom:service:IRCC:1">	 	 
     <IRCCCode>AAAAAQAAAAEAAABgAw==</IRCCCode> 
   </u:X_SendIRCC>	 	 
 </soap:Body>

Tag IRCCCode seems to hold some binary command, which corresponds to every button on the remote. One can easily find those commands in Google, but there comes an easier way. GitHub repository with simple bash scripts to send codes and receive whole list from the device. The only catch is that it’s using really simple authentication code, which should be set on TV and in this case is “0000”, but this can be changed by editing .sh file.

Now, with all the tools needed, the only thing we need is to establish proper order of sending remote codes to the TV. In my case, after sending wake signal, I need to wait 5 seconds for device to fully launch. Then HOME button needs to be pressed, and after that I need to navigate into YouTube application on my applications list.

IMG_20170218_220932

The only obstacle here is, that I have no idea what will the cursor position be on the list. But I can force it to be in upper left corner, by sending proper amount of LEFT and UP buttons’ remote codes. After that, the drill is simple: four times RIGHT, once DOWN and hit CONFIRM. Then waiting 20 seconds for YT app to load and et voilà – it works!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
     #!/bin/bash
 
    IP=YOUR_TV_IP
    MAC="YOUR_TV_MAC"
    HOME="AAAAAQAAAAEAAABgAw=="
    UP="AAAAAQAAAAEAAAB0Aw=="
    DOWN="AAAAAQAAAAEAAAB1Aw=="
    LEFT="AAAAAQAAAAEAAAA0Aw=="
    RIGHT="AAAAAQAAAAEAAAAzAw=="
    CONFIRM="AAAAAQAAAAEAAABlAw=="
 
    function sendCmd {
      ./send_command.sh $IP $1
      sleep 1
    }
 
    wakeonlan -i $IP $MAC
    sleep 5
    sendCmd $HOME
    for i in {1..5}; do sendCmd $UP; done
    for i in {1..10}; do sendCmd $LEFT; done
    for i in {1..4}; do sendCmd $RIGHT; done
    sendCmd $DOWN
    sendCmd $CONFIRM
    sleep 20
    sendCmd $CONFIRM

The sleep command is necessary here, because otherwise send_command.sh script works asynchronously and may send ie. CONFIRM command, when YT application is not yet launched, which will spoil the play. Code can be also found on my GitHub repository.

Never ever throw your code away!

This is the what I have learned recently. Sometimes you write solution which seems wrong or one that you decide not to use. So you discard your changes, use git reset HEAD and prepare to new approach. This is wrong! I recently learned why.

We were rewritting functionalities from old system to the new one and needed to decide where to place new module. There were two solutions and there were also two of us, so we decided to take different approaches and meet the day after to decide which is best.

My solutions lost, so we decided that we won’t implement it. I checked out changes from my git repository and moved on with second approach. But suddenly it turned out (as it sometimes does), that the first approach was the proper one. Sad thing. I new the solution and could write the code from memory, but lost some time and nerves then. So even when you thing that your new solution is a crap and won’t work. Never ever discard your changes. It is easier to revert a commit or pop from git stash, than to write same solution from the beginning.

FB custom API: Accepting friend request on Facebook

This is third post in series of web scraping FB for creation of own API. Today I will show how to accept friend request. Previously in entry Your own Facebook API – logging in I showed how to log into your Facebook account.

Getting list of friend request is quite simple using low-end Facebook interface. You just need to go to https://mbasic.facebook.com/friends/center/requests/ and parse every “a” tag which “href” attribute starts with “/a/notifications.php?confirm=”. If you care about name of user who sent you friend request, you’d like to remember last parsed “a” tag with “href” attribute starting with “/friends/hovercard/mbasic/” and read text value from it. Of course, if your account has many friend requsts, not all of them will be available under single page. To get more of them you’d need to find and parse “a” tag which “href” attribute starts with “/friends/center/requests/”. But it is out of this post’s scope.

We need to add class field containing requests url and method which will return friend requests. Whole class should look following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  class Facebook:
    fbUrl = "https://mbasic.facebook.com"
    loggedIn = False
    receivedFriendRequestsUrl = "https://mbasic.facebook.com/friends/center/requests/"
    headers = {"User-Agent": "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0",
               "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
               "Accept - Language": "en-US,en;q=0.5",
               "Accept - Encoding": "gzip, deflate, br",
               "Referer": "https://mbasic.facebook.com/"
    }
 
    def login(self, login, password):
        response = requests.get(self.fbUrl, headers = self.headers)
        parser = LoginParser()
        parser.feed(response.text)
        form = parser.data
        form["email"] = login
        form["pass"] = password
        response = requests.post(parser.action, form, cookies = response.cookies, headers = self.headers, allow_redirects = False)
        self.cookies = response.cookies
 
    def ensureLoggedIn(self):
        if self.cookies is None:
            raise RuntimeError("Not logged in")
 
    def getFriendRequests(self):
        self.ensureLoggedIn()
        parser = FriendConfirmParser()
        response = requests.get(self.receivedFriendRequestsUrl, cookies = self.cookies)
        parser.feed(response.text)
        return [FriendRequest(self.cookies, username, path) for username, path in parser.requests.items()]

As you can see, I created new classes: FriendRequest and FriendConfirmParser. First is an object representing friend request, containing user name and approval link. The second one is another parser, to find information we’re looking for. They should look like that

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
  class FriendRequest:
    def __init__(self, cookies, username, path):
        self.url = Facebook.fbUrl + path
        self.cookies = cookies
        self.username = username
 
    def accept(self):
        print self.url
        print self.cookies
        requests.get(self.url, headers=Facebook.headers, cookies=self.cookies)
 
  class FriendConfirmParser(HTMLParser):
    requests = {}
    insideHovercardTag = False
    lastHovercardTagValue = None
 
    def handle_starttag(self, tag, attrs):
        if tag == "a":
            attrs = {k[0]: k[1] for k in attrs}
 
            if attrs["href"].find("/a/notifications.php?confirm=") == 0:
                self.requests[self.lastHovercardTagValue] = attrs["href"]
            elif attrs["href"].find("/friends/hovercard/mbasic/") == 0:
                self.insideHovercardTag = True
 
    def handle_data(self, data):
        if self.insideHovercardTag:
            self.lastHovercardTagValue = data
 
    def handle_endtag(self, tag):
        if self.insideHovercardTag:
            self.insideHovercardTag = False

Having above code we can easily get couple first friend requests, display usernames or accept their to our friend list, like on example below.

1
2
3
4
5
6
   fb = Facebok()
   fb.login("email", "pass")
   for request in fb.getFriendRequests():
      print request.username
      if request.username == "Joe Doe":
         request.accept()

Setting your Facebook developer account

This is the second part of FB related article. Checkout my previous post here.

It is a good habit to not test your FB applications on your main account. There are several reasons why. The very first one is that you do not want to bother your friends with things you like, post on your wall, send on messenger, etc. The other one is, if you’re not using public API, you can get banned. So what you want to do is create separate account for your testing, invite some friends and see how it goes.

I don’t reccomend downloading a model photos out of internet, even if they are on creative commons license. You will receive many friend invitations but also many disturbing messenger requests with many of them containing dick pics. It is also most likely that someone will notice your profile is fake and report it. The most honest way IMO is to use your own image for your bot application.

Getting new friends is really easy, as there are many groups specially created for that purpose. You can see who’s looking for new contacts and add them or post your own add. After that if not many of them add you as their friend, you can always write simple bot to add new friends from friend suggestions box. In further posts I will describe how to do that from python, but you can also easily make simple capybara script which will do the job.

Inviting friends to your dev account isn’t easy thing as you do not want to add many people the same sex as you, to not get called a fag. You do not want to invite people living near you, to avoid awkward moments when your common friend introduces you and you do not want  to invite very young persons because of obvious reasons. Obeying such restrictions might be hard using capybara, but isn’t always easy when you have our API. Many kids lie about their age and many of them never make their age and location public, so it is a good practice to write simple chatting bot which will ask your new friend couple questions. I will describe how to chat with people in further posts.

Running such account isn’t always easy thing cause you may easily send some weird message to someon or post random stuff on your wall, so remember be gentle and if someone demands to know who you are and why are you inviting them, be honest, say truth and apologize if needed.

Your own Facebook API – logging in

Facebook API is a powerfull tool which provides you an interface to create games, authorize for an application or utility application, like wall content analyser, etc. But there are couple things you cannot use API to. For example, you can easily send message to your friends, but there is no way to receive messages from them. For such and other reasons I decided to web scrape FB in order to create own API. In this case I used low-end web interface available at http://mbasic.facebook.com. In this post I will describe how to create simple logging in python.

We will need to libraries for our purpose: requests  for making calls to FB and HTMLParser to scrape HTML code and extract useful information. We will also need to know how Facebook’s low-end interface work. In this case we will go to http://mbasic.facebook.com (make sure to logout first) and use web inspector tool to preview HTML code. We should see something like below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 <form id="login_form" action="https://mbasic.facebook.com/login.php?refsrc=https%3A%2F%2Fmbasic.facebook.com%2F&amp;lwv=100&amp;refid=8">
  <input autocomplete="off" name="lsd" type="hidden" value="AVrCfvCI" /> 
  <input name="charset_test" type="hidden" value="€,´,€,´,水,Д,Є" /> 
  <input name="version" type="hidden" value="1" /> 
  <input id="ajax" name="ajax" type="hidden" value="0" /> 
  <input id="width" name="width" type="hidden" value="0" /> 
  <input id="pxr" name="pxr" type="hidden" value="0" /> 
  <input id="gps" name="gps" type="hidden" value="0" /> 
  <input id="dimensions" name="dimensions" type="hidden" value="0" /> 
  <input name="m_ts" type="hidden" value="1466018891" /> 
  <input name="li" type="hidden" value="S6xhV6uh5PgJqdeqoQg1mGd-" /> 
  <input class="bi bj bk" name="email" type="text" value="" /> 
  <input class="bi bj bl bm" name="pass" type="password" /> 
  <input class="m n bn bo bp" name="login" type="submit" value="Log In" />
 </form>

As you can see, what we need is to find a form tag with id “login_form” and extract every input field with it’s name and value attribute. To do that we will need to create parser class which derives from HTMLParser. We should end up with someting like that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class LoginParser(HTMLParser):
    isLoginForm = False
    data = {}
 
    def handle_starttag(self, tag, attrs):
        if tag == "form":
            attrs = {k[0]: k[1] for k in attrs}
            if attrs['id'] == "login_form":
                self.isLoginForm = True
                self.action = attrs['action']
        else:
            if self.isLoginForm:
                if tag == "input":
                    name = ""
                    value = ""
                    for key, val in attrs:
                        if key == "name":
                            name = val
                        if key == "value":
                            value = val
 
                    self.data[name] = value
 
    def handle_endtag(self, tag):
        if tag == "form" and self.isLoginForm:
            self.isLoginForm = False

Now we need to connect to Facebook and retrieve login form and send login request. We can do that with following code with usage of requests library

1
2
3
4
5
6
7
        response = requests.get(self.loginUrl, headers = self.headers})
        parser = LoginParser()
        parser.feed(response.text)
        form = parser.data
        form['email'] = login
        form['pass'] = password
        response = requests.post(parser.action, form, cookies = response.cookies, headers = self.headers})

Here we send initial request to get form, then we feed parser with response and extract login form data. After that we set login and password in data dictionary and send post request to Facebook with login and form data, cookies retrieved at the beginning and headers which should simulate desktop browser. Setting headers is not mandatory, but we do not want to let Facebook know, that we are using our own script to do the job. Whole application should look like following

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import requests
from HTMLParser import HTMLParser
 
class Facebook:
    loginUrl = "https://mbasic.facebook.com/"
    headers = {"User-Agent": "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0",
               "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
               "Accept - Language": "en-US,en;q=0.5",
               "Accept - Encoding": "gzip, deflate, br",
               "Referer": "https://mbasic.facebook.com/"
    }
 
    def login(self, login, password):
        response = requests.get(self.loginUrl, headers = self.headers})
        parser = LoginParser()
        parser.feed(response.text)
        form = parser.data
        form['email'] = login
        form['pass'] = password
        response = requests.post(parser.action, form, cookies = response.cookies, headers = self.headers})
        print response.text
 
class LoginParser(HTMLParser):
    isLoginForm = False
    data = {}
 
    def handle_starttag(self, tag, attrs):
        if tag == "form":
            attrs = {k[0]: k[1] for k in attrs}
            if attrs['id'] == "login_form":
                self.isLoginForm = True
                self.action = attrs['action']
        else:
            if self.isLoginForm:
                if tag == "input":
                    name = ""
                    value = ""
                    for key, val in attrs:
                        if key == "name":
                            name = val
                        if key == "value":
                            value = val
 
                    self.data[name] = value
 
    def handle_endtag(self, tag):
        if tag == "form" and self.isLoginForm:
            self.isLoginForm = False
 
 
fb = Facebook()
fb.login("[email protected]", "secretpassword")

Saving file with vi when forget to sudo

This is a little lifehack I found recently. For sure you ended up, at least once, in a situation when provided some changes to file in vi and then realised that you forget to open editor as a superuser.

vi1

In such situation, you don’t really need to leave your editor. You can easily gain root just for writing operation.

Vi let’s you invoke any shell command when you are in. So for example, you can check if your script is working correctly, just after providing changes and get back to it’s source, just when it ends execution. All shell commands are invoked with exclamation mark prefix, ie. !echo "This is a test"

In our case we need to use linux tee command, which takes standard input and puts it into standard output and may also put it in a specified file.

vi2

By invoking `:w !sudo tee %` we are trying to use editor’s function to write into the file, which should fail. But after that sudo and tee commands come into play, take file’s path (percent file points to file which is being written to by vi) and puts standard input (from the editor) into the file (and also into command mode). After that, vi should notify us that file content has been changed and offer us to reload it.