Hi. I’m Rudy Lee.

Here are some thoughts of mine.

Run Sidekiq Jobs Without Starting Worker Process

You can add the code snippet below to config/initializers/sidekiq.rb if you don’t want to start a separate sidekiq workers.

The configuration below will make sure that the sidekiq jobs will be executed without worker process.

This is handy if you don’t want to open an extra terminal tab or tmux window for worker process.

1
2
3
4
5
# config/initializers/sidekiq.rb
if Rails.env.development?
  require 'sidekiq/testing'
  Sidekiq::Testing.inline!
end

See the official Sidekiq wiki for more information: https://github.com/mperham/sidekiq/wiki/Testing


Regular Expression For A String Containing One Word But Not Another

Last weekend, one our apps that is hosted on Heroku were reporting a lot of R14 errors.

R14 is an error that thrown by Heroku if the machine is running out of memory.

I quickly jumped into (logentries)[https://logentries.com/] to download the log file and opened it in Sublime Text.

However, I was having problem finding the request that is causing the problem because we also have our background job workers reporting R14 errors.

I decided to use regex to find a line that has R14 but doesn’t contain any of the background worker’s name,

This is the regex that I used to find the line:

1
^(?!.*(lowworker|highworker|run)).*R14.*$

The regex above will match the line that has R14 but doesn’t contain lowworker or highworker and run


Monitor Background Jobs with New Relic Query Language

Background

In this blog post, I’ll show you how to set up an alert to monitor your sidekiq jobs using New Relic Query Language and New Relic Alert.

I was given a task to find a solution to monitor our sidekiq jobs. In the past, I used New Relic Sidekiq Plugin ( https://newrelic.com/plugins/secondimpression/131 ) to do this.

The plugin is a Ruby app that connects to your Redis instance, retrieves all of the sidekiq metrics such as jobs, queues and send it to New Relic using the agent library.

This means you need to host the Ruby app somewhere and make sure the plugin can connect to your Redis instance.

However, I found a much better solution using NRQL that doesn’t require you to set up a new server or install any plugins.

New Relic Query Language ( NRQL )

NRQL definition from the official New Relic docs ( https://docs.newrelic.com/docs/insights/nrql-new-relic-query-language/using-nrql/introduction-nrql )

The New Relic Query Language (NRQL), similar to SQL, is a query language for making calls against the Insights event database. NRQL enables you to query data collected from your application and transform that data into dynamic charts. From there, you can interpret your data to better understand how your application is used in a variety of ways.

Using NRQL, You can run a query to get the amount of background jobs that has been executed for a specific time period.

Here is the query to get the count of background jobs:

1
SELECT count(name) FROM Transaction WHERE transactionType='Other'

New Relic Alert and NRQL

First thing you have to do is to create a New Relic alert policy using NRQL. See the screenshots below:

Choose NRQL on the Categorize step

And put the the following query:

1
SELECT count(name) FROM Transaction WHERE transactionType='Other'

After that, you can set up a condition when it will fire the alert.

In the screenshot below, you can see that I set the alert to fire if there are no background jobs running within 15 minutes.

I hope this tutorial will give you an idea on how to monitor your background jobs.


Setting Up Elasticsearch Watcher to Check For Cluster Status on Elastic Cloud

Last week, I was busy migrating our staging and production Elasticsearch clusters from AWS Elasticsearch to Elastic Cloud. The reason behind this migration is because we need dynamic scripting feature in our application and Elastic Cloud is the only managed Elasticsearch hosting that currently supports dynamic scripting.

In terms of pricing, Elastic Cloud is slightly more expensive than AWS Elasticsearch. I think this is because they are using AWS EC2 under the hood. You can compare the pricing of both services here https://aws.amazon.com/elasticsearch-service/pricing/ and https://www.elastic.co/cloud/as-a-service/pricing.

As of now, Elastic Cloud supports the latest version of Elasticsearch which is 5.1.2. If you like living on the edge, I recommend you to try Elastic Cloud.

Creating a watcher

On AWS, we can use Cloud Watch to monitor our Elasticsearch cluster health status as well as monitoring other metrics such as memory and cpu usage. With Elastic Cloud, we have to use Elastic Watcher or Alerting to monitor and trigger alerts.

Currently, there is no UI to set up the watcher on Elastic Cloud. To create a watcher, you have to send a PUT request to your cluster. Please note that this blog post is based on Elasticsearch version 1.7.6 and Elasticsearch Watcher version is 1.0.1.

First thing you have to do is to enable the watcher plugin on the Elastic Cloud clusters configuration. See screenshot below:

The next thing to do is to add an alert recepient email to the Elastic Cloud whitelist. In order to do this, go to Account > Email settings and scroll to the bottom of the page. See screenshot below:

Shortly after that, you will receive an email to confirm this request for whitelist. Confirm the request and you are ready to receive email from Elastic Cloud.

Now open up your REST client app or if you are one of those CLI Guru, you can stick with CURL. As I mentioned earlier, we will send a PUT request to our cluster to create a watcher.

The endpoint of the request is something like this http://elastic-cloud-username:elastic-cloud-password@elastic-cloud-cluster-host:9200/_watcher/watch/cluster_health_watch

You have to replace elastic-cloud-username, elastic-cloud-password and elastic-cloud-cluster-host with your cluster details.

And here is the JSON content of the request: ( please replace the host, auth username, auth password and to email with your cluster details )

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
{
  "trigger" : {
    "schedule" : { "interval" : "10s" }
  },
  "input" : {
    "http" : {
      "request" : {
       "host" : "add-your-elastic-cloud-host-here",
       "port" : 9200,
       "path" : "/_cluster/health",
       "auth" : {
          "basic" : {
            "username" : "your-elastic-cloud-username",
            "password" : "your-elastic-cloud-password"
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : {
      "ctx.payload.status" : { "eq" : "red" }
    }
  },
  "actions" : {
    "send_email" : {
      "email" : {
        "to" : "the-recepient-email-address",
        "subject" : "Cluster Status Warning",
        "body" : "Cluster status is RED"
      }
    }
  }
}

In a nutshell, the request above will create a watcher that will get triggered every 10s, gets the input from our Elasticsearch /_cluster/health page, checks for cluster status ( see condition section ) and sends an email if the condition is matched.

Here is the screenshot of my PUT request using Insomnia REST Client:

After sending the request, we can confirm if the watcher is created successfully or not by visiting this endpoint on our browser http://elasticsearch-cluster-host:9200/_watcher/watch/cluster_health_watch

If the watcher is created successfully, you should see a response like this:

Delete the watcher

You can send a DELETE request if you want to delete the watcher

curl -XDELETE http://elasticsearch-cluster-host:9200/_watcher/watch/cluster_health_watch

Check if your watcher was triggered

You can check if your watcher has been triggered by sending a GET request to /.watch_history*/search?pretty with the following query:

1
2
3
4
5
{
  "query" : {
    "match" : { "result.condition.met" : true }
  }
}

If the query returns a hit, it means that your watcher has been triggered. This is helpful during debugging.

That’s it for now, the next thing I need to figure out is to create alerting for CPU and memory usage. I’ll leave it for another blog post.


Setting Up L2TP VPN to Bypass Great Firewall Of China

Last month, I traveled to China for the second time. Unlike my first trip, this time I am more prepared to bypass the great firewall of China.

During my first trip in China, I was mainly relying on simple SSH tunnel to get access to Gmail and all other blocked services. This solution is unrealiable because I couldn’t use it on my Android phone. Aside from that, I also kept having constant dropouts which explained in this blog post http://blog.zorinaq.com/my-experience-with-the-great-firewall-of-china/

After an extensive research and also a recommendation from one of my friends, I decided to install an L2TP VPN server in Japan. I choose Japan because it’s close to China and I can use Tokyo AWS Region.

I ended up using this ansible playbook that I found when I was looking for tutorials https://github.com/jlund/streisand. It’s basically an Ansible Playbook which help you to install various software such as OpenVPN, L2TP, Tor, etc. You just need to run one shell script and it will install all of those software to your target host.

Running the playbook

Since I already have ansible installed, I just need to clone the project and run the setup script. If not a complete tutorial on how to get started you can check this installation guide here https://github.com/jlund/streisand#installation.

Cloning the project

1
git clone https://github.com/jlund/streisand.git && cd streisand

Running the setup script

1
./streisand

When you run the setup script, it will ask few questions such as where to host the server, AWS Access Keys, etc. I am using AWS because I can use the free tier to run the VPN server. On AWS, it will take around 45 minutes to finish the whole installation process.

Using the VPN

When the whole installation finished, the playbook will create instructions files in the server. You need to SSH to the server in order to view the instruction.

1
ssh ubuntu@server-ip

Go to the NGINX folder to see file

1
cd /var/www/streisand/l2tp-ipsec

Read the instruction

1
cat index.md | more

In the file, you will find instructions on how to connect to the L2TP server from all different operating systems.


HAProxy as a reverse proxy for cloudinary images

We are using in one of our applications Cloudinary to host and resize images on the fly. We are also using Cloudflare for our CDN and DNS management.

I was given a task to setup a CNAME subdomain in CloudFlare to forward the request to Cloudinary. This way we can still have the benefit of serving static images from CDN as well as reducing the Cloudinary bandwidth usage.

My solution is to set HAProxy as a reverse proxy which responsible to fetch images from Cloudinary server. You can see the overview diagram below:

The first thing we have to do is to create an ACL in HAProxy for our cloudinary subdomain

In the configuration below, we are telling HAProxy to forward all requests from cloudinary-asset.rudylee.com to cloudinary-backend:

1
2
3
4
5
6
7
listen  http
        bind 127.0.0.1:8080
        maxconn     18000

        acl host_cloudinary hdr(host) -i cloudinary-asset.rudylee.com

        use_backend cloudinary-backend if host_cloudinary

Next one is to create a new backend.

1
2
3
backend cloudinary-backend
        http-request set-header Host res.cloudinary.com
        server cloudinary res.cloudinary.com:80

Restart HAProxy and you should be able to use the subdomain to serve images from Cloudinary (eg: http://cloudinary-asset.rudylee.com/rudylee/image/upload/12298848/icon/12379593943923.png )

Requesting the images through SSL should work if you have SSL termination configured in your HAProxy.


HAProxy and A/B Testing

Few weeks ago, I was given a task to create an A/B test using HAProxy. I need to make HAProxy to split traffic between two different applications ( Rails and NodeJS )

In this blog post, I’ll explain how to achieve this.

Create ACL for the page you want to A/B test

The first step you have to do is to create an ACL for the URL you want to A/B test. In this example, the URL path is /ab-test-path

1
acl ab_test_url path_beg /ab-test-path

Direct the traffic based on ACL rule and cookie

1
2
3
4
5
6
7
8
9
10
11
# Send user to first backend if they have SITEID cookie with cookie_first_backend value
use_backend first-backend if { req.cook(SITEID) -m beg cookie_first_backend }

# Send user to second backend if they have SITEID cookie with cookie_second_backend value and the URL they request is ab_test_url
use_backend second-backend if { req.cook(SITEID) -m beg cookie_second_backend } ab_test_url

# If the doesn't have any cookie send them to ab-test backend
use_backend ab-test-backend if ab_test_url

# By default send all traffic to the first backend
default_backend first-backend

Create all the backends

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
backend first-backend
        appsession JSESSIONID len 52 timeout 3h
        cookie SERVERID insert

        balance leastconn
        server  backend02a-aws-syd 10.0.105.102:80 check maxconn 3000 inter 30s
        server  backend02b-aws-syd 10.0.106.102:80 check maxconn 3000 inter 30s

backend second-backend
        server  backend03a-aws-syd 10.0.105.103:80 check maxconn 3000 inter 30s

backend ab-test-backend
        balance roundrobin
        cookie SITEID insert indirect nocache maxlife 48h

        server backend02a-aws-syd 10.0.105.102:80 weight 25 cookie cookie_first_backend
        server backend02b-aws-syd 10.0.106.102:80 weight 25 cookie cookie_first_backend

        server backend03a-aws-syd 10.0.105.103:80 weight 50 cookie cookie_second_backend

The final HAProxy config should be something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
listen  http 127.0.0.1:8080
        maxconn     18000

        # A/B Test ACLs
        acl ab_test_url path_beg /ab-test-path

        use_backend first-backend if { req.cook(SITEID) -m beg cookie_first_backend }
        use_backend second-backend if { req.cook(SITEID) -m beg cookie_second_backend } ab_test_url
        use_backend ab-test-backend if ab_test_url

        default_backend first-backend

backend first-backend
        appsession JSESSIONID len 52 timeout 3h
        cookie SERVERID insert

        balance leastconn
        server  backend02a-aws-syd 10.0.105.102:80 check maxconn 3000 inter 30s
        server  backend02b-aws-syd 10.0.106.102:80 check maxconn 3000 inter 30s

backend second-backend
        server  backend03a-aws-syd 10.0.105.103:80 check maxconn 3000 inter 30s

backend ab-test-backend
        balance roundrobin
        cookie SITEID insert indirect nocache maxlife 48h

        server backend02a-aws-syd 10.0.105.102:80 weight 25 cookie cookie_first_backend
        server backend02b-aws-syd 10.0.106.102:80 weight 25 cookie cookie_first_backend

        server backend03a-aws-syd 10.0.105.103:80 weight 50 cookie cookie_second_backend

Create new user on Amazon AMI and give it root access

Setup your new EC2 instance on AWS and choose Amazon AMI.

SSH to your instance using your private key

1
ssh -i <path-to-your-pem-file> ec2-user@<ec2-endpoint-or-ip>

Change to root

1
sudo su

Create new group for your user ( in this case my group name is ‘dev’ )

1
groupadd dev

Create new user and assign it to your recently created group

1
useradd -g dev dev

Give the username root access

1
visudo

Add this to the bottom of the file

1
dev     ALL=(ALL)       NOPASSWD:ALL

Delete the password for ‘dev’ user

1
sudo passwd dev -d

Change to the ‘dev’ user

1
su dev

Try run sudo su whether you can gain root privileges

1
sudo su

Change user back to dev and set authorized_keys for ssh

1
2
3
4
exit
mkdir ~/.ssh
vi ~/.ssh/authorized_keys
chmod -R 700 ~/.ssh

If everything is correct, you should be able to change to root user from dev without providing any password.


Import JSON to Elasticsearch using jq

This is based on http://kevinmarsh.com/2014/10/23/using-jq-to-import-json-into-elasticsearch.html

Install jq first on Mac

1
brew install jq

Import the JSON file to Elasticsearch index

1
cat file.json | jq -c '.[] | {"index": {"_index": "bookmarks", "_type": "bookmark", "_id": .id}}, .' | curl -XPOST localhost:9200/_bulk --data-binary @-

Check for the list of indexes http://localhost:9200/_cat/indices?v

See the list of records http:/localhost:9200//bookmarks/_search


Git

Reopen Last Commit in Git

Run the command below if you want to reopen the last commit in your git

1
git reset --soft HEAD~

This is useful if you miss something in your last commit. Instead of creating new commit and squashing it, you can open last commit and fix it there.