Hi. I’m Rudy Lee.

Here are some thoughts of mine.

HAProxy as a reverse proxy for cloudinary images

We are using in one of our applications Cloudinary to host and resize images on the fly. We are also using Cloudflare for our CDN and DNS management.

I was given a task to setup a CNAME subdomain in CloudFlare to forward the request to Cloudinary. This way we can still have the benefit of serving static images from CDN as well as reducing the Cloudinary bandwidth usage.

My solution is to set HAProxy as a reverse proxy which responsibility to fetch images from Cloudinary server. You can see the overview diagram below:

The first thing we have to do is to create an ACL in HAProxy for our cloudinary subdomain

In the configuration below, we are telling HAProxy to forward all requests from cloudinary-asset.rudylee.com to cloudinary-backend:

1
2
3
4
5
6
7
listen  http
        bind 127.0.0.1:8080
        maxconn     18000

        acl host_cloudinary hdr(host) -i cloudinary-asset.rudylee.com

        use_backend cloudinary-backend if host_cloudinary

Next one is to create a new backend.

1
2
3
backend cloudinary-backend
        http-request set-header Host res.cloudinary.com
        server cloudinary res.cloudinary.com:80

Restart HAProxy and you should be able to use the subdomain to serve images from Cloudinary (eg: http://cloudinary-asset.rudylee.com/rudylee/image/upload/12298848/icon/12379593943923.png )

Requesting the images through SSL should work if you have SSL termination configured in your HAProxy.


HAProxy and A/B Testing

Few weeks ago, I was given a task to create an A/B test using HAProxy. I need to make HAProxy to split traffic between two different applications ( Rails and NodeJS )

In this blog post, I’ll explain how to achieve this.

Create ACL for the page you want to A/B test

The first step you have to do is to create an ACL for the URL you want to A/B test. In this example, the URL path is /ab-test-path

1
acl ab_test_url path_beg /ab-test-path

Direct the traffic based on ACL rule and cookie

1
2
3
4
5
6
7
8
9
10
11
# Send user to first backend if they have SITEID cookie with cookie_first_backend value
use_backend first-backend if { req.cook(SITEID) -m beg cookie_first_backend }

# Send user to second backend if they have SITEID cookie with cookie_second_backend value and the URL they request is ab_test_url
use_backend second-backend if { req.cook(SITEID) -m beg cookie_second_backend } ab_test_url

# If the doesn't have any cookie send them to ab-test backend
use_backend ab-test-backend if ab_test_url

# By default send all traffic to the first backend
default_backend first-backend

Create all the backends

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
backend first-backend
        appsession JSESSIONID len 52 timeout 3h
        cookie SERVERID insert

        balance leastconn
        server  backend02a-aws-syd 10.0.105.102:80 check maxconn 3000 inter 30s
        server  backend02b-aws-syd 10.0.106.102:80 check maxconn 3000 inter 30s

backend second-backend
        server  backend03a-aws-syd 10.0.105.103:80 check maxconn 3000 inter 30s

backend ab-test-backend
        balance roundrobin
        cookie SITEID insert indirect nocache maxlife 48h

        server backend02a-aws-syd 10.0.105.102:80 weight 25 cookie cookie_first_backend
        server backend02b-aws-syd 10.0.106.102:80 weight 25 cookie cookie_first_backend

        server backend03a-aws-syd 10.0.105.103:80 weight 50 cookie cookie_second_backend

The final HAProxy config should be something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
listen  http 127.0.0.1:8080
        maxconn     18000

        # A/B Test ACLs
        acl ab_test_url path_beg /ab-test-path

        use_backend first-backend if { req.cook(SITEID) -m beg cookie_first_backend }
        use_backend second-backend if { req.cook(SITEID) -m beg cookie_second_backend } ab_test_url
        use_backend ab-test-backend if ab_test_url

        default_backend first-backend

backend first-backend
        appsession JSESSIONID len 52 timeout 3h
        cookie SERVERID insert

        balance leastconn
        server  backend02a-aws-syd 10.0.105.102:80 check maxconn 3000 inter 30s
        server  backend02b-aws-syd 10.0.106.102:80 check maxconn 3000 inter 30s

backend second-backend
        server  backend03a-aws-syd 10.0.105.103:80 check maxconn 3000 inter 30s

backend ab-test-backend
        balance roundrobin
        cookie SITEID insert indirect nocache maxlife 48h

        server backend02a-aws-syd 10.0.105.102:80 weight 25 cookie cookie_first_backend
        server backend02b-aws-syd 10.0.106.102:80 weight 25 cookie cookie_first_backend

        server backend03a-aws-syd 10.0.105.103:80 weight 50 cookie cookie_second_backend

Create new user on Amazon AMI and give it root access

Setup your new EC2 instance on AWS and choose Amazon AMI.

SSH to your instance using your private key

1
ssh -i <path-to-your-pem-file> ec2-user@<ec2-endpoint-or-ip>

Change to root

1
sudo su

Create new group for your user ( in this case my group name is ‘dev’ )

1
groupadd dev

Create new user and assign it to your recently created group

1
useradd -g dev dev

Give the username root access

1
visudo

Add this to the bottom of the file

1
dev     ALL=(ALL)       NOPASSWD:ALL

Delete the password for ‘dev’ user

1
sudo passwd dev -d

Change to the ‘dev’ user

1
su dev

Try run sudo su whether you can gain root privileges

1
sudo su

Change user back to dev and set authorized_keys for ssh

1
2
3
4
exit
mkdir ~/.ssh
vi ~/.ssh/authorized_keys
chmod -R 700 ~/.ssh

If everything is correct, you should be able to change to root user from dev without providing any password.


Import JSON to Elasticsearch using jq

This is based on http://kevinmarsh.com/2014/10/23/using-jq-to-import-json-into-elasticsearch.html

Install jq first on Mac

1
brew install jq

Import the JSON file to Elasticsearch index

1
cat file.json | jq -c '.[] | {"index": {"_index": "bookmarks", "_type": "bookmark", "_id": .id}}, .' | curl -XPOST localhost:9200/_bulk --data-binary @-

Check for the list of indexes http://localhost:9200/_cat/indices?v

See the list of records http:/localhost:9200//bookmarks/_search


Git

Reopen Last Commit in Git

Run the command below if you want to reopen the last commit in your git

1
git reset --soft HEAD~

This is useful if you miss something in your last commit. Instead of creating new commit and squashing it, you can open last commit and fix it there.


AWS

Copying Files Between Two S3 Buckets

Today, I have to copy files from one S3 bucket to another S3 bucket which sitting in separate AWS account.

Initially, I was thinking to use S3 clients ( Tranmit or Cyberduck ) to download the files first and manually upload the files again to the other S3 Bucket. However, this approach will consume a lot of bandwidth and really slow if you have a lot of files in your S3 bucket.

After a bit of research, I found that you can easily copy files between two S3 buckets. You can use either s4cmd or AWS CLI.

s3cmd or s4cmd

Run this command to install s4cmd

1
pip install s4cmd

After you finished setting up the AWS credentials, you can start the copying process.

1
2
3
4
5
# format
s4cmd cp s3://<source-bucket> s3://<target-bucket>/ --recursive

#example
s4cmd cp s3://rudylee-images s3://rudylee-new-images/ --recursive

AWS CLI

Install the cli through pip

1
pip install awscli

And configure it

1
awscli configure

The usage is quite similar to s4cmd, see below:

1
aws s3 cp s3://<source-bucket> s3://<destination-bucket>

I prefer using AWS CLI because it has more options and official support. AWS CLI has a built it support can specify the ACL and permission of the objects.

1
2
# The command below will allow the target bucket owner to have full access to the object
aws s3 cp s3://<source-bucket> s3://<target-bucket> --acl "bucket-owner-full-control" --recursive

Since my target bucket is sitting in separate AWS account, I have to set another permission to allow everyone to upload and delete files into my target bucket.

If you want to follow this approach, make sure to delete that permission after you finished with the copying.

The other option is to set the S3 bucket policy manually, see this link: http://serverfault.com/questions/556077/what-is-causing-access-denied-when-using-the-aws-cli-to-download-from-amazon-s3


Create SSH Tunnel To Backup PostgreSQL Database

Today, I was trying to create a backup of production database. The problem is we have two different version of PostgreSQL running on production and these databases can only be accessed from front end(FE) server. We have older version of PostgreSQL client installed in all FE server which means I can’t use it to run pg_dump.

SSH Tunnel to the rescue

One solution to this problem is to create a SSH tunnel. Since I have the latest version of PostgreSQL client installed in my machine, I can run pg_dump locally which will connect to the database through SSH tunnel.

Here is the command I used to create SSH tunnel:

1
2
# Format: ssh -L <local-port>:<db-hostname>:<db-port> <fe-username>@<fe-hostname>
ssh -L 9000:database.com:5432 ubuntu@production.servers.com

After this you can check whether the SSH tunnel is succesfully created by running this and look for port 9000

1
netstat -na | grep LISTEN

If you confirm that the SSH tunnel is working, you can run psql to connect or pg_dump to backup your database

1
psql -h localhost -p 9000

Rails Engines and Vim

At the moment, I am working on a Ruby on Rails projects using Rails Engines ( you can read more about Rails Engines here: http://guides.rubyonrails.org/engines.html ). In this post, I’ll share my tips and trick on how to configure your vim to work with Rails Engines.

NERDTree Bookmark

I am using NERDTree Bookmark to quickly jump between different engines. If you are using NERDTree, you can create a bookmark by putting your cursor on one of the Rails Engines directory and use the command below:

1
:Bookmark <engine name>

After you created the bookmark, you can see the bookmarks list by pressing B inside NERDTree window. See the screenshot below:

I also added these two options to my vimrc file.

1
2
3
" Automatically show bookmarks list when you open NERDTree
let NERDTreeShowBookmarks=1
let NERDTreeChDirMode=2

NERDTreeChDirMode changes the current working directory of your vim to your bookmark directory. This will also enable my favourite rails.vim feature which is open alternate file.

CtrlP Working Path Mode

It is normal for Rails Engines to share similar directory structure and filenames. However, this creates problem when you want to search a file using CtrlP plugin. Combined with NERDTreeChDirMode, you can tell CtrlP to search only in the current working directory.

Add this option to your vimrc file to enable this feature:

1
let g:ctrlp_working_path_mode = 'a'

That’s it for now, I’ll update this post if I find a better workflow or configuration. If you are interested, you can check my full vimrc file here: https://github.com/rudylee/dotfiles/blob/master/vimrc


Symbolic Links With Vagrant Windows

There are few known limitations when using Vagrant on Windows machine. One of these limitations is the lack of symbolic links support on synced folder.

Symbolic links are used heavily by NPM to create shortcut for the libraries. I posted more details about this here: http://blog.rudylee.com/2013/10/24/fix-npm-symlink-problem-in-vagrant/

Most of the time, you can get away with ‘npm —no-bin-link’ solution. However, you need more robust solution if you are using complex tools such as Grunt or Yeoman.

In this post, I’ll show you the proper way to add symbolic links support to your Vagrant machine.

First, you need to add this code snippet inside your Vagrantfile

1
2
3
config.vm.provider "virtualbox" do |v|
    v.customize ["setextradata", :id, "VBoxInternal2/SharedFoldersEnableSymlinksCreate/v-root", "1"]
end

VirtualBox disables symbolic links for security reasons. In order to pass this restriction, you need to boot up the Vagrant machine in Administrator mode.

You can do this by simply right clicking on your Command Prompt or Git Bash icon and click ‘Run as Administrator’. See the picture below if you can’t find it.

After that, boot up the Vagrant machine normally with ‘vagrant up’ command. Wait until the machine boots up, SSH to the machine and try to create symbolic link in the synced folder.

File path 255 character limit

Another annoying problem you might encounter is file path character limit. This happens quite often if you are using a node module with long name. You can easily solve this by following these steps:

Create ‘node_modules’ folder in your home folder

1
  mkdir ~/node_modules

Add symbolic link to the ‘node_modules’ folder you just created inside your project folder

1
  ln -sf ~/node_modules /vagrant/your-project-folder

This solution will ensure that all the node modules are stored inside home directory instead of synced folder.


Enable HTTP authentication on certain domain

Basic HTTP authentication is one simple way to limit public access to your website prior to launch.

The first thing you need is .htaccess file which contains all the configurations. The second one is .htpasswd containing username and password. You can use this website to generate .htpasswd file for you http://www.htaccesstools.com/htpasswd-generator/

In the sample below, I am trying to enable HTTP authentication only on certain domain. On the first line, I set enviroment variable if the domain name is equal to “www.bundabergfestival.com.au”. On line 7, I tell .htaccess file to deny any access by using the live_uri variable. I hope that explanation is pretty straight forward.

1
2
3
4
5
6
7
8
9
10
SetEnvIf Host "^www.bundabergfestival.com.au" live_uri
AuthName "Bundaberg Festival Website Coming Soon"
AuthType Basic
AuthUserFile /var/app/.htpasswd
AuthGroupFile /dev/null
require valid-user
Order allow,deny
Allow from all
Deny from env=live_uri
Satisfy any