Today, I have to copy files from one S3 bucket to another S3 bucket which sitting in separate AWS account.
Initially, I was thinking to use S3 clients ( Tranmit or Cyberduck ) to download the files first and manually upload the files again to the other S3 Bucket. However, this approach will consume a lot of bandwidth and really slow if you have a lot of files in your S3 bucket.
After a bit of research, I found that you can easily copy files between two S3 buckets. You can use either s4cmd or AWS CLI.
s3cmd or s4cmd
Run this command to install s4cmd
pip install s4cmd
After you finished setting up the AWS credentials, you can start the copying process.
Today, I was trying to create a backup of production database. The problem is we have two different version of PostgreSQL running on production and these databases can only be accessed from front end(FE) server. We have older version of PostgreSQL client installed in all FE server which means I can’t use it to run pg_dump.
SSH Tunnel to the rescue
One solution to this problem is to create a SSH tunnel. Since I have the latest version of PostgreSQL client installed in my machine, I can run pg_dump locally which will connect to the database through SSH tunnel.
At the moment, I am working on a Ruby on Rails projects using Rails Engines ( you can read more about Rails Engines here: http://guides.rubyonrails.org/engines.html ). In this post, I’ll share my tips and trick on how to configure your vim to work with Rails Engines.
I am using NERDTree Bookmark to quickly jump between different engines. If you are using NERDTree, you can create a bookmark by putting your cursor on one of the Rails Engines directory and use the command below:
:Bookmark <engine name>
After you created the bookmark, you can see the bookmarks list by pressing B inside NERDTree window. See the screenshot below:
I also added these two options to my vimrc file.
" Automatically show bookmarks list when you open NERDTree
NERDTreeChDirMode changes the current working directory of your vim to your bookmark directory. This will also enable my favourite rails.vim feature which is open alternate file.
CtrlP Working Path Mode
It is normal for Rails Engines to share similar directory structure and filenames. However, this creates problem when you want to search a file using CtrlP plugin. Combined with NERDTreeChDirMode, you can tell CtrlP to search only in the current working directory.
Add this option to your vimrc file to enable this feature:
Basic HTTP authentication is one simple way to limit public access to your website prior to launch.
The first thing you need is .htaccess file which contains all the configurations. The second one is .htpasswd containing username and password. You can use this website to generate .htpasswd file for you http://www.htaccesstools.com/htpasswd-generator/
In the sample below, I am trying to enable HTTP authentication only on certain domain. On the first line, I set enviroment variable if the domain name is equal to “www.bundabergfestival.com.au”. On line 7, I tell .htaccess file to deny any access by using the live_uri variable. I hope that explanation is pretty straight forward.
SetEnvIf Host "^www.bundabergfestival.com.au" live_uri
AuthName "Bundaberg Festival Website Coming Soon"AuthType Basic
Allow from all
Deny from env=live_uri
I really enjoyed the course and definitely learned something new from it. It covers the basic concept about directives, services and dependecy injection. I didn’t understand those features when the first time I learned about AngularJS. In the beginning of learning AngularJS, I tended to copy and paste code without understanding the meaning behind it. This caused confusion when I tried to learn more about the framework.
Code School also released another screencast on how to build AngularJS app from scratch. I’ll suggest you to check that one out as well so you can apply the knowledge that you have learned from the course to build real application. However, you have to become the member to get access to the screencast. There are also some other websites that provide AngularJS videos such as http://www.egghead.io and http://www.thinkster.io
At Captiv8, we are using Amazon AWS to host most of our PHP projects. We are heavily rely on Elastic Beanstalk to help us set up PHP environment, database and load balancer. On top of that, we are also managing our own server image based on Amazon AMI. In this image, we installed additional software and packages that we need for our application. However, this approach has a drawback as it is difficult to maintain the image and track changes. Everytime you need to update the image, you have to create new server, install the new software and export it into new image. This will leave you with bunch of different images and it is hard to tell what are the things that have changed inside each image.
In order to solve these problems, I decided to find a way to automate the process. My first attempt was trying to use Chef to provision the Elastic Beanstalk environment. I have been using Chef for a while to provision my Vagrant machines. It is powerful and more convenient in compare with bash scripts. Since I am already familiar with Chef, I started looking at tutorials on how to use Chef with Elastic Beanstalk. Most of the tutorials that I found don’t provide easy way to integrate Chef with Elastic Beanstalk. One of them mentions about using AWS OpsWorks with Chef but I think it is overkill for the time being. So, I ditched Chef and start looking for another solution.
ebextensions is another solution that I found after checking the official documentation of Elastic Beanstalk. With this solution, you need to to create .ebextensions folder inside your project and create a file to define what are the packages that you want to install into the environment. Elastic Beanstalk will automatically run the script every time you deploy a new version of the application. Aside from that, you can also tell ebextensions to execute shell script in the instance or changing permission of a file. You can read more details about ebextension here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html
This is my directory structure at the moment:
Here is the example of my ebextension config file:
#!/usr/bin/env bash## References: # - http://www.hudku.com/blog/configuration-setup-customizing-aws-elastic-beanstalk/# - http://www.hudku.com/blog/security-credentials-customizing-aws/#.elastic-beanstalk-app## Main configuration, change these for each projectappName="change_this"newrelicLicense="newreliclicense"# Check if this is the very first time that this script is runningif([ ! -f /root/.not-a-new-instance.txt ])thennewEC2Instance=truefi# Install applications if this is new instanceif([$newEC2Instance])then# Allow sudo command to be used as part of beanstalk ebextensions scripts without a terminal grep -q 'Defaults:root !requiretty' /etc/sudoers.d/$appName||echo -e 'Defaults:root !requirettyn' > /etc/sudoers.d/$appName chmod 440 /etc/sudoers.d/$appName# Add sudo command if not already present to .bashrc of ec2-user so that we are logged on as root when we use ssh grep -q "sudo -s" /home/ec2-user/.bashrc ||echo -e "nsudo -sn" >> /home/ec2-user/.bashrc
# Install phpMyAdmin yum -y --enablerepo=epel install phpmyadmin
mv /captiv8/.ebextensions/templates/phpMyAdmin/phpMyAdmin.conf /etc/httpd/conf.d/
mv /captiv8/.ebextensions/templates/phpMyAdmin/config.inc.php /etc/phpMyAdmin/
chmod 644 /etc/httpd/conf.d/phpMyAdmin.conf
chmod 644 /etc/phpMyAdmin/config.inc.php
service httpd restart
# Install New Relic rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm
yum -y install newrelic-php5
echo -ne '\n\' | newrelic-install install
mv /captiv8/.ebextensions/templates/newrelic/newrelic.ini /etc/php.d/
chmod 644 /etc/php.d/newrelic.ini
perl -pi -e "s/PHP Application/$appName/g" /etc/php.d/newrelic.ini
perl -pi -e "s/newrelicLicense/$newrelicLicense/g" /etc/php.d/newrelic.ini
# Install New Relic Server Monitor yum -y install newrelic-sysmond
nrsysmond-config --set license_key=$newrelicLicense /etc/init.d/newrelic-sysmond start
service httpd restart
# Install OSSEC yum -y install mysql-devel postgresql-devel
wget http://www.ossec.net/files/ossec-hids-2.7.1.tar.gz -P /captiv8
tar xzvf /captiv8/ossec-hids-2.7.1.tar.gz -C /captiv8
mv /captiv8/.ebextensions/templates/ossec/preloaded-vars.conf /captiv8/ossec-hids-2.7.1/etc/
fi# If new instance, now it is not new anymoreif([$newEC2Instance])thenecho -n "" > /root/.not-a-new-instance.txt
chmod 644 /etc/php.d/.not-a-new-instance
Inside my ebextensions config file, I call the shell script which will install additional software. The benefit of using shell script is you have more options and it is much easier to customize the software. Since the .ebextensions folder is copied to the instance, you can tell the shell script to copy a template config file that you have prepared before hand. I hope you find this blog post useful.
Around October 2013, I decided to move from city to Gladesville which located 9 kilometres north-west of Sydney CBD. It wasn’t easy decision because I need to spend about 2 hours on commuting every day.
For the first couple months, I was fine with the long commuting time. I can distract myself with my phone or sleeping on the bus. However, I started to feel unproductive and I can feel it affected my concentration throughout the day. So, I tried to do some research on the Internet about productive use of commuting time. Most of the articles that I found suggesting to listen either podcasts or audio books.
I decided to give audio book a try and downloaded Eat That Frog by Brian Tracy. If you are interested, you can easily find the audio version of this book on YouTube. At first, I was a little bit skeptical with the result but it turned out to be really helpful. I can feel that my life is back on track again. I started to use Trello to keep my list of tasks. Once in a while, I’ll update my goals and create separate boards for each project.
Since that, I have been listening to several different audio books such as 168 hours by Laura Vanderkam and Getting things done by David Allen. Both of them shared some basic principles from Eat that frog. I prefer Getting things done because it’s more straight forward and focusing on actions you can perform to improve your productivity. Although, I can say that this book is quite to hard understand so you need to listen it more than once to get a good grasp of the concept.