• Call: +1 (858) 429-9131

Posts Tagged ‘git’

It’s Phab! That makes your life easier

Wednesday, August 13th, 2014 | Posted in Cloud computing

We have been using plenty of different tools for tracking bugs/product management/project management/to do lists/code review; such as ClearCase, ClearQuest, Bugzilla, Github, Asana, Pivotal Tracker, Google Drive etc. We found Phabricator as a “Too Good To Be True” software engineering web application platform originally developed at Facebook. It has code review, wiki, repository browsing,tickets and a lot more to make Phab more fabulous.

Phabricator is an open source collaboration of web applications which help software companies to build better software. It is a suite of applications. Following are the most important tools in phabricator :
Maniphest – Bug tracker/task management tracker
Diffusion- source code browser
Differential – code review tool that allows developers to easily submit reviews to one another via command line tool when they check in code using Git or Subversion
Phriction – wiki tool

How to setup and configure the code review and project management tool – Phabricator

Installation

Server – 4GB Digital ocean droplet
OS – Ubuntu 14.04

1. Install dependencies

apt-get install mysql-server apache2 dpkg-dev php5 php5-mysql php5-gd php5-dev php5-curl php-apc php5-cli php5-json

2. Get code

#cd /var/www/codereview

git clone https://github.com/phacility/libphutil.git

git clone https://github.com/phacility/arcanist.git

git clone https://github.com/phacility/arcanist.git

3. Configure virtual host entry

#add below lines

#######################################################################

DocumentRoot /var/www/codereview/webroot
RewriteEngine on
RewriteRule ^/rsrc/(.*) – [L,QSA]
RewriteRule ^/favicon.ico – [L,QSA]
RewriteRule ^(.*)$ /index.php?__path__=$1 [B,L,QSA]
Order allow,deny
allow from all
#######################################################################
4. Enable the virtual host entry for phabricator.

# a2ensite phabricator.conf
# service apache2 reload

5. Configure the MySQL database configuration for phabricator

– create database
# /var/www/codereview/phabricator/bin/config set mysql.user mysql_username
# /var/www/codereview/phabricator/bin/config get mysql.pass mysql_password
# /var/www/codereview/phabricator/bin/config get mysql.host mysql_host
# /var/www/codereview/phabricator/bin/config storage upgrade
-tweak mysql

Open /etc/mysql/my.cnf and add the following line under [mysqld] section:

sql-mode = STRICT_ALL_TABLES

#service mysql restart

Set the Base URI of Phabricator install

# /var/www/codereview/phabricator/bin/config set phabricator.base-uri

(eg: phabricator.your-domain.com)

Configure Outbound Email – External SMTP (Google Apps)

Set the following configuration keys using /var/www/codereview/phabricator/bin/config set value

– metamta.mail-adapter -> PhabricatorMailImplementationPHPMailerAdapter
– phpmailer.mailer -> smtp
– phpmailer.smtp-host -> smtp.gmail.com
– phpmailer.smtp-port -> 465
– phpmailer.smtp-user -> Your Google apps mail id
– phpmailer.smtp-password -> set to your password used for authentication
– phpmailer.smtp-protocol -> ssl

Start the phabricator daemons

You can start all the phabricator deamons using the script
# /var/www/codereview/phabricator/bin/phd start
To start daemons at the boot time, add this entry to the file /etc/rc.local

/var/www/codereview/phabricator/bin/phd start

Diffusion repository hosting with git

1. Install git

#apt-get install git

2. Create a local repository directory:

#mkdir -p /data/repo

3. Edit the repository.default-local-path key to the new local repository directory.

Go to the Config -> Repositories -> repository.default-local-path

4. Configure System user accounts

Phabricator uses as many as three user accounts. These are system user accounts on the machine Phabricator runs on, not Phabricator user accounts.

* daemon-user – The user the daemons run as

We will configure the root user to run the daemons

* www-user – The user the web server run as

We will use www-data to be the web user

* vcs-user – The user that users will connect over SSH as

We will configure git user to the vcs-user

To enable SSH access to repositories, edit /etc/sudoers file using visudo to contain:

#includedir /etc/sudoers.d
git ALL=(root) SETENV: NOPASSWD: /usr/bin/git-upload-pack, /usr/bin/git-receive-pack, /usr/bin/git

Since we are going to enable SSH access to the repository, ensure the following holds good.

– Open /etc/shadow and find the line for vcs-user, git.

The second field (which is the password field) must not be set to !!. This value will prevent login. If it is set to !!, edit it and set it to NP (“no password”) instead.

– Open /etc/passwd and find the line for the vcs-user, git.
The last field (which is the login shell) must be set to a real shell. If it is set to something like /bin/false, then sshd will not be able to execute commands. Instead, you should set it to a real shell, like /bin/sh.

– Use phd.user as our daemon user;
# /var/www/phab/phabricator/bin/config phd.user root
# /var/www/phab/phabricator/bin/config set diffusion.ssh-user git

5. Configuring SSH

We will move the normal sshd daemon to another port, say 222. We will use this port to get a normal login shell. We will run highly restrictive sshd on port 22 managed by Phabricator.

Move Normal SSHD

– make a backup of sshd_config before making any changes.

#cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup

– Update /etc/ssh/sshd_config, change the port to some othert port like 222.

Port 222

– Restart sshd and verify that you are able to connect to the new port

ssh -p 222 user@host

Configure and start Phabricator SSHD

We now configure and start a second SSHD instance which will run on port 22. This instance will use special locked down configuration that uses Phabricator to handle the authentication and command execution.

– Create a phabricator-ssh-hook.sh file

– Create a sshd_phabricator config file

– Start a copy of sshd using the new configuration

Create phabricator-ssh-hook.sh: Copy the template in phabricator/resources/sshd/ phabricator-ssh-hook.sh to somewhere like /usr/lib/phabricator-ssh-hook.sh and edit it to have the correct settings

##############################################################

#!/bin/sh

# NOTE: Replace this with the username that you expect users to connect with.
VCSUSER=”git”

# NOTE: Replace this with the path to your Phabricator directory.
ROOT=”/var/www/codereview/phabricator”

if [ “$1” != “$VCSUSER” ];
then
exit 1
fi

exec “$ROOT/bin/ssh-auth” $@
##############################################################

Make it owned by root and restrict editing;

#sudo chown root /usr/lib/phabricator-ssh-hook.sh
#chmod 755 /usr/lib/phabricator-ssh-hook.sh

Create sshd_config for Phabricator: Copy the template in /phabricator/sshd/sshd_config.phabricator.example to somewhere like /etc/ssh/sshd_config.phabricator

Start Phabricator SSHD

#sudo /usr/sbin/sshd -f /etc/ssh/sshd_config.phabricator

Note:-
Add this entry to the /etc/rc.local to start the daemon on startup.

If you did everything correctly, you should be able to run this;

#echo {} | ssh git@phabricator.your-company.com conduit conduit.ping

and get a response like this;

{“result”:”phab-server”,”error_code”:null,”error_info”:null}

You should now be able to access your instance over ssh on port 222 for normal login and administrative purposes. Phabricator SSHD runs on port 22 to handle authentication and command execution.

6. To create a git repository

Go to Diffusion -> New Repository -> Create a New Hosted Repository

Upgrade Phabricator

Since phabricator is under development, you should update frequently. To update phabricator:

– Stop the web server
– Run git pull in libphutil/, arcanist/, and phabricator.
– Run phabricator/bin/storage upgrade.
– Restart the web server.
Also you can use a script similar to this one to automate the process:
http://www.phabricator.com/rsrc/install/update_phabricator.sh

Managing Multiple Branches in Subversion

Friday, May 11th, 2012 | Posted in Cloud computing

Subversion (SVN) is a centralized storage for sharing information. The Subversion repository is like a time machine. It keeps all the records of every changes ever committed and allows you to explore this history by examining previous versions of files and directories as well as the metadata that accompanies them. With a single Subversion command, you can check out the repository exactly as it was at any date or revision number in the past.

At the core of the Subversion is a repository, which is the central store of that system’s data. Any number of Users connect to the repository, and then read or write to these files according to their permissions. The first Subversion tool we will use is svnadmin. This tool is for administration tasks, like creating repositories, making backup dumps, and the like.

Creating the Repository

To create a repository, open the command line, change the current directory to where you want to create it, and run svnadmin:

[shell]$ cd /usr/local
$ svnadmin create repo[/shell]

Where,
repo = repository name.You can call it whatever you like.

Now the SVN URL is like “svn://localhost:/usr/local/repo“. Subversion uses this repository to store information about your projects, like file revisions.

Importing Projects

The svn import command is a quick way to copy an unversioned tree of files into a repository, creating intermediate directories as necessary. svn import doesn’t require a working copy, and your files are immediately committed to the repository. You typically use this when you have an existing tree of files that you want to begin tracking in your Subversion repository. Each project have a recognizable project root in the repository, a directory under which all of the versioned information for that project—and only that project—lives.To create a project under our repository use the following command:

[shell]$ svn mkdir myproject1 svn://localhost:/usr/local/repo[/shell]

Most projects have a recognizable “main line”, or trunk, of development; some branches, which are divergent copies of development lines; and some tags, which are named, stable snapshots of a particular line of development.

Creating trunk,branches and tags

we suggest that each project root contain a trunk subdirectory for the main development line, a branches subdirectory in which specific branches (or collections of branches) will be created, and a tags subdirectory in which specific tags (or collections of tags) will be created.

[shell]$ svn mkdir svn://localhost:/usr/local/repo/myproject1/trunk -m “trunk created”
$ svn mkdir svn://localhost:/usr/local/repo/myproject1/branches -m “branches created”
$ svn mkdir svn://localhost:/usr/local/repo/myproject1/tags -m “tags created”[/shell]

Here “-m” option is using to add the label. Commonly the label is related to the changes or bug fixes which we made in that commit. It is very important to recognize that particular change which we made in each commit. In the above commands we are creating directories, so we gave the label as “trunk created”.

Then we can list the myproject as showing below:

[shell]$ svn list svn://localhost:/usr/local/repo/myproject1[/shell]

Output:

[shell]trunk/
branches/
tags/[/shell]

We can import our project code to the repository like:

[shell]$ svn import mycode svn://localhost:/usr/local/repo/myproject1/trunk -m “Initial Import”[/shell]

Output:

[shell]Adding mycode/file1
Adding mycode/file2
Adding mycode/file3
Adding mycode/file4

Committed revision 1.[/shell]

Checking out the code

Most of the time, you will start using a Subversion repository by performing a checkout of your project. Checking out a directory from a repository creates a working copy of that directory on your local machine. We can check out the repository by  using the following command:

[shell]$ svn checkout svn://localhost:/usr/local/repo/myproject/trunk[/shell]

Output:

[shell]A trunk/file1
A trunk/file2
A trunk/file3
A trunk/file4

Checked out revision 1.[/shell]

Subversion chose to create a working copy in a directory named for the final component of the checkout URL. This occurs only as a convenience to the user when the checkout URL is the only bit of information provided to the svn checkout command. Subversion’s command-line client gives you additional flexibility, though, allowing you to optionally specify the local directory name that Subversion should use for the working copy it creates.

[shell]$ svn checkout svn://localhost:/usr/local/repo/myproject/trunk code[/shell]

Output:

[shell]A code/file1
A code/file2
A code/file3
A code/file4

Checked out revision 1.[/shell]

If there are more developers working in our project, it is better to use the branches. This allows you to save your not-yet-completed work frequently without interfering with others’ changes and while still selectively sharing information with your collaborators.

To Create branches

Creating a branch is very simple. You make a copy of your project tree in the repository using the svn copy command. Since your project’s source code is rooted in the /repo/myproject/trunk directory, it’s that directory that you’ll copy. You can give the branch name as you wish. Here we are using names of  the users.

[shell]$ svn copy svn://localhost:/usr/local/repo/myproject/trunk svn://localhost:/usr/local/repo/myproject/branches/sam -m “branch 1.0.0″[/shell]

Output:

[shell] Committed revision 2[/shell]

This command causes a near-instantaneous commit in the repository, creating a new directory in revision 2. The new directory is a copy of /repo/myproject/trunk. So each user can copy the /repo/myproject/trunk as their own branch and checkout it like

[shell]$ svn checkout svn://localhost:/usr/local/repo/myproject/branches/sam mycode-sam[/shell]

Output:

[shell]A    mycode-sam/file1
A    mycode-sam/file2
A    mycode-sam/file3
A    mycode-sam/file4

Checked out revision 2.[/shell]

There’s nothing special about this working copy; it simply mirrors a different directory in the repository. When you commit changes, however, the other users won’t see them when they updates, because they are using their own working copy. Whenever someone needs to make a long-running change that is likely to disrupt the trunk, a standard procedure is to create a private branch and commit changes there until all the work is complete. Subversion gives you the ability to selectively “copy” changes between branches. And when you’re completely finished with your branch, your entire set of branch changes can be copied to the staging branch and checkout this to the staging server for testing. If the code is ready to move to the production server then merge into the production branch and then to production server. If we are doing this first time there is no staging and production branches, so we need to copy the trunk as staging and production branches. After that we have to move the changes in the user branches.

Merging

In Subversion terminology, the general act of replicating changes from one branch to another is called merging, and it is performed using the command svn merge.

cd "directory you merge into"

[shell]$ svn switch svn://localhost:/usr/local/repo/myproject/staging
$ svn up
$ svn merge -rAA:BB svn://localhost:/usr/local/repo/myproject/branches/sam
$ svn ci -m ‘Merged branch sam into staging'[/shell]

Where,

AA = the revision number of the staging branch

BB = the revision number of the user branch (sam)

Now the staging branch is updated with the user branch, and we can test this code by checking out to the staging server. For this we need to login to the staging server and checkout the code in the correct location.

[shell]$ svn co svn://localhost:/usr/local/repo/myproject/staging /path/to/code[/shell]

Now the code in the staging branch is checked out to the staging server. We have to test this and if all are fine then merge the staging branch to the production branch and then checkout it to the production server as we done with the staging. Also update the trunk with the latest code in the production by merging it with the production branch, because we need to create branches from trunk for the next release. once we have merged the trunk we can retire from the user branch.

Taging

Now the trunk is same as the branch “production” repository. Then we have to tag this for the future reference. The tag is used to save the releases which are going to the production. So we can easily track the revisions which we updated to the production.

[shell]$ svn copy svn://localhost:/usr/local/repo/myproject/trunk svn://localhost:/usr/local/repo/myproject/tags/first-release[/shell]

You can give the tag name as you wish. Here we gave the tag name as first-release. This is the flow of branching, you have to repeat these process on the future releases too. Because of using the tag we can easily track the releases. So if any issue occurs it is easy to revert the production.

Some useful commands in svn are given below

svn add – To add files and directories to the repository

svn mkdir – Create a directory in the repository

svn copy – Copy from one location to other

svn move – To rename

svn delete – Delete from the repository

svn ci – Commit the changes in the working copy

svn diff – Shows line-level details of a particular change

svn log – Shows you broad information: log messages with date and author information attached to revisions and which paths changed in each revision

svn cat – Retrieves a file as it existed in a particular revision number and displays it on your screen

svn list – Displays the files in a directory for any given revision

svn log – Examine merge-sensitive history

Subversion has commands to help you maintain parallel branches of your files and directories. It allows you to create branches by copying your data, and remembers that the copies are related to one another. The main advantage of branching is that the users can work independently with their own repository and so it will not affect the other users and also easy to maintain.

DevOps on EC2 using Capistrano

Thursday, April 5th, 2012 | Posted in Cloud computing

DevOps is the combination of development and operation processes. Cloud with your DevOps offers some fantastic properties. The ability to leverage all the advancements made in software development around repeatability and testability with your infrastructure. The ability to scale up as need be real time and among other things being able to harness the power of self healing systems.

The process piece of devops is about taking the principles behind Agile to the entire continuous software development process. The obvious step is bringing Agile ideas to the operations team, which is sorely needed. Traditionally in the enterprise, the application development team is in charge of gathering business requirements for a software program and writing code. The development team tests their program in an isolated development environment for quality assurance which is later handed over to the operations team. The operations team is tasked with deploying and maintaining the program. The problem with this paradigm is that when the two teams work separately, the development team may not be aware of operational roadblocks that prevent the program from working as anticipated.

Capistrano

Capistrano is a developer tool for running scripts on multiple servers, mainly used for deploying web applications on to the servers. It is typically installed on a workstation, and used to deploy code from your source code management to one, or more servers. Capistrano is originally called “SwitchTower”, the name was changed to Capistrano in March 2006 because of some trademark conflict. It is a time saving command line tool and it is very useful to AWS/EC2 servers because we can deploy the code to 1000’s of aws servers by using a single command. For the security of servers we are commonly using aws ssh key authentication. In capistrano  we use this aws ssh key to deploy the web applications to the aws servers.

In Cloud Computing, deploying applications to production/live servers is always a delicate task. The whole process needs to be quick to minimize downtime. Automating the deployment process helps running repetitive tasks minimizing the possibility human error. It is also a good idea to have a proven and easy way to rollback to a previous version if something goes wrong.

It is a standalone utility that can also integrate nicely with Rails. We simply provide Capistrano with a deployment “recipe” or “formula” that describes our various servers and their roles. It is a single-command deployment. it even allows us to roll a bad version out of production and it revert back to the previous release very easily.

Capistrano Deployment

The main functionality of the Capistrano is to Deploy the rails application which we have already developed and we are using the “SVN” or “GIT” to manage the code. It will transfer all the files of our rails application which we have developed in our local host to aws servers directly by simply executing a simple command in our command prompt.

Steps to deploy a rails application

[shell]gem install capistrano[/shell]

Now,we need to capistranize our rails application using the following commands

[shell]capify .[/shell]

It will create two files

[shell]

config/deploy.rb
capfile .

[/shell]

How to set up deploy.rb file

[shell]

require ‘rubygems’
require ‘activesupport’
set :application, “<application name>”
set :scm_username/ “<username>”
set :use_sudo, false
set :repository, “http://#{scm_username}@www.example.com/svn/trunk”
set :deploy_to, “/var/www/#{application}”
set :deploy_via, :checkout
set :scm, :git
set :user, “root”
role :app, “<domain_name>”
role :web, “<domain_name>”
rold :db, “<domain_name>”, :primary => true
namespace :migrations do
desc “Run the Migrations”
task :up, :roles => :app do
run “cd #{current_path}; rake db:auto:migrate;”
end
task :down, :roles => :app do
run “cd #{current_path}; rake db:drop; rake
db:create”
end
end

[/shell]

where,

‘scm_username’ is your user name
‘application’ is an arbitrary name you create to identify your application on the server
‘use_sudo’ specifies to capistrano that it does not need to append ‘sudo’ before all the commands it will run
‘repository’ identifies where your subversion repository is located

If we aren’t deploying to server’s default path, we need to specify the actual location by using the ‘deploy_to’ variable as given below

[shell]
set :deploy_to, “/var/www/#{application}”
set :deploy_via, :checkout
[/shell]

If we are using the git to manage our source code, specify the SCM by using the ‘scm’ variable as given below

[shell]
set :scm, :git
set :user, “root”
role :app, “<domain_name>”
role :web, “<domain_name>”
rold :db, “<domain_name>”, :primary => true
[/shell]

Since most rails users will have the same domain name for their web,app and database, we can simply use our domain variable we set earlier.

[shell]
namespace :migrations do
desc “Run the Migrations”
task :up, :roles => :app do
run “cd #{current_path}; rake db:auto:migrate;”
end
task :down, :roles => :app do
run “cd #{current_path}; rake db:drop; rake
db:create”
end
end

[/shell]

After completion of our settings in the deploy.rb file, we need to commit the application by using “svn commit” command if we use svn.

Then we need to run the following command:

[shell]

cap deploy:setup

[/shell]

It is used to create the directory structure in server.

[shell]cap deploy:check[/shell]

It checks all the dependencies/things like directory permission and necessary utilities to deploy the application by using capistrano.

If everything is successful, you should see a message like:
“You appear to have all necessary dependencies installed”
And finally deploy the application by using the following command:

[shell]cap deploy[/shell]

Command finished successfully

To Clean up the releases directory, leaving the five most recent releases

[shell]Cap cleanup[/shell]

Prints the difference between what was last deployed, and what is currently in our repository

[shell]cap diff_from_last_deploy[/shell]

To Rolls back to the previously deployed version

[shell]cap deploy:rollback:code[/shell]

Amazon’s EC2 cloud cuts the requisition time of the order & delivery stages down to just minutes. This is already a 75% savings in deployment time! But, without automated deployment, you’ll still need a week to get your application installed.