• Call: +1 (858) 429-9131

Posts Tagged ‘ethernet’

Openstack Cloud Software

Wednesday, October 17th, 2012 | Posted in Cloud computing

OpenStack : The Mission

“ To produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively scalable.”

OpenStack is a collection of open source software projects that enterprises/service providers can use to setup and run their cloud compute and storage infrastructure.Rackspace and NASA are the key initial contributors to the stack. Rackspace contributed their “Cloud Files” platform (code) to power the Object Storage part of the OpenStack, while NASA contributed their “Nebula” platform (code) to power the Compute part. OpenStack consortium has managed to have more than 150 members including Canonical, Dell, Citrix etc.

There are 5 main service families under OpenStack

• Nova         –   Compute Service

• Swift         –    Storage Service

• Glance      –    Imaging Service

• Keystone  –    Identity Service

• Horizon    –    UI Service

Open Stack Compute Infrastructure (Nova)

Nova is the Computing Fabric controller for the OpenStack Cloud. All activities needed to support the life cycle of instances within the OpenStack cloud are handled by Nova. This makes Nova a Management Platform that manages compute resources, networking, authorization, and scalability needs of the OpenStack cloud. But, Nova does not provide any virtualization capabilities by itself; instead, it uses libvirt API to interact with supported hypervisors. Nova exposes all its capabilities through a web services API that is compatible with the EC2 API of Amazon Web Services.

Functions and Features:

• Instance life cycle management

• Management of compute resources

• Networking and Authorization

• REST-based API

• Asynchronous eventually consistent communication

• Hypervisor agnostic : support for Xen, XenServer/XCP, KVM, UML, VMware vSphere and Hyper-V

OpenStack Storage Infrastructure (Swift)

Swift provides a distributed, eventually consistent virtual object store for OpenStack. It is analogous to Amazon Web Services – Simple Storage Service (S3). Swift is capable of storing billions of objects distributed across nodes. Swift has built-in redundancy and fail-over management and is capable of archiving and media streaming. It is extremely scalable in terms of both size (several petabytes) and capacity (number of objects).

Functions and Features

• Storage of large number of objects

• Storage of large sized objects

• Data Redundancy

• Archival capabilities – Work with large datasets

• Data container for virtual machines and cloud apps

• Media Streaming capabilities

• Secure storage of objects

• Backup and archival

• Extreme scalability

OpenStack Imaging Service (Glance)

OpenStack Imaging Service is a lookup and retrieval system for virtual machine images. It can be configured to use any one of the following storage backends:

• Local filesystem (default)

• OpenStack Object Store to store images

• S3 storage directly

• S3 storage with Object Store as the intermediate for S3 access.

• HTTP (read-only)

Functions and Features

• Provides imaging service

OpenStack Identity Service (Keystone)

Keystone provides identity and access policy services for all components in the OpenStack family. It implements it’s own REST based API (Identity API). It provides authentication and authorization for all components of OpenStack including (but not limited to) Swift, Glance, Nova. Authentication verifies that a request actually comes from who it says it does. Authorization is verifying whether the authenticated user has access to the services he/she is requesting for.

Keystone provides two ways of authentication. One is username/password based and the other is token based. Apart from that, keystone provides the following services:

• Token Service (that carries authorization information about an authenticated user)

• Catalog Service (that contains a list of available services at the users’ disposal)

• Policy Service (that let’s keystone manage access to specific services by specific users or groups).

Openstack Administrative Web-Interface (Horizon)

Horizon the web based dashboard can be used to manage /administer OpenStack services. It can be used to manage instances and images, create keypairs, attach volumes to instances, manipulate Swift containers etc. Apart from this, dashboard even gives the user access to instance console and can connect to an instance through VNC. Overall, Horizon

Features the following:

• Instance Management – Create or terminate instance, view console logs and connect through VNC, Attaching volumes, etc.

• Access and Security Management – Create security groups, manage keypairs, assign floating IPs, etc.

 • Flavor Management – Manage different flavors or instance virtual hardware templates.

 • Image Management – Edit or delete images.

 • View service catalog.

 • Manage users, quotas and usage for projects.

 • User Management – Create user, etc.

 • Volume Management – Creating Volumes and snapshots.

 • Object Store Manipulation – Create, delete containers and objects.

 • Downloading environment variables for a project.

INSTALLATING OPEN STACK

We can install open stack ESSEX very easily using StackGeek script. Login to your box and install git with apt-get. We’ll become root and do an update first.

sudo  su
apt-get update
apt-get install git

Now checkout the StackGeek scripts from Github:

git clone git://github.com/StackGeek/openstackgeek.git   
cd openstackgeek

Install the Base Scripts

Be sure to take a look at the scripts before you run them. Keep in mind the scripts will periodically prompt you for input, either for confirming installation of a package, or asking you for information for configuration.

Start the installation by running the first script:

./openstack_base_1.sh

When the script finishes you’ll see instructions for manually configuring your network. You can edit the interfaces file by doing a:

vim /etc/network/interfaces

Copy and paste the network code provided by the script into the file and then edit:

auto eth0 
iface eth0 inet static
  address 192.168.1.48		
  network 192.168.1.0		
  netmask 255.255.255.0
 broadcast 192.168.1.255
  gateway 192.168.1.124			
  dns-nameservers 8.8.8.8  
auto eth1

Change the settings for your network configuration and then restart networking and run the next script:

/etc/init.d/networking restart

Then run the second script :

./openstack_base_2.sh

After the second script finishes, you’ll need to set up a logical volume for Nova to use for creating snapshots and volumes. Nova is OpenStack’s compute controller process.

Here’s the output from the format and volume creation process:-

root@manager-System-Product-Name:/openstackgeek# fdisk /dev/sda
Device contains neither a valid DOS partition table,nor Sun,SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xb39fe7af.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p Partition number (1-4, default 1): 3  
First sector (2048-62914559, default 2048): 
 Using default value 2048 Last sector,(2048-62914559,default 62914559): 
Using default value 62914559 
Command (m for help): w The partition table has been altered! 
Calling ioctl() to re-read partition table. Syncing disks.
root@manager-System-Product-Name:/openstackgeek# pvcreate -ff /dev/sda3
 Physical volume "/dev/sda3" successfully created
root@manager-System-Product-Name:/openstackgeek# vgcreate nova-volumes /dev/sda3
 Volume group "nova-volumes" successfully created

Note: Your device names may vary.

Installing MySql

The OpenStack components use MySQL for storing state information. Start the install script for MySQL by entering the following:

./openstack_mysql.sh

You’ll be prompted for a password used for each of the components to talk to MySQL:
Enter a password to be used for the OpenStack services
to talk to MySQL (users nova, glance, keystone): redhat
Note(Here “redhat” is the password given to nova,glance,keystone)

During the installation process you will be prompted for a root password for MySQL. In our install example we use the same password, ‘redhat’. At the end of the MySQL install you’ll be prompted for your root password again.

mysql start/running, process 8796
################################################################################ 
Creating OpenStack databases and users. 
Use your database password when prompted. 
 Run './openstack_keystone.sh' when the script exits. 
################################################################################
Enter password:
After MySQL is running, you should be able to login with any of the OpenStack 
users and/or the root admin account by doing the following:

mysql -u root -predhat
mysql -u nova -predhat nova
mysql -u keystone -predhat keystone
mysql -u glance -predhat glance

Installing Keystone

Keystone is OpenStack’s identity manager. Start the install of Keystone by doing:

./openstack_keystone.sh

You’ll be prompted for a token, the password you entered for OpenStack’s services, and your email address. The email address is used to populate the user’s information in the database.

Enter a token for the OpenStack services to auth wth keystone: redhattoken 
Enter the password you used for the MySQL users (nova, glance, keystone):redhat 
Enter the email address for accounts(nova,glance,keystone):user@company.com
You should be able to query Keystone at this point. 
You’ll need to source the“stackrc” file before you talk to Keystone:
 . ./stackrc   
 keystone user-list    
 Keystone should return a list of users:
+----------------------------------+---------+------------------------+--------+
|                id                | enabled |         email          |  name  |
+----------------------------------+---------+------------------------+--------+
| b32b9017fb954eeeacb10bebf14aceb3 | True    | user@company.com       | demo   |
| bfcbaa1425ae4cd2b8ff1ddcf95c907a | True    | user@company.com       | glance |
| c1ca1604c38443f2856e3818c4ceb4d4 | True    | user@company.com       | nova   |
| dd183fe2daac436682e0550d3c339dde | True    | user@company.com       | admin  |
+----------------------------------+---------+------------------------+--------+

Installing Glance

Glance is OpenStack’s image manager. Start the install of Glance by doing:

./openstack_glance.sh

The script will download an Ubuntu 12.04 LTS cloud image from StackGeek’s S3 bucket.Once it’s done, you should be able to get a list of images:

glance index

Here’s the expected output:

ID              :- 71b8b5d5-a972-48b3-b940-98a74b85ed6a 
Name            :- Ubuntu 12.04 LTS
Disk Format     :- qcow2 
Container Format:- ovf 
Size            :- 226426880

Installing Nova

We’re almost done installing! The last component is the most important one as well. Nova is OpenStack’s compute and network manager. It’s responsible for starting instances, creating snapshots and volumes, and managing the network. Start the Nova install by doing:

./openstack_nova.sh

You’ll immediately be prompted for a few items, including your existing network interface’s IP address, the fixed network address, and the floating pool addresses:

######################################################
The IP address for eth0 is probably 192.168.1.48.
Keep in mind you need an eth1 for this to work.
######################################################
Enter the primary ethernet interface IP: 192.168.1.48
Enter the fixed network (eg. 10.0.2.32/27): 192.168.1.0/24
Enter the fixed starting IP (eg. 10.0.2.33): 192.168.1.1
############################################################################
The floating range can be a subset of your current network. 
Configure your DHCP server to block out the range before you choose it here. 
An example would be 10.0.1.224-255
############################################################################
Enter the floating network (eg. 10.0.1.224/27):  
Enter the floating netowrk size (eg. 32):

The fixed network is a set of IP addresses which will be local to the compute nodes. Think of these addresses as being held and routed internally inside any of the compute node instances.

The floating network is a pool of addresses which can be assigned to the instances you are running. For example, you could start a web server and map an external IP to it for serving a site on the Internet.





Finish Installing Nova


Nova should finish installing after you enter all the network information. When it’s done, you should be able to get a list of images from Glance via Nova:


 nova image-list


And get the expected output we saw earlier from Glance:


root@manager-System-Product-Name:/openstackgeek# nova image-list
+--------------------------------------+------------------+--------+--------+
|                  ID                  |       Name       | Status | Server |
+--------------------------------------+------------------+--------+--------+
| 71b8b5d5-a972-48b3-b940-98a74b85ed6a | Ubuntu 12.04 LTS | ACTIVE |        |
+--------------------------------------+------------------+--------+--------+




Installing Horizon


Horizon is the UI and dashboard controller for OpenStack. Install it by doing:


./openstack_horizon.sh

When it’s done installing, you’ll be given a URL to access the dashboard. 
You’ll be able to login with the user ‘admin’ 
and whatever you entered earlier for your password. 
If you’ve forgotten it, simply grep for it in your environment:

env |grep OS_PASSWORD

The URL will be : http://192.168.1.48


You can login the Openstack dashboard by the following credentials


USER : admin


PASSWORD : redhat

				

				
		

				
		
		

				
MySQL Optimization

				
Friday, December 10th, 2010 | Posted in MySQL


				

					
Database optimization is the process of configuring database to use system resource efficiently and perform tasks quickly. To optimize mysql you should know the work flow of entire system, your hardware, operating system, disk I/O performance etc.

Why to Optimize

You can do more with less. The default mysql setup is optimized for a minimal system because it should work well on a minimal hardware. But when you use a dedicated mysql server with high traffic and complex queries you have to optimize mysql.

MySQL Server tuning Considerations

Here you will find some common optimization parameters.


    

  • MySQL variables
    • 

  • Hardware
    • 

  • Disk
    • 

  • Application
    • 



MySQL Optimization

MySQL global variables don’t have any predefined optimum values. It is a trial and monitor process. It depends on all the above parameters. Here you will see some of the common parameters.

Key-buffer-size

It is size of the buffer used to index blocks for MyISAM tables. On a dedicated mysql server with MyISAM storage engine 25-30% of systems total memory you can allocate for key_buffer_size. To fine tune key_buffer_size you can compare the variables key_reads and the key_read_requests.

This ratio should be at least 1:100.


SHOW STATUS LIKE ‘%key_read%’;

+——————-+————-+

| Variable_name | Value |

+——————-+————-+

| Key_read_requests | 10726813161 |

| Key_reads | 92790146 |

+——————-+————-+

Here the ratio is 1:115 which is acceptable.

But suppose you get a ratio 1: 10 then you need to add more key buffer and upgrade hardware accordingly.

Query Cache

“My website is too slow while loading dynamic pages”. If it is a mysql database related issue, following MySQL variables will be your solution.

query_cache_type

Set the query cache type. There are 3 values 0 ,1 or 2






0
Do not cache any query result




1
Cache query results.




2
Cache results ondemand. Cacheable queries that begin with SELECT SQL_CACHE.






query_cache_size

The amount of memory used to cache query result. Default is 0 which disable query cache.

The optimum value is depend on your application.

query_cache_limit

Do not cache results that are larger than this number of bytes. The default value is 1MB.

Status checking

SHOW STATUS LIKE ‘%qcache%’;

+————————-+———-+

| Variable_name | Value |

+————————-+———-+

| Qcache_free_blocks | 1 |

| Qcache_free_memory | 8371272 |

| Qcache_hits | 23547551 |

| Qcache_inserts | 46909131 |

| Qcache_lowmem_prunes | 5110536 |

| Qcache_not_cached | 2760196 |

| Qcache_queries_in_cache | 0 |

| Qcache_total_blocks | 1 |

+————————-+———-+

There were 46909131 queries and out which 23547551 queries cached and remaining not cached. Here the issue will either the result is greater than query_cache_limit or greater than query_cache_size itself. You have to trial and monitor 🙂

Qcache_lowmem_prunes.

When a query is removed from the query cache, this value will be incremented. If it increases quickly, and you still have memory to spare, you can set query_cache_size high, If it never increases, you can reduce the cache size.


sort_buffer

The sort_buffer is a useful for speed up myisamchk operations. It can also be useful when performing large numbers of sorts.


tmp_table_size


This variable determines the maximum size for a temporary table in memory. The maximum in memory size is minimum of tmp_table_size and max_heap_table_size. You can compare

Created_tmp_disk_tables and Created_tmp_tables to optimize tmp_table_size.


innodb_buffer_pool_size


This variable is target for innodb table and it is similar to key_buffer_size in MyISAM table.

On a dedicated mysql server using innodb you can set this upto 80% of RAM.

Hardware for mysql

If you have large tables(>3GB), you should consider 64 bit hardware as mysql uses a lots of 64bit integers internally.


You need more memory(RAM) if you want mysql to handle large number of connections simultaneously. More RAM will speed up key updates by keeping most of the pages in RAM


Another consideration is Ethernet device, You can use a 1G Ethernet for a dedicated mysql server for fast remote connections.


Disk performance is also an important parameter.

Disk Optimization

Striping disk (RAID 0) will increase both read and write throughput.


Don’t use RAID 1 or mirroring on disk for temporary files.


On Linux, mount the disks with async (default) and noatime.

Optimizing your application

Cache process in your application


Specify the column name in queries(eg dont use SELECT * FROM……)


Use persistent connections


USE EXPLAIN to explain!!.You will see detail below.


Queries and Indexes


Let us start with a simple query SELECT firstname FROM student WHERE id=’145870′;

MySQL start searching from the beginning row to find the student with id 145870. It does not even know it exist or not. An index is a sorted file which have an entry for each row.MySQL can find the corresponding record very quickly by referring this index.

EXPLAIN is a nice tool to understand your queries


EXPLAIN SELECT firstname,lastname FROM student WHERE id=’145870′;


+———-+——+—————+——+———+——+——+————+

| table | type | possible_keys | key | key_len | ref | rows | Extra |

+———-+——+—————+——+———+——+——+————+

| student | ALL | NULL | NULL | NULL | NULL |10000 | where used |

+———-+——+—————+——+———+——+——+————+

The possible_keys is null. In this case mysql will check all the 10000 rows. We can say this query(or table) is not optimized.


Now suppose we have use index for above table and run EXPLAIN again then we will get

+———-+——-+—————+———+———+——-+——+——-+

| table | type | possible_keys | key | key_len | ref | rows | Extra |

+———-+——-+—————+———+———+——-+——+——-+

| employee | const | PRIMARY | PRIMARY | 10 | const | 1 | |

+———-+——-+—————+———+———+——-+——+——-+

The type is “const”, which means that the table has only one matching row. The primary key is being used to find this particular record.


There are many more optimization variables and indxing methods. It is difficult include everything in a single article. But you can start mysql fine tuning while you database is underperfoming.