Deploying in Amazon Web Services

Introduction

ScaleOut StateServer can be licensed from the AWS Marketplace for use in Amazon Web Services running on Amazon EC2. On Amazon EC2, you can quickly deploy many instances of the ScaleOut StateServer service and easily scale the size of the distributed store to meet the needs of your application. ScaleOut Software has released publicly available Linux Amazon Machine Images that are pre-configured with the ScaleOut StateServer daemon, and ScaleOut StateServer’s management tools have been enhanced to simplify the deployment and management of EC2 virtual machine instances. ScaleOut StateServer on AWS allows you to take advantage of cost-effective, pay-as-you-go pricing, so you pay only for the resources you need while allowing flexibility during periods of atypically high or low load. Visit http://www.scaleoutsoftware.com/evaluate-purchase/pricing/ for the most up-to-date pricing details.

Terminology

The following terms defined by Amazon Web Services are used in this help file and defined here for your convenience:

Amazon Machine Image (AMI): Amazon Machine Images are machine images stored within Amazon’s infrastructure. An AMI contains the operating system and other software such as ScaleOut StateServer. A pre-packaged AMI that is configured with ScaleOut StateServer is available in the AWS Marketplace.
Instance: An instance represents a single running copy of an Amazon Machine Image (AMI).
Region: Amazon EC2 allows you to run EC2 instances in multiple geographic locations called regions. When deploying your ScaleOut StateServer instances, it is highly recommended that you select a region with the closest geographical proximity to the majority of your WAN traffic, if applicable.
Availability Zone: Every AWS region comprises two or more isolated units of failure within the Amazon Web Services environment called availability zones. A failure in one availability zone is unlikely to propagate to other availability zones within the same region. Resources within the same availability zone will experience lower average network latency than resources that cross availability zones.
Key Pair: A key pair is a public-key, private-key encryption system used by Linux-based instances for authentication when logging in to the systems via SSH. A key pair consists of a public key and a private key, and the matching key must be provided to authenticate against a running EC2 instance. An instance may have only one key pair defined at launch, and it may not be changed after launch. An instance without a key pair defined at launch will not be able to grant authentication for advanced administration via remote SSH login.
Private IP: A private IP address belongs to a single instance and is only routable from within the instance’s associated EC2 Region. Data transfer fees do not apply to data transferred using private IP addresses. When operating within the same EC2 region, use of the private IP Address is preferred to avoid data transfer fees.
Public IP: A public IP address belongs to a single instance and is routable from within the EC2 environment, including from other EC2 regions, and from external, Internet locations.
Elastic IP (EIP): An Elastic IP (EIP) is a fixed (static) public IP address allocated through EC2 and assigned to a running virtual machine instance. Elastic IPs exist independently of virtual machine instances and may be attached to only a single instance at a time, but they may be reassigned to a different instances with complete transparency to end users. If an Elastic IP is associated with an instance, it invalidates and overrides the original public IP.
Security Group: A security group is a named set of allowed inbound network connection rules for EC2 instances. Each security group consists of a list of protocols, ports, and source IP address ranges. A security group can apply to multiple instances, and an instance can be a member of multiple security groups. Security groups may only be assigned to an instance when the instance is being launched. Changes to a security group’s allowed inbound network connections apply to all instances assigned to that Security Group. By default, the SOSS management tools create a new security group for each deployed SOSS store.
Placement Group: A cluster placement group is a logical entity that enables creating a cluster of instances with special characteristics, such as high speed networking. Using a placement group, a cluster of instances can have low latency, 10 gigabit Ethernet bandwidth connectivity between instances in the cluster.

Prerequisites

To deploy ScaleOut StateServer in Amazon EC2, begin by visiting http://aws.amazon.com/ and signing up for an AWS account.

ScaleOut’s management tools require a set of access credentials to perform actions in EC2 on your behalf. In Amazon Web Services, these credentials are referred to as Access Keys. To retrieve or configure Access Keys for your account, visit the Security Credentials page in the AWS site. Next, record the Access Key ID and the Secret Access Key to be used by ScaleOut StateServer management tools.

	Note
	Credentials defined on this page have full account privileges ("root credentials") and are recommended for debugging and initial testing only. For production environments or where security is a concern, use the AWS Identity and Access Management (IAM) feature to create security-limited credentials. When creating an IAM security policy, the following permissions are required:

IAM Permission	Requirement
ec2:AllocateAddress	Required to create a new Elastic IP addresses.
ec2:AssociateAddress	Required to associate an Elastic IP address to a running instance.
ec2:AuthorizeSecurityGroupIngress	Required to authorize StateServer ports for inbound communication.
ec2:CreatePlacementGroup	Required to create new Placement Groups for groups of SOSS cluster compute instances launched with no explicit existing Placement Group.
ec2:CreateSecurityGroup	Required to create new Security Groups for groups of SOSS instances with no explicit existing Security Group defined at launch.
ec2:CreateTags	Required to launch new EC2 instances (for internal bookkeeping).
ec2:DeleteSecurityGroup	Required to delete auto-generated Security Groups for groups of SOSS instances with no explicit existing Security Group defined at launch.
ec2:DescribeAddresses	Required to query an existing Elastic IP address for association.
ec2:DescribeAvailabilityZones	Required to launch new EC2 instances with a preferred Availability Zone.
ec2:DescribeImages	Required to determine which AMI to use when launching new EC2 instances.
ec2:DescribeInstances	Required to retrieve and display information on existing instances.
ec2:DescribeKeyPairs	Required to list available Key Pairs for selection in the Management Console.
ec2:DescribePlacementGroups	Required to query an existing Placement Group for instance placement and selection in the Management Console.
ec2:DescribeRegions	Required to connect to EC2.
ec2:DescribeSecurityGroups	Required to query an existing Security Group for instance placement or Security Group deletion.
ec2:DescribeSubnets	Required to query an existing VPC for instance placement.
ec2:DescribeVpcs	Required to query an existing VPC Subnet for instance placement.
ec2:RebootInstances	Required to reboot running instances.
ec2:RunInstances	Required to launch new EC2 instances.
ec2:TerminateInstances	Required to terminate existing EC2 instances.

Managing AWS Using the SOSS Command-Line Control Program

The command-line control program, soss_aws_actions.py, encapsulates all the logic necessary to launch and manage a ScaleOut StateServer (SOSS) store running in the Amazon EC2 environment.

This command line control program requires Python 2.6 or greater and the Amazon Python library, boto, which is available through the Python package management utilities pip and easy_install, or may also be installed directly from https://github.com/boto/boto. For Linux-based operating systems, boto may be available in your distribution’s software repository.

The command-line syntax for soss_aws_actions.py is as follows:

soss_aws_actions.py command arguments [options]

where the commands are:

create_group: Create a new host group.
add_instances: Add new instances to an existing group.
read_gateways: Print SOSS gateway information for an existing host group.
list_group: Print SOSS instance information for an existing host group.
terminate: Terminate a single instance or existing host group.
help: Display helps information.

Each command has its own set of required arguments and optional parameters as follows:

create_group

The command-line syntax to create a new SOSS host group is as follows:

soss_aws_actions.py create_group [-h]
    [--access-key ACCESS_KEY]
    [--secret-key SECRET_KEY]
    [--region REGION]
    [--product PRODUCT]
    [--avail_zone AVAIL_ZONE]
    [--version VERSION]
    [--security_group SECURITY_GROUP]
    [--additional_groups ADDITIONAL_GROUPS [ADDITIONAL_GROUPS ...]]
    [--key_pair KEY_NAME]
    [--client CLIENT [CLIENT ...]]
    [--web]
    [--placement_group PLACEMENT_GROUP]
    [--subnet_ids SUBNET_IDS [SUBNET_IDS ...]]
    [--eip_none | --eip_existing | --eip_new]
    [--secure_mgt_port SECURE_MGT_PORT]
    [--secure_svr_port SECURE_SVR_PORT]
    [--mgt_port MGT_PORT]
    [--svr_port SVR_PORT]
    [--int_port INT_PORT]
    store_name num_instances instance_type

where the arguments and options are:

ACCESS_KEY: Access Key ID used for administrative AWS operations. If not specified, will attempt to read from boto config.
SECRET_KEY: Secret access key paired with above access key ID. If not specified, will attempt to read from boto config.
REGION: AWS EC2 region to connect to and query. If not specified, will attempt to read from boto config (default: us-east-1).
PRODUCT: ScaleOut product to launch. One of: SOSS (ScaleOut StateServer), SOCS (ScaleOut ComputeServer), SOHS (ScaleOut hServer), SOSSGEO (ScaleOut StateServer with GeoServer), SOCSGEO (ScaleOut ComputeServer with GeoServer)
store_name: Name for the new AWS SOSS Store (must be unique)
num_instances: Number of instances to launch
instance_type: The type of the instance to launch
AVAIL_ZONE: AWS EC2 availability zone in which to launch the new AWS SOSS host group (default: no preference)
VERSION: SOSS version to install, for example 5.0.3.203. If not specified, use latest version.
SECURITY_GROUP: AWS Security Group Name in which to launch the new AWS SOSS host group. If not specified, create and configure a new Security Group. The Security Group will be modified to support internal communication between instances.
ADDITIONAL_GROUPS: AWS Security Group Name(s) in which to launch the new AWS SOSS host group. These groups are not modified.
KEY_NAME: AWS Key Pair name to allow you to securely connect to your instance after it launches (default: None)
CLIENT: Client access source(s) to add to the Security Group. These sources may be single IP addresses, IP ranges in CIDR format (e.g., 10.11.12.0/24), or the name or unique ID of an existing Security Group (e.g., "My Web Farm" or sg-01234567). Additionally, "localhost" may be specified to determine and add the public-facing IP of the localhost as a client (recommended for management access).
PLACEMENT_GROUP: (Cluster Compute instances only) AWS Placement Group Name within which to launch the new AWS SOSS host group. If not specified, create a new Placement Group.
SUBNET_IDS: AWS VPC subnet ID within which to launch the new AWS SOSS host group. Only one subnet ID is currently supported. If not specified, launch outside of a VPC.
--web: Install the SOSS Web Management Console on the launched instance(s). This option installs Apache and PHP on the instance and configures the SOSS Web Management Console on http://INSTANCE/soss_mgt. If specified, port 80 will be opened to the client access source(s) specified by the --client option in the primary Security Group.
--eip_none: No Elastic IPs will be allocated or assigned (default).
--eip_existing: Assign existing Elastic IPs if available.
--eip_new: Allocate new Elastic IPs and assign them.
SECURE_MGT_PORT: Secure management port (default: 723).
SECURE_SVR_PORT: Secure server port (default: 724).
MGT_PORT: Private management port (default: 720).
SVR_PORT: Private server port (default: 721).
INT_PORT: Private interconnect port (default: 722).
PLACEMENT_GROUP: (Cluster Compute instances only) AWS Placement Group Name in which to launch the new AWS SOSS host group. If not specified, create a new Placement Group.

Examples:

`soss_aws_actions.py create_group MySOSSGroup 20 m2.4xlarge --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY --client sg-01234567`	Create a new SOSS host group of 20 m2.4xlarge instances with the name "MySOSSGroup", and allow a pre-existing Security Group with the id "sg-01234567" to access the private SOSS ports.
`soss_aws_actions.py create_group MySOSSGroup 20 m2.4xlarge --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY --client sg-01234567 --region us-west-1 --avail-zone us-west-1b --key_pair my_ssh_key_pair --eip_new --security_group "ScaleOut" --additional_groups "default"`	As above, but launch in the us-west-1 Region, in the us-west-1b Availability Zone, specifies the "my_ssh_key_pair" as the SSH authentication key pair name, attempt to allocate a new Elastic IP for each instance (maximum of 5 by default), launch the instances into an existing "ScaleOut" Security Group as its primary group, and add the "default" group as a secondary Security Group.

add_instances

The command-line syntax to add instances to an existing SOSS host group is as follows:

soss_aws_actions.py add_instances [-h]
    [--access-key ACCESS_KEY]
    [--secret-key SECRET_KEY]
    [--region REGION]
    [--version VERSION]
    [--web]
    [--eip_none | --eip_existing | --eip_new]
    store_name num_instances instance_type

where the arguments and options are defined above. If specified, the region parameter must be the same region into which the SOSS host group was launched.

Example: Adding 3 m1.large instances to the "MySOSSGroup" SOSS group.

soss_aws_actions.py add_instances MySOSSGroup 3 m1.large --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY

read_gateways

The command-line syntax to display gateway information for an existing SOSS host group is as follows:

soss_aws_actions.py read_gateways [-h]
    [--access-key ACCESS_KEY]
    [--secret-key SECRET_KEY]
    [--region REGION]
    store_name

where the arguments and options are defined above. If specified, the region parameter must be the same region into which the SOSS host group was launched.

Example: List the gateway information for the "MySOSSGroup" SOSS group.

soss_aws_actions.py read_gateways MySOSSGroup --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY

list_group

The command-line syntax to display instance information for all instances in an existing SOSS host group is as follows:

soss_aws_actions.py list_group [-h]
    [--access-key ACCESS_KEY]
    [--secret-key SECRET_KEY]
    [--region REGION] [--name] [--pub_ip]
    [--pub_dns] [--status] [--avail_zone]
    [--security_groups] [--key_pair]
    store_name

where the arguments and options are defined above. If specified, the region parameter must be the same region into which the SOSS host group was launched.

By default, the following information is displayed: Instance ID, Private IP, Instance Type, SOSS Version. Additional information may be displayed for each instance by specifying the following optional parameters; columns are displayed in the order that the parameters are specified.

--name: AWS Name tag
--pub_ip: Public IP
--pub_dns: Public DNS
--status: Status
--avail_zone: Availability Zone
--security_groups: Security Group(s)
--key_pair: SSH Key Pair Name

Example: List the default instance information for the "MySOSSGroup" SOSS host group.

soss_aws_actions.py list_group MySOSSGroup --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY

Example: List additional instance information for the "MySOSSGroup" SOSS group (Instance ID, Private IP, Instance Type, SOSS Version, AWS Name tag, Public DNS).

soss_aws_actions.py list_group MySOSSGroup --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY --name --pub_dns

terminate

The command-line syntax to terminate an instance or existing SOSS host group is as follows:

soss_aws_actions.py terminate [-h]
    [--access-key ACCESS_KEY]
    [--secret-key SECRET_KEY]
    [--region REGION]
    {--store_name STORE_NAME | --instance INSTANCE}

where the arguments and options are defined above. If specified, the region parameter must be the same region into which the SOSS host group was launched. Either a store name or an instance ID must be specified. If a store name is supplied, all instances with that SOSS store name are terminated. If an instance ID is specified with the instance parameter, only the specified instance is terminated.

Example: Terminate all instances in the "MySOSSGroup" SOSS group.

soss_aws_actions.py terminate --store_name MySOSSGroup --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY

Example: Terminate the instance with id "i-01234567".

soss_aws_actions.py terminate --instance i-01234567 --access-key=MY_ACCESS_KEY --secret-key=MY_SECRET_KEY

ScaleOut Product Suite HelpDeploying SOSS in the Cloud