Object Storage API User Guide

How to sign up to use Object Storage
How to access Object Storage and obtain the API keys
Github repo 'Interoute/object-storage-api'
S3 compatibility
Limits for object size and upload operations
Getting started with Object Storage
Setting up to use s3cmd
Creating a new bucket
Setting the CORS configuration for the bucket
Uploading and downloading objects
Get information about an object or bucket
Object Storage regions
List of available regions in Interoute Object Storage
Creating a bucket in a region using s3cmd
Working with storage policies
Setting up the Python requests module and the authentication program
Creating a bucket with a PUT request
Checking a bucket's storage policy with a GET request
And don't forget the CORS
List of available storage policies for Object Storage
Creating and checking buckets in regions using direct API calls
Versioning: Storing multiple versions of an object
Enable or suspend versioning for a bucket
List the versions of an object (GET request)
GET a specific version of an object
Reading and writing object metadata (key-value pairs)

 

How to sign up to use Object Storage

To use Object Storage you must register first as a customer for Interoute Virtual Data Centre (VDC). However there is no commitment to use VDC services in order to access Object Storage. Your use of Object Storage or VDC will be entirely billed on a pay-as-you-go basis.

If you are already an Interoute VDC customer then you can start to use Object Storage immediately.

You will need to start by logging in to the Object Storage web interface to obtain your API access keys.

With your VDC login details, you can access the My Services portal to connect to Object Storage. See this video for guidance: https://cloudstore.interoute.com/resource-centre/videos/overview-vdc-myservices-cloudstore .

How to access Object Storage and obtain the API keys

The user interface for Object Storage is accessed via the Interoute 'My Services' web portal.

Login to the My Services website at https://myservices.interoute.com.(For guidance, see https://cloudstore.interoute.com/knowledge-centre/library/Introduction-My-Services .)

Click on Products in the top menu and select VDC 2.0.

In the 'VDC 2.0' tile window which appears, click the Object Storage tab:

The first time that you start the Object Storage interface, you should see a welcome screen like the following:

The 'API keys' at the bottom of the welcome screen—API Key and Secret Key—are what you will need to access Object Storage via the API.

You can find these keys at any time later from the Api Access button on the main screen of Object Storage.

For information about working with the graphical user interface, see the Object Storage User Guide.

Github repo 'Interoute/object-storage-api'

A repo has been created with useful files for working with the Object Storage API. See:

https://github.com/Interoute/object-storage-api

The JSON file interoute-object-storage-refdata.json contains all of the reference information about regions, API endpoints and storage policies in a structured format for programming use.

S3 compatibility

The Object Storage API is compatible with the Amazon S3 standard. This means that you can connect to Object Storage using any client program or programming library that is based on that standard.

There are numerous open source and proprietary client programs available which will connect Object Storage with your computer operating system as a 'storage device' for different purposes, such as a networked drive or a remote backup device.

The following client tools have been tested with Interoute Object Storage:

S3 compatibility means that, in most cases, you can switch your client tool between Interoute Object Storage and any other compatible object storage service by simply changing the API access keys and API host endpoint address.

Most client tools require a particular form of configuration file containing the API keys and information about the host server. For example files, see https://github.com/Interoute/object-storage-api. An example based on using the 's3cmd' tool is given in the 'Getting Started' section.

Limits for object size and upload operations

For uploading using the API, including using a client tool, the maximum allowed PUT size (that is, for one API call) is 10 GB. Using a multipart upload, the platform allows a maximum of 10,000 part-uploads. Note that client tools may impose a 'single PUT' limit that is smaller than 10 GB. Multipart uploads are recommended for objects greater than 100 MB in size.

Interoute recommends that the size of a single stored object should not exceed 5 TB. Please contact Interoute Support for technical advice on storing very large objects in Interoute Object Storage.

Note that the upload limits are different when using the user interface: multipart uploads are applied automatically (based on a fixed 10 MB part-upload size) and the user interface can upload objects with a maximum size of 96 GB.

Getting started with Object Storage

In Object Storage, files are stored as 'objects'. Every object must be contained in a 'bucket'. And buckets can be sub-divided through the use of 'folders'.

Important

Bucket names are unique for all users, that is, you cannot use a bucket name that any other user has already used.

Buckets created using the API will not be correctly visible in the Object Storage user interface, until you do a 'put bucket CORS' API call for the bucket. See below for an example of doing this.

 

There is a limit of 100 buckets per account and an unlimited number of objects per bucket.

The following is a short tutorial to setting up the 's3cmd' client tool, and performing some basic API operations.

Setting up to use s3cmd

s3cmd is a command-line tool for Linux (or Mac OS), written in Python. You can install it in Ubuntu/Debian by doing:

$ sudo apt-get install s3cmd

You can also use Python 'virtualenv' and the pip installer if you prefer to use Python applications that way.

s3cmd requires a configuration file. The minimum possible file for Interoute Object Storage looks like this (based on the 'EU' region of Object Storage; you will need another configuration file to use a different Object Storage region):

[default]
host_base = s3-eu.object.vdc.interoute.com
host_bucket = %(bucket)s.s3-eu.object.vdc.interoute.com
access_key = $INTEROUTE_S3_KEY
secret_key = $INTEROUTE_S3_SECRET
use_https = True

(You can download this file at: https://github.com/Interoute/object-storage-api/blob/master/s3cfg.)

You need to insert values for the API keys in place of $INTEROUTE_S3_KEY and $INTEROUTE_S3_SECRET. The default location for configuration information is the file '.s3cfg' in your home directory; otherwise you can specify a configuration file using the '-c' option on the command line.

(For a detailed list of s3cmd input options, see: http://s3tools.org/usage )

Creating a new bucket

With a configuration file installed at ~/.s3cfg, this command will create a new bucket with name 'my-bucket-1':

$ s3cmd mb s3://my-bucket-1
Bucket 's3://my-bucket-1/' created

Bucket names have to be unique so it's likely another user has used this name already and you will get an error message 'BucketAlreadyExists'. If so, try a different name.

Let’s check the bucket is present by typing:

$ s3cmd ls
2016-10-25 16:35 s3://my-bucket-1

Buckets created with s3cmd will use the default storage policy for the object storage region, see “Working with storage policies” below for information about how to set storage policies for buckets.

Setting the CORS configuration for the bucket

Buckets created using the API are not correctly visible in the Object Storage user interface, because the interface requires a CORS (Cross-Origin Resource Sharing) configuration for the bucket. It is easy to add this to a bucket with one command.

For one time only, you need to download this file to your local computer (where s3cmd is running):

https://raw.githubusercontent.com/Interoute/object-storage-api/master/CORS.cfg

Then for every newly-created bucket you apply this configuration to the bucket:

$ s3cmd setcors CORS.cfg s3://my-bucket-1

The file CORS.cfg needs to be in the current directory (or use a filepath string).

Uploading and downloading objects

Assuming there is a file 'picture.gif' in the current directory, this command will upload it into the bucket just created:

$ s3cmd put picture.gif s3://my-bucket-1
upload: 'picture.gif' -> 's3://my-bucket-1/picture.gif'  [1 of 1]
 1189 of 1189   100% in    0s   823.37 kB/s  done

or to make the object name different from the filename:

$ s3cmd put picture.gif s3://my-bucket-1/picture-draft.gif
upload: 'picture.gif' -> 's3://my-bucket-1/picture-draft.gif'  [1 of 1]
 1189 of 1189   100% in    0s   823.37 kB/s  done

Check for the presence of the new object:

$ s3cmd ls s3://my-bucket-1
2016-10-25 16:37    1189   s3://my-bucket-1/picture.gif

(The information there is the date/time last modified, and the object size in bytes.)

To download an object, use a 'get' command:

$ s3cmd get s3://my-bucket-1/picture-draft.gif
download: 's3://my-bucket-1/picture.gif' -> './picture.gif'  [1 of 1]
 1189 of 1189   100% in    0s   1152.14 kB/s  done

Or to change the destination filename:

$ s3cmd get s3://my-bucket-1/picture-draft.gif picture-draft-2.gif

Get information about an object or bucket

Use an 'info' command to get information about a bucket or object:

$ s3cmd info s3://my-bucket-1

For buckets that have been treated by a 'setcors' you will see an output line beginning:

cors:  ...

Object Storage regions

The Object Storage service is offered in several regions, each of which contains two or more data centres, as per the following table.

List of available regions in Interoute Object Storage




Region Data centres API endpoint (S3 'host base')
EU Amsterdam (NL), London (UK), Slough (UK) s3-eu.object.vdc.interoute.com
DE Berlin (DE), Frankfurt (DE) s3-de.object.vdc.interoute.com
CH Geneva (CH), Zurich (CH) s3-ch.object.vdc.interoute.com

For creating and managing buckets and objects, you should use the API endpoint for the region where the specific bucket is (to be) located.

There are a few exceptions to this, where you can use any API endpoint to see information about buckets which are not in the corresponding region. For example, if you do an 'ls' listing of buckets (as in 's3cmd ls'), you will see all of the buckets from all of the regions.

The default region is 'EU' and buckets will be created in that region, if no region is specified.

Creating a bucket in a region using s3cmd

In s3cmd, you add the 'region' option to the create bucket command:

$ s3cmd mb s3://my-bucket-ch-1 --region=ch
Bucket 's3://my-bucket-ch-1/' created

Note that if you try to specify the default region, this does not work!

$ s3cmd -c s3cfg-kentp10 mb s3://my-bucket-eu-1 --region=eu
ERROR: S3 error: 400 (InvalidLocationConstraint): The specified location constraint is not valid.

This is one of the peculiarities of the S3 API standard that you need to be aware of.

Working with storage policies

The storage policy applies to every object in a bucket and is fixed at the point of creation of the bucket. The policy determines how an object is replicated in the data centres of an Object Storage region.

The table below shows the currently available storage policies.

s3cmd cannot be used to set or check storage policies for a bucket. This section shows using Python for making calls to the Object Storage API.

Setting up the Python requests module and the authentication program

Let's assume you are working with a standard Python version 2 install (version 2.7 or 2.6) on Linux/Mac OS. The 'requests' module is a convenient tool for processing API calls. First you need to install it into your Python installation:

$ sudo pip install requests

Next, you need to download a helper program, awsauth.py, to generate the required authentication information for the API call. Download it from this link: https://raw.githubusercontent.com/Interoute/object-storage-api/master/awsauth.py. Store this file in the folder where you will be running Python.

Start Python in interactive mode and generate 'OSauth' to store the authentication information, based on the values of your API keys for Object Storage:

$ python
>>> import requests
>>> from awsauth import S3Auth
>>> OSauth = S3Auth('INTEROUTE_S3_KEY', 'INTEROUTE_S3_SECRET', service_url='s3-eu.object.vdc.interoute.com')

Since you need this basic setup all the time, it is handy to create a file such as api-start.py which you can reuse. With that file installed in your working directory, simply input:

$ python -i api-start.py

Creating a bucket with a PUT request

With the above setup steps done, a plain PUT request will create a bucket with the default storage policy for the Object Storage region being used. For the EU region, it is a '1+1+1' policy with object copies stored in the London, Slough and Amsterdam data centres. Insert the new bucket name 'testingnewbucket1' at the front of the Object Storage API endpoint for the EU region:

>>> requests.put('http://testingnewbucket1.s3-eu.object.vdc.interoute.com', auth=OSauth)

You will get an error if you use a bucket name that is already taken by another user; in that case, modify the bucket name.

To set a storage policy, you include a value for 'x-gmt-policyid' in the request header. The value needs to be a policy ID, as listed in the table below. For example, this API call will create a bucket with the 'London + Slough' storage policy:

>>> requests.put('http://testingnewbucket2.s3-eu.object.vdc.interoute.com', auth=OSauth, headers={'x-gmt-policyid': 'd06b2c336f4e77ce1e01da4819b77476'})

Checking a bucket's storage policy with a GET request

The storage policy for a bucket is not currently visible in the graphical user interface, and S3-compatible client tools may not show it either, because it is not part of the S3 API as defined by AWS. However it is simple to obtain the information using a direct API call.

The API will return information about a bucket by making a GET request for the bucket, and looking at the response header for the key 'x-gmt-policyid':

>>> r=requests.get('http://testingnewbucket2.s3-eu.object.vdc.interoute.com', auth=OSauth)
>>> r.headers
CaseInsensitiveDict({'content-length': '226', ..., 'x-amz-bucket-region': 'eu',...  'x-gmt-policyid': 'd06b2c336f4e77ce1e01da4819b77476'})

You only need to look up the 'policyid' in the table below to identify the storage policy that is being used. Note also the response header 'x-amz-bucket-region' which contains the bucket's region.

And don't forget the CORS

You need to set a CORS configuration for a new bucket. Here is how it can be done using an API call. It looks a little complicated as the call needs to include an MD5 checksum.

First, load the content of the 'CORS.cfg' file and calculate the MD5 checksum:

>>> import hashlib
>>> import base64
>>> with open('CORS.cfg') as fh:
...    CORSconfig = fh.read()
...	
>>> md5hash=hashlib.md5()
>>> md5hash.update(CORSconfig)
>>> base64.b64encode(md5hash.digest())
'O1U2y75bb80G5e03q+KElg=='

API PUT call to upload the CORS configuration:

>>> requests.put('http://testingnewbucket1.s3-eu.object.vdc.interoute.com/?cors', auth=OSauth, data=CORSconfig, headers={'content-md5':'O1U2y75bb80G5e03q+KElg=='})

API GET call to check the configuration:

>>> requests.get('http://testingnewbucket1.s3-eu.object.vdc.interoute.com/?cors', auth=OSauth).text
u'*......>'

Since the 'CORS.cfg' file is always the same, you can pre-load it and have the encoded MD5 checksum ready to use. See the file api-start.py for an example of that.

List of available storage policies for Object Storage

The following table shows the available storage policies and the policy ID to use when creating a bucket with the API.






Storage policy ID Object Storage region Replication type Replication data centres Notes
af465a16f7aac15f6283316b0113057e EU 1+1+1 London, Slough, Amsterdam Default policy for the EU region
d06b2c336f4e77ce1e01da4819b77476 EU 2+1 London (2), Slough (1) Policy for UK-sovereign data storage
433635bcc26110055a871ded8e775df9 EU 2+1 Amsterdam (2), London (1)  
fb329ad8180518ffc9e7280d35262535 EU 2+1 London (2), Amsterdam (1)  
0a4b800afcaaf87ad2aebf8e057eb9ce DE 2+1 Berlin (2), Frankfurt (1) DE-sovereign data storage. Default policy for DE region
322e6a9962547f9be483c4e0a5be5e5a DE 2+1 Berlin (1), Frankfurt (2) DE-sovereign data storage
6d37797515208e3926609d8a7f590220 CH 2+1 Zurich (2), Geneva (1) CH-sovereign data storage. Default policy for CH region
65cfca43dfbc3314ee2fa9f8b879e6b8 CH 2+1 Zurich (1), Geneva (2) CH-sovereign data storage

The default policy for a region is the storage policy that will be used if none is specified in the API call.

Note: the contents of this table are available in the JSON file interoute-object-storage-refdata.json in a structured format for programming use.

Creating and checking buckets in regions using direct API calls

To specify a region with a PUT request for a bucket, a string needs to be passed as data in the API call. For the CH region, the string contains 'ch' (in lower case):

>>> createBucketCH = 'ch'

For API requests in a non-default region, always use the specific API endpoint for that region.

Create a bucket in the CH region, with the default storage policy:

>>> requests.put('http://my-ch-bucket-2.s3-ch.object.vdc.interoute.com', auth=OSauth, data=createBucketCH)

Create a bucket in the CH region, with a non-default storage policy:

>>> requests.put('http://my-ch-bucket-3.s3-ch.object.vdc.interoute.com', auth=OSauth, data=createBucketCH, headers={'x-gmt-policyid': '65cfca43dfbc3314ee2fa9f8b879e6b8'})

Query a bucket's region using a GET request with '?location':

>>> requests.get('http://my-ch-bucket-2.s3-ch.object.vdc.interoute.com/?location', auth=OSauth).text
u'ch'

Note: the region/location response is 'empty' for the default region!

>>> requests.get('http://testingnewbucket1.s3-eu.object.vdc.interoute.com/?location', auth=OSauth).text
u''

Versioning: Storing multiple versions of an object

The default behaviour of Object Storage is to replace an object if another object is uploaded with the same location and name. However if you enable versioning for a bucket then objects are not replaced but are stored in multiple versions, each one being tagged by a unique 'version id'.

Important

Versioning of objects is not currently supported through the Object Storage user interface. In the user interface, you can only see and access the most recent version of an object.

 

In the following examples, a new bucket 'testversioning' has already been created in the EU region. Replace 'testversioning' with the bucket name that you are using.

Enable or suspend versioning for a bucket

Versioning is controlled at the bucket level. It can be enabled or suspended at any time.

Enabling requires a PUT request for a bucket, with an 'Enabled' string as data:

>>> requests.put('http://testversioning.s3-eu.object.vdc.interoute.com/?versioning', auth=OSauth, data='Enabled')

And suspending requires a PUT request of this form:

>>> requests.put('http://testversioning.s3-eu.object.vdc.interoute.com/?versioning', auth=OSauth, data='Suspended')

Use this GET request to check the versioning status of a bucket:

>>> requests.get('http://testversioning.s3-eu.object.vdc.interoute.com/?versioning', auth=OSauth).text

When you GET an object which is 'versioned' the response headers contain an extra item 'x-amz-version-id' whose value is the unique version ID of the object retrieved:

>>> requests.get('http://testversioning.s3-eu.object.vdc.interoute.com/testobject', auth=OSauth).headers
CaseInsensitiveDict({'content-length': '17', ....  'x-amz-version-id': 'fe191147-83cc-ae4f-91c9-246e96052298' ....})

List the versions of an object (GET request)

The '?versions' GET request returns information about all the versions of all of the objects in a bucket. The 'parseString' method is used below to pretty-print the output, which is otherwise hard to read, and the output has been abbreviated for only the version-related content:

>>> import xml.dom.minidom
>>> print xml.dom.minidom.parseString(requests.get('http://testversioning.s3-eu.object.vdc.interoute.com/?versions', auth=OSauth).text).toprettyxml()
<?xml version="1.0" ?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Name>testversioning</Name>
	<Prefix/>
	<KeyMarker/>
	<VersionIdMarker/>
	<Version>
		<Key>testobject</Key>
		<VersionId>fe191147-83cc-ae4f-91c9-246e96052298</VersionId>
		<IsLatest>true</IsLatest>
		<LastModified>2017-02-09T11:11:20.651Z</LastModified>
	</Version>
	<Version>
		<Key>testobject</Key>
		<VersionId>fe19114e-af9d-667f-8c7c-246e96053900</VersionId>
		<IsLatest>false</IsLatest>
		<LastModified>2017-02-09T10:20:00.664Z</LastModified>
	</Version>
	<Version>
		<Key>testobject</Key>
		<VersionId>fe19114e-dc2a-bc1f-8c7c-246e96053900</VersionId>
		<IsLatest>false</IsLatest>
		<LastModified>2017-02-09T10:18:45.918Z</LastModified>
	</Version>
</ListVersionsResult>

In this example, the object 'testobject' is being stored in three versions.

Use the parameter 'prefix=STRING' to filter results to match objects whose names begin with STRING:

>>> print xml.dom.minidom.parseString(requests.get('http://testversioning.s3-eu.object.vdc.interoute.com/?versions&prefix=STRING', auth=OSauth).text).toprettyxml()

GET a specific version of an object

Use the 'versionId' parameter to GET a specific version of an object:

>>> requests.get('http://testversioning.s3-eu.object.vdc.interoute.com/testobject?versionId=fe19114e-dc2a-bc1f-8c7c-246e96053900', auth=OSauth).text
u'this is version 1'

Reading and writing object metadata (key-value pairs)

Objects (and folders) can have user-defined metadata attached to them, in the form of key-value pairs.

Object metadata is returned from an API call in the HTTP headers. The metadata headers start with the string 'x-amz-meta-'.

For example, for an object 'picture.gif' a key-value pair has already been attached to it. This has key equals 'content-type' and value equals 'image/gif'.

The 'info' command in s3cmd will read out the metadata for an object (other information has been deleted in the output here):

$ s3cmd info s3://my-bucket-1/picture.gif
s3://my-bucket-1/picture.gif (object):
  File size: 1189
  Last mod: Tue, 25 Oct 2016 16:27:18 GMT
  ......
  x-amz-meta-content-type: image/gif

To write metadata to an object through the API, you upload a header using the 'x-amz-meta' format. For example, to add the key-value pair: key = 'keyword', value = 'aviation' with s3cmd:

$ s3cmd modify --add-header x-amz-meta-keyword:aviation s3://my-bucket-1/picture.gif

To change the value for an existing key, do 'add-header' again with the new value.

To delete a key-value pair, use 'remove-header':

$ s3cmd modify --remove-header x-amz-meta-keyword s3://my-bucket-1/picture.gif

Note that you don't include the value in the remove command.