.. Copyright (c) 2008-2016 OpenShot Studios, LLC
(http://www.openshotstudios.com). This file is part of
OpenShot Video Editor (http://www.openshot.org), an open-source project
dedicated to delivering high quality video editing and animation solutions
to the world.
.. OpenShot Video Editor is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
.. OpenShot Video Editor is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
.. You should have received a copy of the GNU General Public License
along with OpenShot Library. If not, see .
.. _getting_started_aws_ref:
Getting Started (on AWS)
========================
OpenShot Cloud API is easy to install and configure. *Launch* your own *private instance* using
`AWS Marketplace `_, or by following this link:
https://aws.amazon.com/marketplace/pp/B074H87FSJ/. The following video walks you though the configuration process.
| See also :ref:`getting_started_azure_ref`.
| See also :ref:`api_endpoints_main_ref`.
.. youtube:: DVGQZNDGwGo
This tutorial video covers the basics of how to configure an OpenShot Cloud API instance, and
walks you through the process of creating your first video project.
Launch an Instance
------------------
After you launch an instance of the OpenShot Cloud API, it is initially
configured to run as both a web server and task server (**server**), and as a video processor (**worker**). Once you have
started your instance, you can access the web server without changing anything.
Access your new instance in a web browser. Use its public DNS or IP address.
http://YourInstanceIP/
.. image:: _static/root-ui.png
Architecture
------------
This diagram illustrates the basic architecture of OpenShot Cloud API. A **server** listens for video editing tasks, and
then sends those to an available **worker**. A worker processes each job (one at a time), rendering a final video. When a worker
is done, it can optionally POST to a webhook URL, upload the video to S3, or the video file can be downloaded as needed. All servers
and workers share files using HTTP, for accessing source assets (i.e. logos, videos, music files) and for sharing the final
rendered output video.
The **server** should be configured with a large enough EBS volume to handle all files you intend to store. And if you
are done with a project after exporting/rendering, it is best to delete it (and the source files will be removed automatically).
.. image:: _static/diagram.png
REST: Representational State Transfer
-------------------------------------
REST is a design pattern for web services modeled around HTTP requests. Each operation uses its own HTTP method:
======= ==================
Method Description
======= ==================
GET Get data from the API (i.e. request a project, clip, or export)
POST Create data on the API (i.e. create a project, clip, or export)
PUT Update existing data from the API (i.e. update a project, clip, or export)
PATCH Update partial existing data from the API (i.e. update **only** a Clip's position)
DELETE Delete existing data from the API (i.e. delete a project, clip, or export)
======= ==================
Before You Begin
----------------
OpenShot Cloud API uses AWS IAM user credentials to access S3 and SQS. Before you begin, you will need to create a new
IAM user on AWS, and give it the following permissions: **AmazonS3FullAccess & AmazonSQSFullAccess**.
The following details will be required:
- **IAM User**
- Access Key
- Secret Access Key
- Permissions: AmazonS3FullAccess & AmazonSQSFullAccess
- **SQS Queue**
- Create a queue named **OpenShotAPIExportQueue**
- Standard queue with default settings
OpenShot Cloud API will not work without this configuration, so please verify you have things configured correctly before
moving forward.
Server Settings
---------------
To configure the settings of your new instance, SSH into the instance using your private key and the **ubuntu** user:
.. code-block:: bash
ssh -i private-key.pem ubuntu@xxx.xxx.xxx.xxx
Run the following command to configure your **server** for the first time:
.. code-block:: bash
config-openshot-cloud
When setting up your first instance, it is recommended to configure it as both a **server**
and **worker**, by choosing the first option: ``(B)oth``.
.. code-block:: bash
Choose a ROLE for this instance.
(B)oth - Runs both the HTTP API, DB, and video processing
(S)erver - Runs only the HTTP API and the DB (no video processing)
(W)orker - Runs only the video processing tasks
When configuring your server, it is very important to provide the following AWS settings. If these
settings are invalid, OpenShot Cloud API will fail to process video exports, and S3 features
will throw errors.
.. code-block:: bash
# This information is available for your AWS IAM User (mentioned above)
AWS Access Key ID?
AWS Secret Access Key?
AWS SQS Queue? (OpenShotAPIExportQueue)
AWS Region (us-east-1, us-west-2, eu-west-1, etc...)?
Worker Settings
---------------
If you are only running a single instance (server and worker), you can skip this section. If you are launching additional
worker instances, this section covers how to connect them back to the OpenShot Cloud API server.
To configure the settings of your new **worker** instance, SSH into the instance using your private key and the **ubuntu** user:
.. code-block:: bash
ssh -i private-key.pem ubuntu@xxx.xxx.xxx.xxx
Run the following command to configure your **worker** for the first time, and choose the second
option: (W)orker.
.. code-block:: bash
config-openshot-cloud
For each **worker** instance you launch, you will be prompted to provide the **Cloud API URL** of the **server**
(which can be an internal IP address) and to provide AWS Settings (for SQS and S3 - same as configuring the server above).
This allows the **worker** to watch the SQS queue for new video export tasks/jobs, download and upload files from S3,
and use the API URL to update the export record with progress, status, and the final rendered video file.
NOTE: The **worker** must be able to communicate over port 80 with the API **server**.
Now that you have configured your worker instance, it should be listening for new tasks from the server, and updating the
**Export** record on the server when it completes tasks. Each **Export** record contains the hostname
of the worker that processed it. So, create a test Project and then create an Export, and you
will see the hostname of the **worker** added to the Export record.
Verify AWS Configuration
------------------------
When **config-openshot-cloud** has completed, it will try and verify your AWS configuration, and
will let you know if anything went wrong. If you see a failure, please double check your AWS credentials
and SQS queue name and run **config-openshot-cloud** again. You can also access http://YourInstanceIP/aws/validate/
at any time to verify your AWS configuration.
Admin Interface
---------------
A full admin interface is available at http://YourInstanceIP/cloud-admin/, which allows you to create additional user
logins, manage your video projects, and search your data.
Python Example Script
---------------------
Here is a simple example client script, using Python 3. It connects, creates a new project, uploads a file,
creates a clip, and then exports a new video. It should give you a good sense of how
things work in general, regardless of programming language. Learn more about the :ref:`api_endpoints_main_ref`.
.. code-block:: python
import os
import time
from requests import get, post
from requests.auth import HTTPBasicAuth
PATH = os.path.dirname(os.path.realpath(__file__))
CLOUD_URL = 'http://cloud.openshot.org'
CLOUD_AUTH = HTTPBasicAuth('demo-cloud', 'demo-password')
##########################################################
# Get list of projects
end_point = '/projects/'
r = get(CLOUD_URL + end_point, auth=CLOUD_AUTH)
print(r.json())
##########################################################
# Create new project
end_point = '/projects/'
project_data = {
"name": "API Project",
"width": 1920,
"height": 1080,
"fps_num": 30,
"fps_den": 1,
"sample_rate": 44100,
"channels": 2,
"channel_layout": 3,
"json": "{}",
}
r = post(CLOUD_URL + end_point, data=project_data, auth=CLOUD_AUTH)
print(r.json())
project_id = r.json().get("id")
project_url = r.json().get("url")
##########################################################
# Upload file to project
end_point = '/projects/%s/files/' % project_id
source_path = os.path.join(PATH, "example-video.mp4")
source_name = os.path.split(source_path)[1]
file_data = {
"media": None,
"project": project_url,
"json": "{}"
}
r = post(CLOUD_URL + end_point, data=file_data, files={"media": (source_name, open(source_path, "rb"))}, auth=CLOUD_AUTH)
file_url = r.json().get("url")
print(r.json())
##########################################################
# Create a clip for the previously uploaded file
end_point = '/projects/%s/clips/' % project_id
clip_data = {
"file": file_url,
"position": 0.0,
"start": 0.0,
"end": 30.0,
"layer": 1,
"project": project_url,
"json": "{}"
}
r = post(CLOUD_URL + end_point, data=clip_data, auth=CLOUD_AUTH)
print(r.json())
##########################################################
# Create export for final rendered video
end_point = '/projects/%s/exports/' % project_id
export_data = {
"video_format": "mp4",
"video_codec": "libx264",
"video_bitrate": 8000000,
"audio_codec": "ac3",
"audio_bitrate": 1920000,
"start_frame": 1,
"end_frame": None,
"project": project_url,
"json": "{}"
}
r = post(CLOUD_URL + end_point, data=export_data, auth=CLOUD_AUTH)
export_url = r.json().get("url")
print(r.json())
##########################################################
# Wait for Export to finish (give up after around 40 minutes)
export_output_url = None
is_exported = False
countdown = 500
while not is_exported and countdown > 1:
r = get(export_url, auth=CLOUD_AUTH)
print(r.json())
is_exported = float(r.json().get("progress", 0.0)) == 100.0
countdown -= 1
time.sleep(5.0)
# Get final rendered url
r = get(export_url, auth=CLOUD_AUTH)
export_output_url = r.json().get("output")
print(r.json())
print("Export Successfully Completed: %s!" % export_output_url)
Node.js Example Script
---------------------
Here is a simple example client script, using JavaScript and Node.js. It connects, creates a new project, uploads some files,
creates some clips, exports a video, and then downloads the new video file. It should give you a good sense of how things
work using asynchronous JavaScript. Learn more about the :ref:`api_endpoints_main_ref`.
.. code-block:: javascript
var request = require('request-promise');
var fs = require('fs');
var Promise = require("promise");
var promisePoller = require('promise-poller').default;
var projectId = '';
var projectUrl = '';
var exportId = '';
const projectData = {
'json': '{}',
'name': 'My Project Name'
};
const protocol = 'http';
const server = 'cloud.openshot.org';
const auth = { 'user': 'demo-cloud',
'pass': 'demo-password'};
// Create a new project
post('/projects/', projectData)
.then(function(response){
projectUrl = JSON.parse(response).url;
projectId = JSON.parse(response).id;
console.log('Successfully created project: ' + projectUrl);
// Add a couple clips (to be asynchronously processed)
const promiseArray = [];
promiseArray.push(
createClip({
"path": "media-files/MyVideo.mp4",
"position": 0.0,
"end": 10.0
}));
promiseArray.push(
createClip({
"path": "media-files/MyAudio.mp4",
"position": 0.0,
"end": 10.0
}));
promiseArray.push(
createClip({
"path": "media-files/Watermark.JPG",
"position": 0.0,
"end": 10.0,
"layer": 2
}));
// Wait until all files and clips are uploaded
Promise.all(promiseArray).then(function(responseArray){
// Export as a new video
exportData = {
"export_type": "video",
"video_format": "mp4",
"video_codec": "libx264",
"video_bitrate": 8000000,
"audio_codec": "ac3",
"audio_bitrate": 1920000,
"project": projectUrl,
"json": '{}'
};
post('/exports/', exportData)
.then(function(response){
// Export has been created and will begin processing soon
var exportUrl = JSON.parse(response).url;
exportId = JSON.parse(response).id;
console.log('Successfully created export: ' + exportUrl);
// Poll until the export has finished
var poller = promisePoller({
taskFn: isExportCompleted,
interval: 1000,
retries: 60*60*1000,
timeout: 2000
}).then(function(exportOutputUrl) {
// New exported video is ready for download now
console.log('Download ' + exportOutputUrl);
request(exportOutputUrl).pipe(fs.createWriteStream('Output-' + projectId + '.mp4'));
});
});
});
});
function isExportCompleted() {
return new Promise(function (resolve, error) {
get('/exports/' + exportId + '/', {})
.then(function(response) {
var exportStatus = JSON.parse(response).status;
var exportOutputUrl = JSON.parse(response).output;
if (exportStatus == 'completed') {
console.log('Export completed: ' + JSON.stringify(response));
resolve(exportOutputUrl);
}
});
});
}
function createClip(clip) {
// Create new File object (and upload file from filesystem)
var fileData = {
'json': '{}',
'project': projectUrl,
'media': fs.createReadStream(clip.path)
};
return new Promise(function (resolve) {
post('/files/', fileData)
.then(function(response) {
// File uploaded and object created
var fileUrl = JSON.parse(response).url;
console.log('Successfully created file: ' + fileUrl);
// Now we need to add a clip which references this new File object
var clipData = {
'file': fileUrl,
'json': '{}',
'position': clip.position || 0.0,
'start': clip.start || 0.0,
'end': clip.end || JSON.parse(response).json.duration,
'layer': clip.layer || 0,
'project': projectUrl
};
return post('/clips/', clipData).then(function(response){
var clipUrl = JSON.parse(response).url;
console.log('Successfully created clip: ' + clipUrl);
resolve();
});
});
});
}
function post(endpoint, data) {
// Prepare request object and POST data to OpenShot API
const r = request.post(protocol + '://' + auth.user + ':' + auth.pass + '@' + server + endpoint, function (err, response, body) {
if (err) {
return err;
} else {
console.log(response.statusCode + ': ' + body);
return body;
}
});
// Append form data to request form-data
const form_data = r.form();
for ( var key in data ) {
form_data.append(key, data[key]);
}
return r;
}
function get(endpoint, data) {
// Prepare request object and GET data to OpenShot API
return request.get(protocol + '://' + auth.user + ':' + auth.pass + '@' + server + endpoint, function (err, response, body) {
if (err) {
return err;
} else {
console.log(response.statusCode + ': ' + body);
return body;
}
});
}
C# Example Script
---------------------
Here is a simple example client script, using C# (.NET version: 5.0.103). It connects, creates a new project, uploads a file,
creates a clips, exports a video, and then prints the new video file link. It should give you a good sense of how things
work in C# (This code has been made synchronous on purpose). Learn more about the :ref:`api_endpoints_main_ref`.
This also uses 2 NuGet packages: Newtonsoft, MimeMapping
.. code-block:: csharp
using System;
using System.Collections.Generic;
using System.Threading;
using System.Net.Http;
using System.Net.Http.Headers;
using System.IO;
using Newtonsoft.Json.Linq;
using MimeMapping;
namespace OpenShot
{
class Program
{
static void Main(string[] args)
{
String MakeGetRequest(string url, HttpClient client)
{
Console.WriteLine("GET: " + url);
var response = client.GetAsync(url).GetAwaiter().GetResult();
Console.WriteLine("Response StatusCode: " + (int)response.StatusCode);
if (response.IsSuccessStatusCode){
return response.Content.ReadAsStringAsync().GetAwaiter().GetResult();;
}else{
Console.WriteLine("Request Failed with message:" +
response.Content.ReadAsStringAsync().GetAwaiter().GetResult());
}
return "";
}
String MakePostRequest(string url, Dictionary formData, HttpClient client)
{
var requestContent = new FormUrlEncodedContent(formData);
Console.WriteLine("POST: " + url);
var response = client.PostAsync(url, requestContent).GetAwaiter().GetResult();
Console.WriteLine("Response StatusCode: " + (int)response.StatusCode);
if (response.IsSuccessStatusCode){
return response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
}else{
Console.WriteLine("Request Failed with message:" +
response.Content.ReadAsStringAsync().GetAwaiter().GetResult());
}
return "";
}
String MakePostRequestWithFile(string url, Dictionary formData, HttpClient client, FileInfo fileInfo)
{
var requestContent = new MultipartFormDataContent();
var fileContent = new StreamContent(fileInfo.OpenRead());
fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
Name = "\"media\"",
FileName = "\"" + fileInfo.Name + "\""
};
fileContent.Headers.ContentType = MediaTypeHeaderValue.Parse(MimeMapping.MimeUtility.GetMimeMapping(fileInfo.Name));
foreach (KeyValuePair kvp in formData)
{
requestContent.Add(new StringContent(kvp.Value), kvp.Key);
}
requestContent.Add(fileContent);
Console.WriteLine("POST: " + url);
var response = client.PostAsync(url, requestContent).GetAwaiter().GetResult();
Console.WriteLine("Response StatusCode: " + (int)response.StatusCode);
if (response.IsSuccessStatusCode){
return response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
}else{
Console.WriteLine("Request Failed with message:" +
response.Content.ReadAsStringAsync().GetAwaiter().GetResult());
}
return "";
}
HttpClient client = new HttpClient();
// Connection info
string serverUrl = "http://cloud.openshot.org/";
// Basic Auth
String username = "demo-cloud";
String password = "demo-password";
String encoded = System.Convert.ToBase64String(
System.Text.Encoding.GetEncoding("ISO-8859-1").GetBytes(
username + ":" + password));
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", encoded);
////////////////////////////////////////////////////////////////////////////////////////
// Get list of projects
Console.WriteLine("\n Get list of projects \n");
String responseGetRequest = MakeGetRequest(serverUrl + "projects/", client);
Console.WriteLine(responseGetRequest);
////////////////////////////////////////////////////////////////////////////////////////
// Create new project
Console.WriteLine("\n Create new project \n");
Dictionary payload = new Dictionary
{
{ "name", "API Project" },
{ "width", "1920" },
{"height", "1080"},
{"fps_num", "30"},
{"fps_den", "1"},
{"sample_rate", "44100"},
{"channels", "2"},
{"channel_layout", "3"},
{"json", "{}" }
};
String responsePostRequest = MakePostRequest(serverUrl + "projects/", payload, client);
Console.WriteLine(responsePostRequest);
dynamic jsonObj = JObject.Parse(responsePostRequest);
Int32 projectId = jsonObj.id;
String projectUrl = jsonObj.url;
Console.WriteLine("Project ID: " + projectId);
Console.WriteLine("Project URL: "+ projectUrl);
////////////////////////////////////////////////////////////////////////////////////////
// Upload file to project
Console.WriteLine("\n Upload file to project \n");
payload = new Dictionary
{
{"media", ""},
{"project", projectUrl},
{"json", "{}" }
};
String filePath = "D:\\test-file.mp4";
responsePostRequest = MakePostRequestWithFile(
projectUrl + "files/",
payload,
client,
new FileInfo(filePath)
);
Console.WriteLine(responsePostRequest);
jsonObj = JObject.Parse(responsePostRequest);
String fileUrl = jsonObj.url;
Console.WriteLine("File URL:" + fileUrl);
////////////////////////////////////////////////////////////////////////////////////////
// Create a clip for the previously uploaded file
Console.WriteLine("\n Create a clip for the previously uploaded file \n");
payload = new Dictionary
{
{"file", fileUrl},
{"position", "0.0"},
{"start", "0.0"},
{"end", "2.0"},
{"layer", "1"},
{"project", projectUrl},
{"json", "{}"}
};
responsePostRequest = MakePostRequest(projectUrl + "clips/", payload, client);
Console.WriteLine(responsePostRequest);
////////////////////////////////////////////////////////////////////////////////////////
// Create export for final rendered video
Console.WriteLine("\n Create export for final rendered video \n");
payload = new Dictionary
{
{"video_format", "mp4"},
{"video_codec", "libx264"},
{"video_bitrate", "8000000"},
{"audio_codec", "ac3"},
{"audio_bitrate", "1920000"},
{"start_frame", "1"},
{"end_frame", ""},
{"project", projectUrl},
{"json", "{}"}
};
responsePostRequest = MakePostRequest(projectUrl + "exports/", payload, client);
Console.WriteLine(responsePostRequest);
jsonObj = JObject.Parse(responsePostRequest);
String exportUrl = jsonObj.url;
Boolean isExported = false;
Int16 countdown = 500;
while (!isExported && countdown > 1){
String responseExportGetRequest = MakeGetRequest(exportUrl, client);
jsonObj = JObject.Parse(responseExportGetRequest);
String progress = jsonObj.progress;
if (progress != null && progress !=""){
float progressFloat = float.Parse(progress);
if (progressFloat == 100.0){
isExported = true;
}
}
countdown -= 1;
Thread.Sleep(5000);
}
responseGetRequest = MakeGetRequest(exportUrl, client);
jsonObj = JObject.Parse(responseGetRequest);
Console.WriteLine(responseGetRequest);
Console.WriteLine("Export Successfully Completed: " + jsonObj.output);
}
}
}