Make Zencoder optional

[baycom]

There are so many open source tools like ffmpeg, why do we have to rely on a paid external service? Feature request: Make Zencoder optional in favor of a built in transcoding engine.

[tf]

It would indeed be nice to decouple Pageflow from the concrete encoding solution used and we had some internal discussions how such an abstraction could look like. Most likely this would involve developing (or finding) a gem that provides a facade in front of different encoding services (zencoder/transloadit/encoding.com/local ffmpeg). Parts of that could probably be extracted from Pageflow itself.

Still, pursuing this road is currently not a priority for our own usage of Pageflow since we are quite happy with having Zencoder deal with all the gory details of file encoding. But if anybody would like to step up and push things forward, we'd be glad to give pointers.

[juansev]

tf, as i told you by Twitter, we are interested in using Pageflow without Zencoder. I know you plan to generate an abstract layer to generalize it but my idea is to just fork and "replace" all the essential functionaly that requires Zencoder with code for using ffmpeg or other solution. If the application is good enough structured maybe i can implement that kind of abstract layer to make independent the video_encoder. What is the basic functionality (first, to be able to display videos from a video server, second, and i don't really know if that is going to be a requirement, to be able to upload the videos).

I have experience (and we are using):

• Wowza (dedicated server)
• FFmpeg and FFServer processes
• ScaleEngine CDN

[tf]

Forking might not even be necessary. Video encoding is handled by two Resque jobs called SubmitFileToZencoderJob and PollZencoderJob. You could monkey patch those two jobs in your Pageflow application and override the perform_with_result methods to invoke ffmpeg or another custom encoding stack.

By default these jobs are intended to control an asynchronous encoding process. But for your experiment the submit job could just as well block until all encoding is done, turning the poll job into a no-op which simply reports the encoding to be finished.

Both jobs are passed either a VideoFile or AudioFile object. Using the Paperclip attachments on those objects, you could obtain the source files for encoding. Both of these models respond to a bunch of methods which tell you the paths of generated files which Pageflow expects: i.e.

video_file.webm_high.path # => "pageflow/video_files/000/000/001/high.webm"`

Letting Pageflow.config.zencoder_options[:s3_host_alias] point to the correct destination where your encoding process places output files, Pageflow could generate correct URLs to your files.

Finally you could even opt out of S3 storage by changing Pageflow.config.paperclip_s3_default_options as suggested in #54.

Still, please keep in mind, that patching Pageflow like that means using private interfaces which are not guaranteed to stay around in future versions. All in all, I'd prefer this process to forking though, which might quickly leave you with an obsolete snapshot of the codebase.

[juansev]

Thank you very much for the information. I have already setup correctly the s3 connection (for uploading photos) and i am going to use the information you provided. I saw that you were using an interface (ZencoderApi). Could i implement another one? I think i am currently using Pageflow 0.3.0 with rails 4.0.4 , can i update to the latest version without any problems or is there any kind of incompatibility (i am using Ubuntu 12.04, ruby 1.9.3p0 ).

This is my gemfile:

source 'https://rubygems.org'


# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '4.0.4'
# Use mysql as the database for Active Record
gem 'mysql2'
# Use SCSS for stylesheets
gem 'sass-rails', '~> 4.0.3'
# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'
# Use CoffeeScript for .js.coffee assets and views
gem 'coffee-rails', '~> 4.0.0'
# See https://github.com/sstephenson/execjs#readme for more supported runtimes
gem 'therubyracer',  platforms: :ruby

# Use jquery as the JavaScript library
gem 'jquery-rails'
# Turbolinks makes following links in your web application faster. Read more: https://github.com/rails/turbolinks
gem 'turbolinks'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '1.5.3'
gem 'activesupport','4.0.4'
# bundle exec rake doc:rails generates the API under doc/api.
gem 'sdoc', '~> 0.4.0',          group: :doc
gem 'sprockets-rails', '2.0.1'
# Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
gem 'spring',        group: :development
gem 'activeadmin', :git => 'https://github.com/codevise/active_admin.git', :branch => 'rails4'
gem 'ransack'
gem 'inherited_resources', '1.4.1'
gem 'formtastic', '2.3.0.rc2'
gem 'devise'
# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'

# Use unicorn as the app server
# gem 'unicorn'

# Use Capistrano for deployment
# gem 'capistrano-rails', group: :development

# Use debugger
# gem 'debugger', group: [:development, :test]
gem 'pageflow'

[tf]

You could indeed provide an alternative implementation of the ZencoderApi interface. Still the data passed to the methods there is a lot more denormalized and tailored to Zencoder conventions. You would for example have to parse S3 bucket names and paths from URLs manually. Moreover, the configuration space provided by Zencoder (and used by Pageflow) is rather big. Have a look at [ZencoderVideoOutputDefinition](https://github.com/codevise/pageflow/blob/master/lib/pageflow/zencoder_video_output_definition.rb) which generates the configuration hash used by ZencoderApi. Also, I am not sure how easy it is to realize the asynchronous interface of ZencoderApi, which is expected to return a job_id` which can be used to poll for encoding progress.

Still, in theory, I agree that replacing Zencoder with a custom component that supports the same interface would be ideal for Pageflow. I'm not sure though if that is a realistic goal to shoot for.

Updating to master should not be a problem if you follow the steps in the changelog.

[cskujawa]

Howdy @baycom , I did a proof of concept implementing Pageflow with Minio in place of S3 and a custom FFMPEG transcoder container in place of Zencoder. I tried to make it easily to deploy via docker compose. Also added some setup/management scripts to get folks goin if they try it. I'm not a full time developer, and I likely won't be making updates to this project, but wanted to share as an example of how Pageflow could be entirely self-hosted.
https://github.com/cskujawa/self-hosted-pageflow