ruby on rails - How to correctly use Resque workers? -

i have following tasks in rails application:

• download video
• trim video ffmpeg between given duration (eg.: 00:02 - 00:09)
• convert video given format
• move converted video folder

since wanted make happen in background jobs, used 1 resque worker processes queue.

for first job, have created queue @queue = :download_video it's task, , @ end of task going forward next task calling resque.enqueue(convertvideo, name, itemid). in way, have created chain of queues enqueued when 1 task finished.

this wrong, since if first job starts enqueue other jobs (one another), get's blocked 1 worker until first list of queued jobs finished.

how should optimised? tried adding more workers way of enqueueing jobs, results wrong , unpredictable.

another aspect each job saving status in database , need jobs processed in right order.

should each worker single job above , have @ least 4 workers? if double amount 8 workers, improvement?

have considered using sidekiq ?

as said in sidekiq documentation :

resque uses redis storage , processes messages in single-threaded process. redis requirement makes little more difficult set up, compared delayed_job, redis far better queue sql database. being single-threaded means processing 20 jobs in parallel requires 20 processes, can take lot of memory.

sidekiq uses redis storage , processes jobs in multi-threaded process. it's easy set resque more efficient in terms of raw processing speed. worker code need thread-safe.

so should have 2 kind of jobs : download videos , convert videos , download video job should done in parallel (you can limit if want) , each stored in 1 queue (the "in-between queue") before being converted multiple convert jobs in parallel.

i hope helps, link explains quite best practices in sidekiq : https://github.com/mperham/sidekiq/wiki/best-practices