Commit 65ef0d34 by Jeremy Wadsack Committed by Tute Costa

Add fog_options to configuration to be passed to fog #create

We found that uploading large files to S3 would result in a socket error
("connection reset by peer") occasionally and lately much more
consistently. In researching this I saw that many people got this error
when uploading too large of a file without multipart chunking. I would
have assumed fog did this automatically but the default chunk size may
be too high. In order to address this I wanted to drop the chunk size to
100MB.

Rather than hard-code this I opted to expose a `fog_option`
configuration option that lets me pass any additional options I want to
the fog's `#create` call. This is similar to the `fog_attributes` option
implemented in CarrierWave which [addresses the same
problem](http://stackoverflow.com/a/11867978/201911).

We've been running this now for a week in production and it seems to
resolve the issue.

https://github.com/thoughtbot/paperclip/pull/2135
parent e60f0002
master: master:
* Improvement: Add `fog_options` configuration to send options to fog when
storing files.
* Drops support to end-of-life'd ruby 2.0. * Drops support to end-of-life'd ruby 2.0.
* Improvement: Paperclip now supports aws-sdk v2 * Improvement: Paperclip now supports aws-sdk v2
@betesh, @davetchen, @betesh, @davetchen,
......
...@@ -33,6 +33,10 @@ module Paperclip ...@@ -33,6 +33,10 @@ module Paperclip
# that is the alias to the S3 domain of your bucket, e.g. # that is the alias to the S3 domain of your bucket, e.g.
# 'http://images.example.com'. This can also be used in # 'http://images.example.com'. This can also be used in
# conjunction with Cloudfront (http://aws.amazon.com/cloudfront) # conjunction with Cloudfront (http://aws.amazon.com/cloudfront)
# * +fog_options+: (optional) A hash of options that are passed
# to fog when the file is created. For example, you could set
# the multipart-chunk size to 100MB with a hash:
# { :multipart_chunk_size => 104857600 }
module Fog module Fog
def self.extended base def self.extended base
...@@ -98,12 +102,14 @@ module Paperclip ...@@ -98,12 +102,14 @@ module Paperclip
log("saving #{path(style)}") log("saving #{path(style)}")
retried = false retried = false
begin begin
directory.files.create(fog_file.merge( attributes = fog_file.merge(
:body => file, :body => file,
:key => path(style), :key => path(style),
:public => fog_public(style), :public => fog_public(style),
:content_type => file.content_type :content_type => file.content_type
)) )
attributes.merge!(@options[:fog_options]) if @options[:fog_options]
directory.files.create(attributes)
rescue Excon::Errors::NotFound rescue Excon::Errors::NotFound
raise if retried raise if retried
retried = true retried = true
......
...@@ -484,6 +484,25 @@ describe Paperclip::Storage::Fog do ...@@ -484,6 +484,25 @@ describe Paperclip::Storage::Fog do
assert_equal @dummy.avatar.fog_credentials, @dynamic_fog_credentials assert_equal @dummy.avatar.fog_credentials, @dynamic_fog_credentials
end end
end end
context "with custom fog_options" do
before do
rebuild_model(
@options.merge(fog_options: { multipart_chunk_size: 104857600 }),
)
@dummy = Dummy.new
@dummy.avatar = @file
end
it "applies the options to the fog #create call" do
files = stub
@dummy.avatar.stubs(:directory).returns stub(files: files)
files.expects(:create).with(
has_entries(multipart_chunk_size: 104857600),
)
@dummy.save
end
end
end end
end end
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment