Commit 4debddb8 by Christian Becker Committed by Tute Costa

Instead of copying files in adapters, create hard links where possible (#2290)

Rebased #2120 to master.
Paperclip duplicates the original files quite a lot as part of its validation process. (#1642, #1326).
When uploading large files (several hundred megabytes to gigabyte range), this becomes a problem: The web server will be busy creating 3 - 4 duplicates on disk, while the app (and potentially the user) are waiting for the upload operation to complete.
This pull request introduces hard links instead of ```FileUtil.cp``` where possible to keep the logic as-is but save time and disk space.
parent b55b0f2e
...@@ -52,8 +52,18 @@ module Paperclip ...@@ -52,8 +52,18 @@ module Paperclip
end end
def copy_to_tempfile(src) def copy_to_tempfile(src)
FileUtils.cp(src.path, destination.path) link_or_copy_file(src.path, destination.path)
destination destination
end end
def link_or_copy_file(src, dest)
Paperclip.log("Trying to link #{src} to #{dest}")
FileUtils.ln(src, dest, force: true) # overwrite existing
@destination.close
@destination.open.binmode
rescue Errno::EXDEV, Errno::EPERM, Errno::ENOENT => e
Paperclip.log("Link failed with #{e.message}; copying link #{src} to #{dest}")
FileUtils.cp(src, dest)
end
end end
end end
...@@ -23,7 +23,7 @@ module Paperclip ...@@ -23,7 +23,7 @@ module Paperclip
def copy_to_tempfile(source) def copy_to_tempfile(source)
if source.staged? if source.staged?
FileUtils.cp(source.staged_path(@style), destination.path) link_or_copy_file(source.staged_path(@style), destination.path)
else else
source.copy_to_local_file(@style, destination.path) source.copy_to_local_file(@style, destination.path)
end end
......
...@@ -37,7 +37,7 @@ module Paperclip ...@@ -37,7 +37,7 @@ module Paperclip
@queued_for_write.each do |style_name, file| @queued_for_write.each do |style_name, file|
FileUtils.mkdir_p(File.dirname(path(style_name))) FileUtils.mkdir_p(File.dirname(path(style_name)))
begin begin
FileUtils.mv(file.path, path(style_name)) move_file(file.path, path(style_name))
rescue SystemCallError rescue SystemCallError
File.open(path(style_name), "wb") do |new_file| File.open(path(style_name), "wb") do |new_file|
while chunk = file.read(16 * 1024) while chunk = file.read(16 * 1024)
...@@ -84,6 +84,17 @@ module Paperclip ...@@ -84,6 +84,17 @@ module Paperclip
def copy_to_local_file(style, local_dest_path) def copy_to_local_file(style, local_dest_path)
FileUtils.cp(path(style), local_dest_path) FileUtils.cp(path(style), local_dest_path)
end end
private
def move_file(src, dest)
# Support hardlinked files
if File.identical?(src, dest)
File.unlink(src)
else
FileUtils.mv(src, dest)
end
end
end end
end end
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment