Commit 4debddb8 by Christian Becker Committed by Tute Costa

Instead of copying files in adapters, create hard links where possible (#2290)

Rebased #2120 to master.
Paperclip duplicates the original files quite a lot as part of its validation process. (#1642, #1326).
When uploading large files (several hundred megabytes to gigabyte range), this becomes a problem: The web server will be busy creating 3 - 4 duplicates on disk, while the app (and potentially the user) are waiting for the upload operation to complete.
This pull request introduces hard links instead of ```FileUtil.cp``` where possible to keep the logic as-is but save time and disk space.
parent b55b0f2e
......@@ -52,8 +52,18 @@ module Paperclip
end
def copy_to_tempfile(src)
FileUtils.cp(src.path, destination.path)
link_or_copy_file(src.path, destination.path)
destination
end
def link_or_copy_file(src, dest)
Paperclip.log("Trying to link #{src} to #{dest}")
FileUtils.ln(src, dest, force: true) # overwrite existing
@destination.close
@destination.open.binmode
rescue Errno::EXDEV, Errno::EPERM, Errno::ENOENT => e
Paperclip.log("Link failed with #{e.message}; copying link #{src} to #{dest}")
FileUtils.cp(src, dest)
end
end
end
......@@ -23,7 +23,7 @@ module Paperclip
def copy_to_tempfile(source)
if source.staged?
FileUtils.cp(source.staged_path(@style), destination.path)
link_or_copy_file(source.staged_path(@style), destination.path)
else
source.copy_to_local_file(@style, destination.path)
end
......
......@@ -37,7 +37,7 @@ module Paperclip
@queued_for_write.each do |style_name, file|
FileUtils.mkdir_p(File.dirname(path(style_name)))
begin
FileUtils.mv(file.path, path(style_name))
move_file(file.path, path(style_name))
rescue SystemCallError
File.open(path(style_name), "wb") do |new_file|
while chunk = file.read(16 * 1024)
......@@ -84,6 +84,17 @@ module Paperclip
def copy_to_local_file(style, local_dest_path)
FileUtils.cp(path(style), local_dest_path)
end
private
def move_file(src, dest)
# Support hardlinked files
if File.identical?(src, dest)
File.unlink(src)
else
FileUtils.mv(src, dest)
end
end
end
end
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment