Portable file copying in C++

Background

If any of you ever had an Amiga (and used it for anything but playing games) you will likely remember a commercial file manager called Directory Opus. The 4.xx series was the de facto two paned file commander for the Amiga. Directory Opus - or just DOpus - later evolved into a fully featured replacement for the Amiga's desktop, Workbench, which gave great joy to a lot of people and caused general outrage from a whole lot more.

I remember both fondly, and on my Windows box, I still use DOpus as an explorer replacement. However, for many people DOpus 4.12 was the embodiment of the Amiga Experience. Simple, clean and highly efficient.

Which brings me to the actual topic, I want to discuss. DOpus, in all its incarnations, had (and has) a very nice progress dialog for copying and moving files. The layout goes sorta like this:

• The name of the file currently being copied,
• source and destination,
• a progress bar for that file,
• a summary of the entire copy process (all selected files and dirs) and
• a progress bar for the entire operation.

An interesting thing about the bottom progress bar is that it shows percent of the entire operation measured in bytes. This makes the progress fairly accurate even when multiple files of vastly different sizes are being copied.

My current attempt to copy files portably in C++ comes from a (yes, childish indeed!) desire to re-implement that progress dialog.
Most modern APIs already have a way to copy files, and they're generally quite easy to use. However, they usually take a source file name and a destination directory and return when the file is copied. With all the progress information I want to display, I need to be able to poke into the copy function itself and find out how it's coming along. This suggests that I have to put together a file copy function myself, so that's what I'm trying to do in this post.

Requirements

 A module that copies a file has to know certain things:

• The name of the file to be copied.
• The name of the destination file name (the name itself need not be the same as the source file name).
• If the destination file already exists, should we overwrite or abort?

The first two will likely be arguments to the function. The last may be an argument or we could make the non-existence of the destination file a precondition. For now, I will make it an argument to the function, because it is fairly simple but still puts some logic in place that can be extended later on.

C++ and C already has machinery that can operate on files: C has FILE* handles and C++ has <iostream>s. Using these facilities and standard algorithms it is actually possible to implement the entire copy operation as a one-liner, but then I'm back where I started: I know of no way to get the current state out of an algorithm invocation.

The idea then is to copy the file in batches. The size of each batch does not matter too much, though it should be small enough that progress updates make sense, yet we also do not want to copy everything byte-wise. I chosen (somewhat arbitrarily) to use a chunk size of 32Kb. This will give no updates for files smaller than that, of course, but that's hardly a limitation.

To represent each file name I have chosen to use the Boost.FileSystem library version 3. This library is the basis of the current standardisation efforts, so we may as well get acquainted with it anyway.

Note that Boost.FileSystem already has a function, copy(from, to), that does exactly what it says on the tin. Again, though, I really want those status updates and I can't get them from that function!

With this little initial though about the function, we're really not quite ready to implement anything! I never had quite that much patience, though, so let's get to it. We're only talking about copying files! How hard can it be...?

Copying files

Sooner or later the entire logic for this operation will be rolled into a class; that class will then maintain the source files, sizes and destination directory. For now though we will concentrate on the core function: From file (path and file name), to file (path and file name) and whether to overwrite if it exists. That suggests the following function signature:

 bool copyFile(boost::filesystem::path const& from,
               boost::filesystem::path const& to, bool failIfExists);

The function will return true if the file copy succeeded, false otherwise. from must exist and to must be a complete path with a file name.

Without further ado here's an implementation that matches what I've discussed so far. We'll walk through it afterwards.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

bool copyFile(boost::filesystem::path const& from, boost::filesystem::path const& to, bool failIfExists) { std::size_t const chunkSize = 32768; boost::filesystem::ifstream infile{from}; if(!infile.good()) return false; if(failIfExists && exists(to)) return false; try { boost::filesystem::ofstream outfile{to, std::ios_base::out | std::ios_base::trunc | std::ios_base::binary}; std::unique_ptr<char[]> buffer{new char[chunkSize]}; std::streamsize n{chunkSize}; while(n == chunkSize) { n = infile.rdbuf()->sgetn(&buffer[0], chunkSize); outfile.rdbuf()->sputn(&buffer[0], n); } } catch(boost::filesystem::filesystem_error& e) { infile.close(); return false; } return true; }

Line 4 simply sets the chunkSize to 32Kb. It is used in line 17 when we allocate a buffer to use.

On line 6 we open the source file for reading as a standard stream. Boost.FileSystem includes stream classes that are identical to the standard streams, except for taking path arguments instead of string-based ones. On line 7 we check that the source file opened correctly.

Line 9 checks if the destination file exists. exists() is a function from namespace boost::filesystem and will be found through argument dependent lookup.

Any trouble with the destination file will be flagged using exceptions, so we wrap the next section in a try-clause.

The flags to the ofstream constructor make sure we create an output file, remove its contents and use binary for transfer. Strictly speaking the binary flag is not necessary, as do not use the high-level parts of the library; instead we access the buffer directly. That makes sense in this case: We don't want to interpret anything, we just want a byte-for-byte true copy.

In line 17 we allocate a buffer on the heap. Using std::unique_ptr here takes care of the cleanup automatically.

std::streamsize is used for counting the number of bytes transferred in an I/O operation, or for the size of a buffer. It is basically a signed version of std::size_t.

On line 21 we fill the buffer from the input stream's underlying buffer. rdbuf() gives a pointer to the underlying buffer and sgetn(b, n) fills buffer b with up to n bytes from the input. It returns the actual number of bytes transferred.

The operation on line 22 is analogous, but for the output stream. sputn(b, n) takes n bytes from buffer b and puts it into the output.

The buffer address is taken as &buffer[0]. This works because std::unique_ptr provides operator[] when it is used for arrays.

In the catch-clause I only check for the Boost.FileSystem provided errors. Anything else must be handled further up in the system.

Wrapping up

So, what have we achieved? Copying from one file to another sounds a little like re-inventing the wheel, doesn't it? In some sense, yes, but as I said, I want to get progress information out of the copy operation, and that means implementing it myself. As said, this will end up being a class, with member data and member functions. Some of that data just might be bytes-copied-so-far, both for the current file and for the total. These counters can now be updated by inserting appropriate += expressions after line 22 in the loop.

In a simple test run the above function used between 1.298 and 1.556 seconds on several successive runs to copy a 1Gb random file from one folder to another on a local hard drive. GNU cp used on average 1.3 seconds for the same file, but did have the quickest copy at 1.272 seconds. So at this point we could start fine-tuning the buffer size, but frankly I don't think it's worth it.

I may change that tune once we start having to do other stuff in the core loop, though. We'll see.