I wanted to post this here in case someone else experiences the same problem. I have an NTFS-formatted external USB drive attached to a Linux (Debian Wheezy) server via USB 2.0 and the transfer speeds had been extremely slow. Excruciatingly slow. I’m talking in the range of 25 – 55 KB/s which is basically really horrible to the point of being unusable. Actually it was usable if I only did one and only one thing with the drive. For example I could watch a movie over the network from a file that was on the drive. But if there were any other activities that would try to read or write to the disk as the same time it was basically a no go.
At first I was not even sure what the problem was. So I did some testing. I had a very large file that was part of a download. Whenever I would copy the file over the network to the slow drive the speed was slow. But then I decided to copy it to a different computer. It was still going through wireless. The speed was vastly higher.
So that eliminated it possibly being related to the transfer being done via wireless.
Next I copied the file to the computer with the slow drive, but to a different drive on that machine. Sure enough the transfer speed was almost identical to the other computer – fast, basically.
So the problem truly was something with the external USB drive. I even downloaded a backported kernel for Debian Wheezy thinking it might be some arcane kernel issue. In fact it didn’t help. I even noticed the speed went down slightly.
I scoured through /var/log/syslog looking for any output that might indicate something amiss. But there was nothing there. Everything looked normal. The drive was detected as /dev/sdb1 and there were no issues. No error messages. Nothing.
Then I was doing some searching and something made me consider the fact that the drive is NTFS formatted, hence it uses the ntfs-3g library. Even though some people made comments in forums about ntfs-3g having poor performance, I didn’t really believe them. I’ve relied on this library for many years and there have never been any performance issues that I was aware of.
But I started to delve deeper. I decided to check the mount options for the drive. I noticed that one of the mount options was sync. According to this page the sync option means that all I/O is done synchronously, as opposed to asynchronously. I also noticed that in the official documentation for ntfs-3g from the site of the maintainers, there was no mention of the sync option at all.
So I decided to remove the option from the mount options in /etc/fstab. When I unmounted and remounted the drive and ran the test again – Bingo! The speed was literally almost 1,000 times faster, in the range of 20 MB/s.
I must have put the sync option into the mount options for an external drive that I had years ago, and every time I added a new drive I would just copy that line as a template since I have a few other custom mount options, such as specifying a specific UID and GID.
Since that drive is pretty integral to my home network – it is the primary media drive, among other things – it was kind of bottlenecking everything including the other systems which accessed it and would be slowed down because of the crappy I/O rates.
I had to deal with skipping music, choppy videos, and excruciatingly slow transfer speeds. That has all now quite thankfully been resolved. I’m still kind of in shock.
The test transfer of the large file, which before would have taken on the order of hours to complete had I let it, now takes only seconds.
By the way, this is the same drive I wrote about that is in the USB 3.0 enclosure. The Linux system its attached to does not even have a USB 3.0 port and unfortunately there’s no way to add one to it. But it really doesn’t matter since the I/O rate is now well above what I need on my home network.
Leave a Reply
You must be logged in to post a comment.