Crashplan Online Backup LOST MY ENTIRE BACKUP ARCHIVE

Crashplan Online Backup LOST MY ENTIRE BACKUP ARCHIVE

After almost an entire week of not being able to backup to Crashplan because of “archive maintenance” on their part, I was informed today that instead of merely maintaining my backup archive, Crashplan LOST THE ENTIRE THING. That’s right, Crashplan lost all of my data, unrecoverably. It’s gone – my entire online backup archive of all of my data – my life’s work – 2.4 tb of everything – completely gone. And Crashplan doesn’t seem to have any remorse.

I’ve been using online backup for a number of years. Before I switched to Crashplan, I was using Backblaze for online backup. Backblaze was great – a lightweight, easy to use front end, fast backup speed, low system resources, good support response times, and most importantly, a reliable service. However, the one thing Backblaze didn’t do at the time was offer unlimited, or very long retention. Backblaze would delete data from my archive if it didn’t see that data connected to my computer for a month – so if I went on vacation for 2 months and stored away my external hard drives, my data would be automatically expunged at the end of the first month. Because of this little quirk of their service, I decided to switch backup providers to Crashplan. Note that nowadays, I believe that Backblaze retains data for 3-6 months, which is much much better.

Anyway, I decided that having longer than 1 month retention times for my data was important, and switched to Crashplan. My initial impression of Crashplan was good – a flexible service with lots of configuration options. However, over the last few months of using it, and leading up to this giant Crashplan data loss disaster, have seen a darker side. Backup upload speed was nothing like I saw using Backblaze. I would usually only see 100kbps or so upload speed, even when I knew I had much more bandwidth. The backup service would load slowly, and would frequently re-scan my computer. The UI was finicky – if the wrong settings were changed, my entire backup archive would be expunged by the system – with no time to go back and undo anything. Sure they give big warnings of this, but a small grace period would be nice. And throughout all of these issues, Crashplan support sucked. I’d wait days to get a response, had multiple support agents dealing with my case on multiple threads – the whole thing was a mess.

So when I received the below email this afternoon, I was extremely disappointed and immediately concerned for the safety of my now un-backed-up data, but unfortunately I wasn’t very surprised – this is just the kind of disaster I’ve come to expect from Crashplan.

Below you’ll find a transcript of my current support history, starting from the time when I reported my “unable to backup” issue, and ending with my response to their meager attempt to remedy the situation.

I had a brief call with tech manager Brad W. this evening about the issue – he was a nice, knowledgable guy, and offered me a 2tb seed drive to re-seed my data. He also confirmed that I’m the only account that this data loss situation happened with, and that they’re interested in finding a suitable solution. I haven’t accepted a solution yet.

For now, the situation is unresolved, and my data is still gone. And since I’m still traveling at the moment, I don’t even know when I’ll be able to get home to re-seed all of my data to a drive, even if they do overnight one to me.

What should I do? Should I accept their 2tb seed drive resolution and move on? Should I jump ship and backup with somebody else?

Crashplan Support Thread

“What happened to my data?! “Unable to restore due to a backup archive I/O error”

Jad
Dec-09 01:58 pm

I’m very concerned.. I was looking into my crashplan settings to see
why my backup was taking so long, and noticed that although it says
I’m connected, it’s saying the backup destination is unavailable. Also
online I get a notice when trying to view my flies “Unable to restore
due to a backup archive I/O error”. What’s going on here? Is all my
data gone? Why am I not backing up? Do you need any diagnostic info
from my computer or anything? Help help – there have been way way too
many hiccups with my backup so far since I switched from backblaze to
crashplan. Once again thinking of switching back. I just don’t want to
be always worrying about my backup crashing all the time – that is not
good.

screenshots attached. thanks!

-Jeffrey
Screen Shot 2011-12-09 at 2.54.43 PM.png (quick view)
Screen Shot 2011-12-09 at 2.53.16 PM.png (quick view)

==

Dec-10 2011 02:05 pm
David W.
CrashPlan
Hello,

First, your data is still on our servers and safe. Your archive is in queue for maintenance. Unfortunately, there are a few large archives ahead of yours in the queue. Archive maintenance is necessary to ensure reliable data for restoration. This is also what caused the error that you saw when you went to the Web Restore portal.

As soon as archive maintenance has run on your archive, your backups will resume normally. Unfortunately, you’ll have to wait for the other archives in front of yours in the queue first.

Please let us know if you have further questions.

Sincerely,
~David

==

Dec-10 2011 03:16 pm
Jad
Hi David, thanks for your response!

It seems like my archive has been in the queue for quite some time, and as a result I’ve been unable to back up for a few days – how can this happen? One of the reasons I chose Crashplan was the promise of “Real-time continuous backup” – which I took to mean that my files would always be backed up, as my upload bandwidth permitted – so that when I added or changed files, they would be backed up ASAP. I’m a little disappointed that maintenance on your side would result in me being unable to backup for so long. Can anything be done restore my backup functionality I paid for? Also, I understand that planned system maintenance is probably expected – and bet that I actually agreed to it in the license agreement I accepted – although I can’t find any type of availability reference in this doc – https://support.crashplan.com/doku.php/eula – but I’m not a lawyer and am almost definitely not reading/understanding it correctly.

Anyway, let me know if anything can be done to get my backup restarted, or if it would be at all possible for me to get a drive to backup to while I wait for my online backup to be restored.. Could I do another seed drive and have it added to my existing backup archive (not replace it)?

Thanks

-Jeffrey

==

Dec-10 2011 03:48 pm
David W.
CrashPlan
Hello,

Unfortunately, another Seed Drive would replace your archive rather than be added to the current archive. Also, the only thing we can do while your computer is in the maintenance queue is to wait for it to complete. This is part of the reason that we highly recommend that you use CrashPlan to backup to more than one destination (i.e. another computer or an external harddrive).

Sincerely,
~David

==

Dec-10 2011 04:38 pm
Jad
This is rediculous. It’s been 4 days since I was able to back up, and now I’m receiving warning emails from your system – for a problem that I have no control over! I actually DO backup to other locations with Time Machine – however that has nothing to do with the fact that Crashplan+, which I pay for, has not allowed me to backup for 4 days! What is going on here? Why in the world would I pay for such shitty service? Since I switched from backblaze to crashplan, I’ve had nothing but problems.

Please fix this, immediately.

-Jeffrey

==

Dec-12 2011 02:35 pm
Chad V.
CrashPlan
Hi Jeffrey,

Thanks for calling–give me a call back when that scan finishes!

~Chad

1-877-680-7526

==

Dec-12 2011 02:46 pm
Chad V.
CrashPlan
Hello Jeff,

Could you please send me your CrashPlan log files so I can take a closer look at the behavior you’re seeing? Please do the following:

– Open CrashPlan
– Double click the CrashPlan logo in the upper right
– Type: getlogs [REDACTED]
– Press enter

CrashPlan will automatically zip up your log files and attach them to this ticket.

~Chad

==

Dec-12 2011 02:49 pm
Chad V.
CrashPlan
Hi Jeffrey,

Okay–got a response back *right* away! Apparently after the archive move we ran a maintenance job on it (which is normal) and yours says it’s still pending because there were a few folks ahead of you in line. I’ve ‘bumped’ you to the top of the line so your archive should run maintenance and finish tonight, hopefully. It should be ready to access later on tonight/tomorrow. I hope this helps.

~Chad

==

Dec-12 2011 02:58 pm
Jad
Ok, thanks very much for bumping me up to the front of the line – hopefully I’ll be able to backup soon.

Is suspending a paying customers backup for 4 days normal or acceptable? This seems like a bit long to go without backing up.. what do you think?

-Jeffrey

==

Dec-12 2011 03:18 pm
Eugene O.
CrashPlan Support
Hello Jeffrey,

Thank you for your patience as we looked into this issue. Unfortunately, I have some bad news regarding your maintenance job. Your backup archive had corrupted data, and the archive could not be properly repaired by archive maintenance in this situation. This is a very rare situation. We are still looking into the root cause, but unfortunately much of the archive was affected and very little was able to be rebuilt. I sincerely apologize for this situation and the inconvenience of this event.

We definitely want to get you backed up and secured again, and I can expedite shipment of a seed drive to you so that we can upload your data again. We will cover the cost of the seed drive and all shipment, and we can get this out to you today if that would work for you. Could you contact us back with the shipping address, or simply confirm that the shipping address we previous used is the one you wish to use again?

From our records, I have the shipping address as:
[redacted]

Please let us know if we should use this one, or if you’d prefer a different address. Again, please accept our apology for this issue. Please contact me if you have any questions.

Regards,
~Eugene

==

Dec-13 2011 03:17 pm
Jad
Eugene-

Thanks for the heads up on the issue. After not being able to backup for almost a week, and now being informed that my ENTIRE BACKUP IS LOST, I’m extremely unhappy. How in the world can this situation possibly occur? Crashplan advertises itself as a reliable, real time backup solution. I was sold on the promise that my data with Crashplan would always be protected, maintained, and available to me at all times through the app, website, and mobile. app. To tell me that my data not only became corrupted, but was also unrepairable and then completely lost is absolutely unacceptable, for any level of service. Crashplan was my primary backup, and I’m extremely scared now that my data is unprotected and GONE.

I understand how competitive the online backup industry has become recently, and that one of the main hallmarks of marketing and competition is the reliability and ease of use of your service. You advertise both “Files secured in data centers worldwide” and “Real-time continuous backup”, yet I have experience none of this. My backup has been stalled for almost a week, and now your data center has completely lost my data.

Up until I started having issues with Crashplan last week, I had actually been recommending you to various friends and family. In fact, I set up both my Mother and Sister with Crashplan. How do I explain to them that the service I had once recommended has completely lost my data?

Regarding your “solution” – it’s weak, and highlights the fact that you really don’t care about me or my data. You’re offering to send me ONE measly 1tb seed drive? Did you not look at my account and see that I have 2.5 tb of data selected? At very very very least, you could offer to seed all of my current data to your online data center. A 1TB seed drive service costs $124.99 – so are you saying that all of my data – my life’s work – is only worth $124.99 to you? That’s insane.

I understand that periodically data issues do occur, and I appreciate you owning up to the issue. I’d like to have my confidence in your service restored and continue out the rest of my 4+ year prepaid account. I’d like a swift and appropriate resolution to this issue. Please let me know what you can do for me. Also, look forward to my blog post about this, and associated information sent over to Consumerist, ETC.

Your un-backed-up almost former customer,
Jeffrey

==

Dec-13 2011 05:00 pm
Brad W.
CrashPlan
Hello Jeffrey,

Here’s a recap of our phone call.

Here is what we can do to get you backed up again as quickly as possible.

We can send you a 2TB hard drive so you can seed your data to that. The remainder of your backup archive will need to be uploaded via the internet. We can comp you for the time it takes for the last part of your backup archive to upload to our servers.

We have your email address so we can send out a drive overnight once you make your final decision as to what you want to do.

~Brad

Dec-13 2011 05:26 pm

**UPDATE 2011-12-14 13:03 EST**

Looks like this is actually not an isolated issue, despite what Crashplan told me point blank on the phone last night. Facebook user Diane Dusek just posted on the Crashplan Facebook wall that she is having a very very similar issue. Screenshot of the Facebook conversation:

** UPDATE 2011-12-14 17:30 EST**

It’s 5:30 – the end of the day. I’ve called Crashplan twice today to talk to somebody about resolving my issue and moving on. On the first call I was put on hold for a brief minute, and when the guy came back on the line I was told that somebody could call me back shortly. 30 minutes later and no call, I called back and was once again put on hold, then told that somebody would definitely call me back within 30 minutes. That was 45 minutes ago. I’m simply being brushed off by Crashplan. Do they thin that this issue will simply go away, like all of my data did? I couldn’t have imagined how they could make the situation of losing a customer’s data worse, but somehow they managed to.

** UPDATE 2011-12-14 19:09 EST **

I just received a personal call from Matthew Dornquast of Code 42 software, makers of Crashplan. He was extremely apologetic, and did a great job of being extremely nice and understanding, and explaining the entire situation to me in as much detail as he could. I took a few notes from the call, and with Matthew’s permission, am publishing them. In the end, Crashplan is sending me out a 3tb seed drive so I can get my entire archive back online, and they’re also issuing me an account credit for the previous seed drive service I had purchased. Also of note, Matthew mentioned that he had not in fact read my blog post on the subject, but did read through the support thread, and heard from the Crashplan social media person that there was an issue that needed his attention.

Notes from call with Code 42’s Matthew Dornquast

  • The problem with data being lost was due initially to a hardware failure in one of the data centers Crashplan uses
  • Crashplan manage 62 petabytes of data across its data centers,  which is a huge amount.
  • In this incident, a single server had a hardware issue, which normally wouldn’t take down the whole thing and cause any data loss.
  • However, there was also an element of human error in this case, which ultimately caused a loss of data for about 20 customers.
  • All 20 customers did not lose all of their data – only some did.
  • Only 20 users were affected, out of the millions of Crashplan customers
  • Human error was a blind spot for Crashplan’s system, and there was no process in place to prevent it.
  • As of now, there’s a new process to prevent this type of human error from happening again.
  • This year Crashplan is expecting 380% growth. As a growing company, they are obviously still having growing pains.
  • For the 20 people affected by this snafu, Crashplan is offering free seeding to get data back into cloud.
  • Crashplan is a backup company, not an archive company. Don’t put all eggs in one basket – multiple backups is the way to go so that if one backup fails, there is a second backup. [ed. Which I do – I use Apple’s Time Machine as my local backup]
  • In this issue, software in the Crashplan system software detected data issue, and started healing immediately.
  • System was able to repair a lot of the data, and of the 20 people affected, very few actually lost all of their data.
  • Crashplan normally only offers 1tb seeding drives. But in this special case I can be sent a 3tb drive for seeding my 2.4 tb of data. [Although Brad told me last night that it was technically impossible for them to import more than 2tb of data from a seed drive into their system, Matthew assured me that his team would figure out a way to make it happen on his request]
  • Looking forward, Crashplan is coming out with new mobile stuff – iPad, android, windows phone 7 apps
  • New features for travelers – don’t use mobile access points, don’t run on battery power.
  • Summary – screwup by software people, 20 people affected, not the end of the world, working hard to get affected customers backed up again, putting systems in place to prevent this from happening again.
  • Responsible message to remind people of: Crashplan isn’t an archive service, it’s a backup service. Important to remember that multiple backups are the way to go for better security.

**UPDATE 2011-12-15 13:20 EST**
Crashplan released an official statement on the issue that caused the loss of my data:

On Dec 13th a storage node in one of our Minneapolis data centers experienced a hardware failure. While typically a non-event, additional human error escalated this failure into backup archive corruption for 20 of our customers, affecting their backup in varying degrees.Since CrashPlan automatically detects and heals around archive corruption, the affected customers don’t need to do anything – their backup archives will automatically heal over the internet. (However, to accelerate this process, we have offered overnight free seeding services to those customers.)While CrashPlan manages over 60 petabytes of consumer data globally and this event was limited to 20 people, we nevertheless take this failure very seriously and sincerely apologize for this lapse.Further, our operations team has modified our processes to avoid this human error in the future.



**UPDATE 2012-01-07 22:43 MST**

I've redacted my actual ticket number from the command line noted above. It seems that a user was able to successfully add their own crashplan's log files to my support thread by using this line.. not good at all.

**UPDATE 2012-01-07 10:33 EDT**

Reader and beauty photographer Ashley Karyl just commented below that she's been having big issues with crashplan too. She's also noticed some new behavior. According to network monitoring tool Little Snitch running on Ashley's computer, Crashplan made a connection to Amazon Web Services S3 cloud storage service this morning. The behavior deviates from the standard Crashplan behavior of connecting directly to Crashplan's servers for backup. Crashplan has obviously had its share of issues lately. There's even a static message posted on its support site indicating that they're getting more support issues than normal"

"We're currently seeing unprecedented demand for technical support, due to our recent rapid growth. As a result, our support response times are significantly longer than we want them to be.
Realtime live phone support and chat is available for customers experiencing urgent issues (help restoring files, lost or stolen hardware).
We are working diligently to resolve the delayed response times and apologize for the inconvenience."

So could this Amazon AWS connection attempt indicate that Crashplan is making an emergency switch over to S3 to fix their own server farm's I/O issues? Maybe, maybe not. Also to be considered is the fact that Crashplan just updated its software, which existing customers computers will automatically download and update themselves with. Very frequently developers use S3 to distribute software downloads - so it's very likely that Ashley's S3 connection was simply to download the latest version of the Crashplan client. In fact, I think this is very likely the case.

However, it's still worth noting that things have been a bit bumpy for Crashplan lately, as evidenced by the static support message, and increased number of comments posted here.

Below is Ashley's email explanation, and screenshot of her Little Snitch connection log.

Hi Jeffrey,
I didn't grab a screenshot of the message as it popped up in Little Snitch, however I did just find it in the rules and I've attached a screenshot of that instead. Immediately afterwards the upload started to work normally and has continued for the last few hours. It could be that Crashplan are simply using S3 for the download of updated software, which many developers do, however it would seem strange in a case like this given the network they have.At the moment it is uploading nicely but this is after a week of problems and I'm still looking at a month to upload the rest assuming there are no further delays. I'm still on the fence about signing up but some options are simply too expensive and it's a pain for me to start again with this slow connection while I'm waiting for fibre to be enabled in this area.ThanksAshley
PS I'm a photographer as well and want my archives backed up offsite to ensure I am covered if there was a fire or theft at the house.

 

**UPDATE 2012-04-05 17:13 EDT**

In light of the torrent of recent comments, and in the interest of getting some sort of official response from Crashplan on this ongoing issue, I've just tweeted directly to Code42 Software's CEO Matthew Dornquast. During my initial issues with Crashplan, which prompted me to write this blog post, I had a phone call with Matthew - he was generally very nice, knowledgeable, and pleasant to talk to, and helped speed me through the process of getting my backup re-seeded. Hopefully he'll take heed of this post and let everybody know what's going on here, and why users are continuing to experience issues.

My Original Tweet
Matthew Dornquast's Public Twitter Stream

https://twitter.com/Jeffzilla/status/188011324305969154


**Update 20120-09-26**

After having crashplan crash multiple times on my Macbook Air running 10.8.2, I wrote in to support to ask for advice. Their response? "you're running out of memory and need to manually increase memory". Really? I have to manually fix your app that keeps crashing? Terrible.

Here's my initial problem:

Whats going on here? My computer has been on and connected to the internet just fine lately... Error message I received: "Computer "My Computer Name" has been unable to reach any backup destinations for 3 DAYS. Back up to multiple destinations to reduce your risk of losing data."

I forwarded Kelly my Crashplan log files. Then her response after reviewing my log files:

Good day Jeffrey,I believe that you may not be backing up due to out of memory errors. Looking through your logs it appears that the CrashPlan backup engine is running out of memory. Please follow the instructions below to allocate more memory to the CrashPlan backup engine.Edit the CrashPlan engine's com.crashplan.engine.plist ("the plist file") file to allow it to use more java memory. You will need to use Terminal for this, and edit the file using the 'sudo' command. When prompted, type in your computer's Admin password. Note that you will not see the text-cursor move when you do - this is normal.1. Stop the backup engine by typing this into the Terminal application:sudo launchctl unload /Library/LaunchDaemons/com.crashplan.engine.plist2. Run this command to edit the backup engine:sudo nano /Library/LaunchDaemons/com.crashplan.engine.plist3. In /Library/LaunchDaemons/com.crashplan.engine.plist, find this line: -Xmx512m4. Edit that line to something larger such as 640, 768, 896, or 1024. E.g.: -Xmx1024mThis sets the maximum amount of memory that CrashPlan can use. CrashPlan will not use that much until it needs it. I would recommend starting out setting it to 768, and go higher only if you continue experiencing problems. You can increase it above 1024 if you have a really large file-selection.5. Hold the Control key and tap the x key to exit. Choose "y" to confirm it.6. You'll see the prompt "File Name to Write." Hit enter to save to the existing location.7. Start the backup engine by typing:sudo launchctl load /Library/LaunchDaemons/com.crashplan.engine.plistIf you have any problems with this process, you can call us directly at 1-855-411-4242. We are available during the week 7-7 Central Time. Please let me know if you have any questions.Regards,
~Kelly

So, for an out of memory problem, they're trying to instruct me on how to go in and manually fix it. For a consumer facing product, this seems much much much too complex to expect me to do on my own. Crashplan, fix your app.

And it seems like they are working on fixing it with a Native Mac Crashplan App!

I got a nice tidbit of information from support rep Kelly - Crashplan is working on a Mac Native Crashplan App. That's right, native code, no java wrapper crappyness. Here's Kelly's words:

The reason that it is not native is because CrashPlan is a cross platform program. Currently we are working on a native Mac client, but this does not have a time frame on when it will be released.

- Kelly S., Crashplan Support

Hoping for that new native crashplan client to be released soon...

**Update 2014-09-06**

Still no native MacOS App... let me know if you hear any updates on this..

Categorized: ,

Comments

214 responses to “Crashplan Online Backup LOST MY ENTIRE BACKUP ARCHIVE”

  1. Damn man. That’s really messed up. This is exactly the reason why I’m working on putting together a rack mount server, colocated in a data server someplace secure, in nice RAID configuration that *I* control. Events like these will always crop up when relying on “cloud” services, so I’m building my own cloud. Sorry to hear about this.

  2. I think you had a sane plan: Time Machine + CrashPlan. It sucks that this happened & that CrashPlan has such a poor response plan (assuming they have a plan at all to compensate for losses like this).

    What I would do in your shoes?? You prepaid for 4 years, so write that off — but don’t let those ass wipes off the hook. Use their service; and do what they recommend to get all 2.4TB backed up. Then re-add Backblaze to the mix. Two clouds are better than one, right? Maybe.

    That might require extra bandwidth monitoring and throttling; but could add some peace of mind.

    I’m curious about what you use for Time Machine. The previous commenter mentions remote RAID drives in a rack somewhere. I’m thinking about sticking with Backblaze and a Drobo …

  3. Just curious, what do you think the value of your potentially lost data is? And how much have you paid Crashplan to store it for you?

    1. Priceless. Photo library since 2000, all of my college research, everything i’ve ever done.

      60,000 song itunes library – so, just that could be argued to be valued at $60,000 on itunes, plus the countless hours spent making playlists, etc.

    2. One commenter above said that you prepaid for 4 years, but I note that you did not address that issue in your response. A person who prepaid for 4 years of unlimited storage today (I am assuming you had the “Crash Plan + Unlimited” plan, since you mention 2.4 T of data) would pay $139.99 for that storage.

      Cost to store 2.4T for a month on Amazon S3: $239
      Cost to store 2.4T for a month on Rackspace Cloud Files: $360
      Cost to store 2.4T for a month on Google Cloud Storage: $305
      Your cost to store 2.4T for a month on Crashplan: $2.92

      Does anything seem odd to you about those numbers?

      Crashplan’s pricing doesn’t even cover the cost of buying a consumer-grade hard disk (whose warranty won’t even extend to 4 years) large enough to store your data, much less building some sort of redundant system to store multiple copies, or motherboards and software and AC power and cooling and security and sysadmins and bandwidth and all of the rest.

      But they’re gonna make it up in volume, right?

      What they’re proposing is some sort of bubble-inspired cross between a Ponzi scheme and a perpetual motion machine.

      Consider yourself lucky you got an early wake-up call while you’ve still got your local copy of your data and sign up with a provider that’s actually charging enough to provide the service you need on a sustainable, ongoing basis – or do it yourself. Or, better yet, sign up for a couple of them. (No, I don’t work for any of them.)

  4. Jon Moses

    Yeah, that’s not really acceptable in a backup service.

    “We lost your data, oopsie. How about you send it to us again.”

    Ask for a refund and tell everybody they suck.

    1. Yep, planning on it….

    2. TByrd_McCloy

      You cry and scream like some atrocity has occurred here. Look at those figures that nice man left.

      Cost to store 2.4T for a month on Amazon S3: $239
      Cost to store 2.4T for a month on Rackspace Cloud Files: $360
      Cost to store 2.4T for a month on Google Cloud Storage: $305
      Your cost to store 2.4T for a month on Crashplan: $2.92

      And Amazon S3 still lost people’s data!
      The point here is that nothing is fool proof except 2 copies in 2 different places via 2 different mechanisms of backup.

      You have the right idea with Crashplan (offsight) + Time Machine (local), but you’re asking for beyond enterprise service for literally dollars per month. Get real dude. I wait more than 30 minutes for a call back on a $25,000 Commvault Tape library backup system with $1600 per year in maintenance. Hell, I’m at 24 hours right now. Assuming the system even lasts 10 years without catastrophic failure, what is that like $350 a month total cost of ownership + $70/tape?

      Try 5 day hard drive replacement on a $100,000 tier 1 NAS for $8,500 maintenance a year, and have an engineer tell you to manage your expectations when that NAS loads final cut projects for your editors 10 times slower than a $10,000 Xserve. (suck it omneon btw :)

      Believe me, it doesn’t get better, you can’t spend your way up and out of this one. Not from a hardware reliability standpoint or a quality of service standpoint. These companies don’t exist to service the “always right” customer anymore. They don’t give a shit about you. Why do you think they only respond when you threaten a PR disaster?

      When you find some offering from Drobo / Dell / Sans Digital / Highpoint / Lacie that is at any point more than 1 firmware update or 6 months away from some “unpredictable” data disaster, followed by utterly atrocious support, please do post here. I’ll check back.

      Otherwise, just take their offer to reseed your data for free and keep getting a false peace of mind for $3.00 / month.

  5. Check out their Facebook wall (https://www.facebook.com/CrashPlan?sk=wall). There’s at least 2 other people who had the same problem you did (and one of them lost their entire backup as well).

  6. “This year Crashplan is expecting 380% growth. As a growing company, they are obviously still having growing pains.

    Crashplan is a backup company, not an archive company. Don

  7. I am sorry to hear about your luck with crashplan.

    Have you considered backing up your other data (website and emails)?

    I am sure that CrashPlan will figure out how to get your 2.4 T of data.

  8. Nathan

    OMG. Where to begin.

    Jason: The cheapest RAID box you will buy will set you back at least a $1,000. Drobo w/ some 3TB drives (at least 2 drives). And then.. you will still lose. I’ve been supporting desktops, server and networks for more than 17 years. All solutions fail; just a matter of time.

    Greg: You started off good, but lost your own point. First off; not everyone is backing 2.4TB to CrashPlan. I would bet the average user is under 50GB and less than 5% have more than 1TB. In volume, CrashPlan disk space will probably cost them less than 30 cents per Gigabyte or $15 per user on average. Secondly; they repeatedly sell themselves as a Backup company, not an archive. That is significant as they are not a “primary” source of data. Google Apps is an example of a “primary” data source. If they lost your data, it would be gone. Your CrashPlan data is one source of many – if CP loses your data, you have at LEAST one other copy.

    This brings me to Jeffrey’s case. Jeffery has 2 other sources of data intact – his original and his Time Machine backup. Jeffrey was struck by lightning (actually, you are 10 times more likely to be struck by lightning -280/yr/USA) figuratively speaking. Even so, he wasn’t harmed. There is nothing he lost – all his data is intact. One of his backup destinations suffered corruption and is being repaired.

    I don’t recall the last time I heard someone cry so loud who hasn’t actually suffered any injury. In fact, Jeffry came out $125 ahead – as they reimbursed him for fee he had already paid prior.

    All companies have customer support issues. In this case CrashPlan had a constant stream of correspondence addressing his concerns. Ultimately, the CEO called him personally. They’ve reimbursed him and done everything (seemly) possible to get the situation rectified.

    Sheeze! Reminds me of the case a few years back when a Judge out east sued a small dry cleaning shop for a $1,000,000 because they screwed up his pants… except the Judge actually sustained some damage.

    And.. I can’t believe you guys got me to so OMG. Sheeze!

    I love CrashPlan. I backup to them remotely and also to a Drobo unit locally. Multi destination as I KNOW everything fails.

    -Nathan

    https://www.lightningsafety.noaa.gov/medical.htm

    1. Nathan, the issue here is not that Crashplan failed, it’s how they responded to it. Lack of transparency, lack of urgency, lack of empathy. That will severely shake anyone’s trust in a company, and Crashplan is ultimately a company built upon trust.

    2. Elliott

      I stumbled across this while looking up CrashPlan. some of the points are good, but Chris is just being unrealistic. Near as I can tell from reading the story, it looks like CrashPlan themselves didn’t know what happened, and when they did they let Jeff know right away (it still could have been handled better, but geez, what more do you want? Companies are run by people, and people drop the balls sometimes) And the fact that the CEO himself personally called Jeff is service that is non-existent in any business I’ve heard of for 20 years. I’m not sure how much more ‘transparent’ you can get other than an apology from the CEO. Is Chris some kind of bitter ex employee or something?

    3. Yep, it was pretty awesome that the CEO himself called me personally.. and from his personal cell phone too, without caller ID blocked or anything – took a personal risk, and his personal after-work time to chat with a customer – that was awesome.

      I actually just finished re-syncing all my data with the new seed, and I’m not back up and running, still with crashplan. I’m going to be posting a followup post sometime soon once I’ve assessed where I’m at again.

      I don’t know who Chris is, I believe he’s a random reader who found my blog via search. In any case, I appreciate all the back and forth!

  9. While not inexpensive, data recovery on your original failed HD may be an option… esp. if you could nail CrashPlan for the expenses. ~sf

    1. Yep, I had an unbackedup 500gb hdd fail on me a few years ago “head crash” , and used Drivesavers to get my data back. Cost a ton, but I got 100% of my data back. Rockin service, but a zillion times more expensive than a good backup plan.

  10. […] CrashPlan lost data for (at least) 20 users, reveals major issues with their infrastructure (Jeff) (readability) […]

  11. This whole matter speaks volumes about CrashPlan’s transparency.

    1. Support staff repeatedly assure you that it’s a “normal” maintenance job. No mention of hardware failure.

    2. Official statement admits hardware failure and loss of data, but claims that the data will automatically rebuild and that the seed drives were simply to speed up the process. This contradicts what support staff and Matthew told you, that the data was unrepairable!

    It’s great that you got the matter resolved, but I’m still alarmed at how they completely overlooked their customer relations issue.

  12. Something doesn’t make any sense to me. You said that you lost all your pictures, college research..etc… What happened to your local copy of that data?

    1. Nothing happened to my original copies of the data, or my time machine backup – they’re all safe and sound. However, the crashplan backup of all of that data was lost. So in the extremely rare case that I would simultaneously lose my original plus on-site backup (as in a robbery or fire), I would permanently lose my data. I was particularly scared about my data in this case, because the Crashplan loss of data (LOD) occurred while I was traveling, and had guests staying in my apartment – making myself particularly vulnerable to mishap or robbery. Anyway, Crashplan sent me a 3tb drive which I did a complete backup to, and they’re now loading it into their system as we speak. I’ll be writing a followup blog post as soon as all my data is loaded in and everything is back to where I was with Crashplan.. I’m looking forward to having a better experience with them going forward!

    2. Elliott

      Wait wait wait–so you didn’t actually lose anything?

    3. Nope, I didn’t actually lose any data – during the time my crashplan backup data was lost, I was traveling and had guests staying in my apartment where I housed both my original hard drives as well as my time machine backup drive. So there was a time where all my eggs were in one apartment-shaped basked, with no off-site backup. So there was significantly higher risk I was exposed to. but in the end, I got home, physically separated my backup from my originals, and then got to re-seeding with crashplan. Now that the seed is complete, my originals and time machine backup drive sit side by side, prone to local hazards, however there’s a safe copy on crashplan’s servers in a separate location.

  13. My MacBook’s data is also lost, deleted from the server. I only discovered the fact because the CrashPlan client was hogging CPU on my laptop and the client app was crashing on start. Now the website tells me “Unable to restore due to a backup archive I/O error” or throws other errors. In the My Computers section, the MacBook listing shows backups of 83 files totaling 6.9MB (instead to the GB of data I should have.)

    I’ve used Arq to backup my user data to S3. Looks like I should start the SuperDuper! – Time Machine tango as well.

    I never liked Java on the Mac anyway.

  14. Cyprian

    The same issue here – CrashPlan started to upload my files from scratch for no obvious reason. Backup history (files versions) has been lost! This is very very disappointing!

  15. I’m loving my crashplan… But I would never trust an outside source with my data so I use the free version of crashplan and backup to one of my servers that is located in a data center. All of my servers and laptops are very happy with crashplan. I moved from spideroak after I was having too many issues.

    Sorry to hear about your issues but know that the software is sound, even if their servers are not.

  16. “Crashplan is a backup company, not an archive company.”

    I think that’s a cop out – whats the point of paying for a backup when it isn’t 100% reliable? I understand that they handle a large amount of data – but that’s their (only!) job, not an excuse.

    I feel your pain on crashplan’s support – its absolute rubbish. I have been having connectivity issues – it takes days for them to reply to emails, and the “support” seems to consist of acknowledging that there are problems and saying sorry – but nothing is resolved – the issues continue.

    Growth isn’t an excuse either in my opinion – if crashplan can’t handle the load they could always limit new sign ups instead of taking our money up front and then providing a sub standard service couldn’t they?

    1. I agree, it sucks – their typical responses suck, and their service is a bit clunky. UI and system integration needs a major update (java?!).

      However, the CEO was nice to me when he frantically called. And no other online backup services offer what they do. So it’s a touch situation – go with a so-so company with a good feature offering for a good price, or go with a more solid company with a more conservative feature offering at a higher price. I’m still not sure where I fall on that, but I’m sticking with Crashplan for now, because I’m lazy and already have my data with them, and don’t want to re-re-re-re-upload everything to another service.

    2. I’m sure their CEO was nice to you – after all your rightfully damning entry has hit Digg, and I found it high up in Google so this dose of reality has the potential to cost them a lot of business (as it should).

      The “we’re only a backup company” line particularly scares me – I’m questioning if their “backup” has any value at all if (according to their own words) I can’t 100% rely on it.

      I think you’re being way too easy on them, this could have been an absolute disaster if you found this out in an emergency. Plus you are re-seeding everything anyway so no is the ideal time to move providers.

      Thats my 2c anyhow.

      In my situation I have been unable to backup for close to a week now. No response from support, according to their system the ticket hasn’t even been assigned to anyone. I haven’t even been able to complete an initial backup of 200gb.

      Swamped support staff, systems that aren’t coping, the omens aren’t good for Crashplan.

      Do you know of any others? Even if they are more expensive, I’d pay extra for something that I can rely on

  17. I signed up for the demo with a view to starting on the 4 year unlimited plan but I’ve also been having a problem for the best part of a week now with an almost full time message that the destination drive is disconnected. The quantity of data shown as being backed up in the Destinations > Online panel also goes up and down. Yesterday it said I had 54 gigs backed up and then a few hours later there was just 45 gigs. Has 9 gigs been lost?

    Crashplan have admitted they have a problem and say they have upgraded the hardware but need to sort out some I/O issues that will take time. They suggested I could start from scratch with a different server but that would mean writing off twelve days of backups and frankly it’s all rather reminiscent of when I tried with Mozy a few years back but eventually gave up. I also failed to get a response from support initially until I raised the point via Twitter.

  18. Curiously just a minute after my last comment Little Snitch flashed on my screen and asked if I would allow Crashplan to connect with Amazon S3. Then when I went to the main app it said a new version of Crashplan was available and being downloaded. Time will tell if this fixes anything.

    1. Ashley that’s very very interesting that little snitch reported Crashplan connecting to S3… Perhaps they’ve had such disastrous internal I/O issues that they’ve made an emergency switch to Amazon AWS S3? No idea… Could you screenshot the little snitch notification, or the log file with the connection attempt details for me to post?

    2. I’ve just sent an email with a screenshot to the address shown on your contact page.

    3. Got it, thanks!

  19. I switched to Crashplan after Mozy’s price hike; I agree that their advertised service is unparalleled. But after waiting ~2 months to finish my initial backup I woke up this morning to see that my crashplan central backup had, for some reason, restarted from scratch. I had 115GB backed up, now I have 2. No idea if the data are still there somewhere or just poofed into the ether.

    For all the people saying “wait, you didn’t lose anything?” – he did. He lost the service that he’s paying for. If you were paying a security company to guard something for you, is it ok for them to go away and still charge you so long as you don’t get robbed during that time? No, of course not – and anyone who thinks otherwise doesn’t understand the 1st thing about data security.

    I want to be able to stick with Crashplan but so far it’s been pretty rough.

    1. carlos

      The same happened to me yesterday Evan, 30 Gb data lost. Crashplan says “missing backup”, this is ridiculous. Could somebody recommend me an alternative service? I’d like having versioning and more than 30 day backup.

      Thanks.

    2. Got an email from CP:

      David W., Mar 23 06:03 pm (CDT):

      Hello Evan,

      I’m writing to inform you that your backup was on a system that needed to be taken offline, and the backup was then restarted on a new system as a precaution.

      In certain rare circumstances, CrashPlan needs to service equipment/archives in such a way that prevents existing backups from being available for a period of time. During these periods, it is standard procedure to run a second backup archive in parallel until either your data backs up again or your archive can be brought back online. This is done as a safety measure. For this reason, we have redirected your backup archive to a new server and your backup is rebuilding.

      Your other backup archive is still on our system and is intact, and if you require data from that archive, we can make the archive available. If your backup will rebuild faster by simply uploading again, then no action is required. Your backup will reconnect and rebuild the archive automatically.

      Please accept our apologies for any inconvenience caused by this event.

      To compensate you for time required to rebuild or re-locate your archive, we’d like to offer you a complimentary subscription extension of 3 months. If you have any questions please let me know.

      Regards,
      ~David

      My reply to David:

      So the old backup can be brought back online? It will certainly not be faster to rebuild again.

      A few things:

      1) if you’re going to take a system offline, you should inform the impacted users in advance if possible or at the very least ASAP. Simply having the backup disappear is unacceptable, for obvious reasons. This should include an estimate of how long the system will be down, and an assurance that data will be made available by a given date. It shouldn’t be hard to know (1) the estimated time for system recovery and (2) the IDs of impacted users. Getting out in front of these issues goes a long way to maintaining your customers faith.

      2) you say this occurs in “rare circumstances” – can you give me a sense of how rare? Is this something that most likely won’t happen again in my 4-year subscription? When I read about other users having problems it often is said that these things happen in “rare circumstances”; I realize that one generally only hears about the bad things that happen, but unfortunately that reads like boilerplate to me.

      3) on a somewhat related note – I have no idea how long it will take to rebuild the archive, as CrashPlan’s “estimated time remaining” function is essentially useless. So far as I can tell it estimates time remaining based on the instantaneous upload speed at any particular moment, rather than the average upload speed over the life of the backup. This means that there are times that CP tells me my ~100G backup will be completed in 2 hours, and other times it estimates 60+ days.

      I switched to CP from Mozy because your advertised service is notably better at a significantly better price. But it’s worth noting that in two years with Mozy I never once have the connectivity problems I’ve had, nor did I ever have to rebuild my archive. I’d be much more comfortable with where things are if the old archive were re-connected ASAP so I can confirm that it is indeed all still available. I assume you’re familiar with the Jeffrey Donenfeld blog post where a tech support staff (with your same name/handle, no less) assured him that his data was secure – until it turned out to have been lost. With this in mind I’m sure you’ll understand that I don’t blindly accept your assurances that the data are safe, but rather would like to confirm this for myself. If the data are NOT available, I’d like you to provide me with a seed drive to reseed the archive as quickly as possible.

      Your offer of 3 months free is appreciated, and I accept, I suppose (though my account seems not to reflect this change as yet) – however the bigger point is that the service I’m paying for is to have the backup NOW, not in 4 years and 3 months. As such I still lack backup of the last 15G of my previous archive, and I remain somewhat concerned about the status of the remaining 100G. I urge you all to take some time to really lock down your core service, as your success depends on it being (a) bulletproof and (b) perceived as bulletproof.

      Thanks for your attention,

      Evan

      We’ll see what response I get. And we’ll see if these comments will parse HTML code…

    3. Wow Evan, this is really exceptionally bad – I totally agree with your stance on this – unacceptable to take your data offline, even for a minute, without thoroughly preparing you. And then what’s with rebuilding your archive? If they have the power to bring your existing archive back online, why don’t they just do that? Sounds sketch.

      Thanks very much for posting this comment.. I’m sure a ton of people will find it useful…

      And yeah, I’m sure they know about this blog post – during the week that my whole situation was going down, I got a frantic call from the CEO as he was driving home – not good.

      Cheers!

    4. hi Jeffery – If it’s all right with you I’m going to continue documenting my interactions with CrashPlan here, since this is a fairly visible space and I think it’s important that they know these things will out. Let me know if it’s annoying!

      Got a one line response to my page-long email of the other day:

      “Hello [Evan]

      The migration of your archive is complete and has been verified on the new server. Please let us know if you have any further issues or concerns.

      Sincerely,
      ~David”

      Frankly I’m a bit stunned at this response, given the time and care I feel I put into my previous email… I was going to post my reply but I’m not sure I can formulate one that adequately expresses my disappointment without some time to think about it, so I guess I’ll wait for a bit.

      For what it’s worth, my archive does appear to have been restored, however the promised 3 months of free membership have not yet been applied, so far as I can tell.

    5. Sure thing, go for it.

  20. It seemed OK following the software update for a day or so but now the problem is back and it keeps saying the destination drive at Crashplan is disconnected. I’m just curious if I’m alone or others are finding the same?

    1. carlos

      Same here! I’ve reset my backup, but the same…

    2. tschaenk

      Yup. Same here.
      I’m not able to backup for 2 weeks now.

  21. Damn, I always check this post. Scary. Hope it doesn’t to me… I only just finished backing up my main stuff, 150gb.

  22. Lost all backup, now its re-backing up all 50 Gigs of my data…. bad after backing up 25% of it has lost access to the backup archive….. No response to either mails or call…..these clowns need to be exposed….

  23. Any recommendation for a replacement service? After last CrashPlan update I’ve been without backup for 11 days now

    1. Unfortunately, no, not yet – for the money, CP is still the best out there, IF their service works as advertised. If it doesn’t, than anything else is better…

    2. Hmm, I had the exact same problem and received the exact same email message. (Googling “CrashPlan needs to service equipment/archives in such a way that prevents existing backups from being available for a period of time” is what took me to this blog!) …

      I do feel better knowing that you did have your archive restored though .. but you are right in everything you say about this breaking confidence .. and CrashPlan’s latest response was simply crap :S

    3. Just to add to my previous comment ….

      I never replied to CrashPlan after my initial ticket to them, and them telling me what I advised above.

      They automatically extended by subscription by 3 months.

      And a couple days ago I was sent an update to my ticket, advising that a senior tech is migrating a server that contains my archive and that it should be reconnected to the cloud shortly. A few hours later, I received a confirmation that the migration had been successful.

      Sure enough, my CrashPlan Tray tells me the backup is complete and I can access all my files via the web or Restore function.

      This did go some way to alleviating my fears. Though they absolutely should warn users about temporarily disconnecting a customer’s archive beforehand.

  24. […] Backblaze, or Crashplan Vs Mozy, or any other services, please be sure to read my ongoing saga with Crashplan losing my entire backup archive. Cheers! […]

  25. The problems discussed above seem to be with the personal Crashplan offerings. One would think their business software and support would be a little different.

    One would think.

    I’m a small business computer consultant with a few clients that need an off-site backup solution.

    I recently ran some very simple tests on the Crashplan PROe Enterprise trial version and it seemed ok. Someone I know works for one of their very large clients and the software seems to have worked for her. This experience, plus a lot of Googling led me to hope their software and support would be fine.

    So, I recently purchased a 5 client PROe license from them for about $345. This license just lets me use their Crashplan PROe software with my hardware and bandwidth, and provides no storage on their servers. The very first client computer I installed it on had problems. Crashplan did a simple test folder backup to my server, and then would not backup any changes to files in that folder, and would not backup additional folders selected. Then my in-house test client computer failed backing up.

    I went through all the documents I could find; briefly tried to find a clue in the log files; and ran some Google searches. I found nothing even remotely useful, and then had no idea where to look for these issues. I suspect the problem is caused by a simple issue on the client or server end, but I can’t find it.

    I tried calling their support line and choosing the Pro/Enterprise queue and got a voicemail system after a few rings. So I then submitted a trouble ticket with some log files. About 24 hours later, I got a reply from someone who said it was being escalated to Tier 2 support, and I would hear back soon. About 24 hours later (now), I still have no solution, and no idea when I’ll have a solution. I tried calling their Pro/Enterprise line again and this time the system said the mailbox is full.

    I started thinking about what would happen if I had a client who’d lost everything in some type of situation and was depending on my Crashplan backup of their data, and I had some problem restoring it. How exactly would I explain to this client who is losing business every day that I may or may not know in the next few days, or even weeks, whether I can get them up and running again?

    At this point, I decided if this is “Enterprise” level support, I don’t want anything to do with this company, so I tried calling their Sales line to see if I could arrange for a refund. This was a surprise – even their sales line’s voicemail box was full!

    I’ve just now tried requesting a refund by sending a reply using the support ticket, and cc’ing their information@code42.com address in case it gets read occasionally. Let’s just say I’m not expecting a quick resolution to this situation.

    1. Wow, this is pretty terrible.. but seems consistent with the rest of users experiences, unfortunately.. Let me know if you hear back from them and how they resolve it with Entreprise clients like you…

    2. It

    3. As much as I hate to do this, I finally gave up on Crashplan Enterprise, at least for now. Soon after the initial tests with the new version, I ran into some more problems. I just couldn’t get it to work in a predictable way, and couldn’t find documentation to explain why it was doing what it was doing. I wasn’t about to go through the “wait a week for an answer” support queue again. So, I gave up and asked for a refund.

      To their credit, the refund was relatively quick & easy.

      Perhaps in time I’ll revisit Crashplan Enterprise, as there really isn’t anything quite like it on the market that I can find.

    4. See, this is what’s so annoying about this. Here CP is losing a customer that probably could have been retained by simply responding to criticism. A strong, proactive response, acknowledging the problem and spelling out a verifiable action plan for solving it. Might have kept Jack, and would certainly retain some percentage of customers like Jack.

      This thread provided an opportunity for Crashplan to establish itself as a responsive, attentive company that understands and respects the needs of its users. Instead they look secretive and out of their depth.

      This stuff is really not hard. Responding to these kinds of things is essentially free. I know you are reading this (yes, you) – if you think this thread will just evaporate on its own, you’re crazy. Respond! Egads. Oh, and while you’re at it, I want my promised 3 months of free service – though my data was restored, my account has yet to reflect this change.

  26. Jeffrey,
    All to often I hear horror stories about trusted Online Storage companies loosing peoples data. Let us not forget what happened to Carbonite in 2009! At Stage2Data we have never lost any data in over 15 years. Please feel free to send me an email and i will show the difference experience makes.

    Stephen.Pyott@stage2data.com

    [MOD edited to consolidate Stephen’s email address into this comment, after he added it himself in a subsequent comment]

    1. Hi Stephen, thanks for your comment on my blog. Would love to hear about how you can compete with crashplan’s offerings. I know a lot of people feel stuck with CP because of their price and unlimited space/retention… me included. So, what else is out there? What should I switch to?
      -Jeffrey

    2. Have you seen s2d’s prices? Wow. They may be perfect but completely unaffordable.

    3. What happened to Carbonite in 2009? I’m using them for offline. Right now I solely use crashplan to back up to friends’ computers.

  27. Esteban

    Found your blog via some google searching. Very interesting read. It’s been a couple of months now; out of curiosity, has your experience with Crashplan improved or what backup service are you currently using?

    1. After Crashplan sent me the 3tb seed drive, I seeded all my data to them, and they loaded it into their system. It took a few days for the whole thing to sync up, but now that it’s synced, things have been running smoothly. No major issues for me, besides random rescans. So far I’ve been able to reach all of my data.

      I have looked at lots of other services, but haven’t found any with comparable features/price – if I found a service that could compete, I would very likely switch over. But currently crashplan has all of my data (about 2.7 tb), and so switching would take a considerable amount of re-uploading…. hopefully something better will come out in the future. Let me know what you find..

    2. Esteban

      I’ve just been looking at Crashplan vs Backblaze. I was leaning Backblaze until I read about the 30 day thing for external drives on their website. That doesn’t seem good for a backup service and was kind of a deal breaker for me.

    3. Yep, that was my issue too – they get rid of your data if it’s not seen by the system for 30 days. Since I travel a lot, and often have gaps of more than a month when I’m not able to connect to my external drives, this was also a deal breaker for me. I wish they had a longer retention time – I’d actually even pay a bit more for that – say, an extra $2/month for 1 year retention, or something?

  28. Just had my issue sorted out by CrashPlan’s team.
    It took them around two weeks to process my request and help me out but now everything is up and running.
    I’ve been running CrashPlan on Mac OS X Lion and the source of the issue was (somehow strange and sudden) incompatibility of my system’s version of Java Log4j. The solution was to replace log4j’s jar located at:
    /Library/Java/Extensions/log4j-1.2.13.jar
    with CrashPlan’s one:
    /Applications/CrashPlan.app/Contents/Resources/Java/lib/log4j-1.2.16.jar
    Though I’m not sure why they didn’t put their on CrashPlan’s classpath since they already include the jar

  29. Dheeraj

    Just one thing to all the people thinking of shifting back to Backblaze or some other service.
    I just finished my first month of service of CrashPlan+ and am happy with it, though this specific page has scared me a lot.

    I don’t have anything personally against Backblaze or anything like that but let me tell you Backblaze throttles your backup, not traditionally in the sense though.
    Backblaze has a maximum limit of 10gb per day (if you carefully look in the settings), I tried disabling that limit but couldn’t get through it (but maybe it was because I was on a trial, someone please confirm).

    Most backup services are shit for us heavy data users. Just as someone pointed out the prices of storing data on other services, I think the unlimited services do a great job by saving us money over data size.

    Other than the Backblaze daily limit, all the other unlimited backup services are almost the same, equally boring (of course, Carbonite throttles).
    Even if something is almost perfect, you’ll notice one thing – data retention:
    Very few unlimited services actually backup your external hard drives.
    Even if they do, they require them to be plugged in one every 30 days, or else they’re considered deleted. And all deleted files are removed from the backup after a month.

    Crashplan here is different, it allows you to backup external drives forever and same with deleted files, it doubles as an archive (I’m pretty sure someone from CrashPlan will be very unhappy about this, but yes, finally it stores all your files).
    This is something which almost no one else provides (except two services – I’ll write them down below). So yeah, I agree CrashPlan+ losing your data is shitty and unacceptable but then nothing can beat the price and the data retention feature. So consider this before shifting from CrashPlan.

    The only two ‘backup’ services which can store files forever are http://www.onlinestoragesolution.com (29$/year) and http://www.unlimitedfilestorage.com (49$/month).
    Personally, I’ve tried the first service and its lost my data quite a few times (almost seems like they do it willingly and purposefully) and the second one is too expensive and too simple for the price but maybe it does the job well – storing an unlimited amount of files at the same time backuping them up.

    So all the people looking for unlimited data retention, consider these points, consider CrashPlan again. And for people looking for an archive, look for a file hosting service which will not delete your files as long as you pay the subscription (not a viable alternative, I know, but still works out).

    Maybe you can check LiveDrive (if you do not need data retention forever – same 30 days with LiveDrive) and its resellers which offer backup at extremely cheap prices (eg. Zovo, Backeverythingup).

    Dheeraj
    (Not affiliated with any of the above services)

  30. Dheeraj

    I was wrong, Backblaze does not throttle, the limit it shows on the settings page is just the maximum amount of data that can be transferred based on the last upload speeds of the uploading files.

  31. Jeffrey, a few months later, but I’m now in the exact same position. “Archive Maintenance” ran on 3/25 and right after that my 2.4TB of data was gone. I’ve had a ticket open with CP since then, and a week ago it was finally escalated. Since then I’ve heard nothing from CP, despite repeated requests for a status update. (At least tell me you’re working on it!) I haven’t tried phone support yet, but based on your experience I think I need to.

    Thanks for publishing your experience and giving others in the same boat a place to vent! Like you, I really don’t want to switch providers (again), but until they tell me otherwise, I’m not sure if I’ll ever get my data back.

    Geoff

    1. Wow, sorry to hear this happened to you too. So, did you actually lose any data, or are you in the same position as me in that your backup is lost, but you still have your primary copy under your own care? Either way, it’s completely unacceptable that this happened. Let me know if you ever hear back from them about this..

    2. Same as you – multiple backups of thousands of NEFs that I don’t want to lose – but two of the sets are onsite so I really want the security of a CP backup.

      Interestingly, one of the backup sets is a local CP backup to a dedicated disk that looks like it’s hosed too. But who knows, without more help from CP there’s just no way to see what’s really there and what’s not. That’s probably the most frustrating part of this experience – the not knowing. I’m a patient guy, but after a couple of weeks of being in limbo I’m feeling less and less comfortable each day.

    3. Just a quick update. Not sure if Jeffrey’s tweet was the nudge CP needed, but I I finally got a response on Friday. One of CP’s techs (Brad) is working with me on the problem and has been tremendously helpful. My problem is different than Jeffrey’s – my files are still intact at CP – it’s the CP client on my machine that’s causing all the problems.

      With Brad’s help I was able to rebuild the CP cache and now I see my 200,000+ files showing on the main CP dashboard screen as being included in the backup. (It used to show just one file being backed up, despite hundreds of thousands still in the selection lists.) My problem now is that I can’t keep the CP dashboard running long enough to see if it’s actually doing the backups. Every time I open it I see it “backing up” to my local CP disk, and “scanning” my main large file volume, and then when the scanning gets to about 202,000 files the UI simply goes away. Launch it again, same thing. The CP service is taking lots of CPU right now and CP claims it’s doing a local backup, so we’re making progress – just not sure if it’s working yet or not.

      Brad’s going to continue working with me this weekend so I’m hopeful we’ll have this resolved before Monday. I’ll drop another update when I know more.

  32. I’ve just tweeted directly at Code42 CEO Matthew Dornquast, asking for some sort of response to all the readers of this blog post and Crashplan customers who have had issues. Full details are at the bottom of this blog post (just above the start of the comments section), and you can see my tweet here: https://twitter.com/Jeffzilla/status/188011324305969154

    Also, on a technical/blog maintenance note – I’m going to keep appending my updates and notes to the bottom of this blog post, instead of creating new blog posts for each update. I think that keeps all the information related to this topic consolidated under one url and easy for people to read in its entirety.

    -Jeffrey (this blog’s author)

  33. A James

    I’ve been using CrashPlan + paid for almost two years. Today, I just checked my Central Backup Archive and nothing exists before 18 March 2012. It’s all gone. This isn’t the end of the word I still had the backup for everything after 18 March.

    Until this afternoon…

    My CrashPlan Central backup has been completely wiped. I have one 90KB (kilobyte) file backed up. I have no idea which file that is.

    When I look at the files I should be backing up they are all ‘ticked’, yet they are not being backed up, nor does the archive exist.

    I can’t even start a new backup because CP ‘thinks’ the relevant folders are ‘ticked’ as above.

    So I’m currently unprotected, have nothing backed up, have no backup archive, and I’m unable to back anything up.

    And, I only found this out by accident — potentially, I could have gone weeks without discovering this. I’ve submitted a support ticket, but for the time being there’s nothing I can do my end.

    If my HD crashed I would lose everything — though I’ve just thrown everything in Dropbox for the time being.

    This is not really acceptable.

  34. I am using both Backblaze (1.75yrs) and Crashplan (2mo) for backup, so all my eggs are not in one basket. I am often having “backup destination was disconnected” incidents shown in the Crashplan history log, and then too many hours offline between backups restarting. I am seeding over 2TB online, which will take over 3 months, specially due to these long interruptions. But, for the price, I will hope for the best from these companies, which seem to be sincere in providing good service over the long run. Backblaze has been fairly smooth sailing so far , but Crashplan offers that extra retention. It will be interesting to see what happens if I totally lose my local data and need restoration. I do have local manual backup copies to work from first, even a 2TB ioSafe. Fingers crossed nothing ever happens.

    Backing up those 1080p videos and 5mb photos sure takes a lot of space. And with the NSA reviewing all the files as they pass over the internet, it is amazing the amount of data that can be handled.

  35. Ashley

    I am also finding that the destination is disconnected with a rather annoying regularity. Crashplan actually moved me to another server saying it should be faster but it seems no different from before really so I suspect they are all being run at the very limits of their capacity.

    Regarding the files being reviewed as they pass over the internet the Crashplan app is supposed to encrypt the files before they leave your computer and it would require a password to decrypt them. I think many users would be upset if they found that was not the case.

  36. A James

    As a follow up to my previous post about my archive, I’ve finally had correspondence from Crashplan. My backup has ‘reappeared’ and is no longer just 90KB — which is good, but it doesn’t inspire confidence.

    However, my entire backup archive and all my versions have vanished (from before 18 March) and are not recoverable. It’s good that my files are backed up, but if I one was corrupted I wouldn’t be able to ‘go back’ to an uncorrupted version. Again, this does not inspire confidence.

    I’ve used Crashplan for several years on a Mac, Ubuntu, and a PC, with no problems. I also suspect they’ve suddenly become too popular and have taken on more than they can reliably manage. It’s a shame.

    Although I don’t want to look elsewhere I am going to have to, as confidence and trust are the main characteristics of any backup service.

  37. After reading the comments of this post, I understood I am not the only victim!
    I took me 4 weeks to complete a 50GB backup on CrashPlan Central. During the initial backup, I noticed that the process was very weird: there were days that the backup was like 70% or 80% complete and the next day it went back to like 35% or 40% and started backing up the same untouched, unmodified files! I am SURE I was not changing 10-20GB of files and I am also sure they were simply losing track of what needs to be backed up online because my local backup archive (completed long before the online backup) was not behaving like that (i.e. it was not re-backing up those files)!
    After that things were running fine for a week, until today when the backup software re-scanned my files and started the online backup at 34%!!! So, I wanted to see if things are even worse and tried to restore a file from the online archive (restore from local archive was tested and is OK). Chose yesterday (April 15th) as the date of reference and the dialog box says

  38. A James

    I’m still to hear from Crashplan about my missing backup archive, I appreciate it might take time, but their customer service didn’t used to be like this, perhaps confirming my suspicions that they have grown too quickly too fast over the last year or so.

    The backup archive isn’t that important for me, and I appreciate they are not an archive service, but the loss of data leads me to question their reliability.

    Personally, on my Mac, I’ve switched to Backblaze as of yesterday, and just taken out a paid subscription. They uploaded around 23 GB of my data in around 10 hours. They also seem like decent guys running the service, judging by their recent reddit appearance (which gives some useful info about their service):

    https://www.reddit.com/r/IAmA/comments/rhrt4/we_are_the_team_that_runs_online_backup_service/

    So long Crashplan — it was nice while it lasted but now it’s over.

    Incidentally, on a mid-2011 iMac 21.5 ins i5 8GB Ram, Backblaze seems to run faster in the background, with less systems resources. My computer is once again silent.

    1. Thanks for your comment, and link to the Reddit AMA . I just left my own comment on that thread, asking for a response to the comments from all of my blog readers. Heres’s my comment: https://www.reddit.com/r/IAmA/comments/rhrt4/we_are_the_team_that_runs_online_backup_service/c4cz7xx

      Hopefully this comment on Reddit will help get some sort of response out of Crashplan.

    2. kadajawi

      Backblaze seems light on the hardware because it does not do much. If the metadata of a photo is changed, it has to be uploaded again. Do some new keywording and all your photos may have to be uploaded again! You can also only deselect folders that you do not want to have backed up, which can be quite troublesome too.

      Also if you have many files it will get slower, and if you have too many files (and too long file names!), move around stuff a lot etc. then you will eventually have the computer crawl to a halt, and when the database gets over 1 GB you’ll have to start all over again. As in you can start backing up from start. That has just happened to me, and I only have 900 GB of files backed up (been with them since the end of 2008). They say they are working on it, but for me it is either start again with Backblaze or move to Crashplan, who seem to have the better, or at least more advanced software.

  39. Another week, another update… Sigh.

    After my initial wait I had finally started to make progress on my issue. As I mentioned above, it looked to be a local cache problem that was preventing the backups from populating correctly. We got that resolved, but I have been unable to get a backup to run since then. Crashplan starts the backup and then just disappears once it’s through scanning the files and starting the synchronization. Re-launch it, same thing. I sent more log files to Brad, but it’s now been a week with no response. And getting email every three days from Crashplan telling me my backup hasn’t run in three days isn’t making this easier. I’m going to start calling and bugging people until this either gets fixed or I get my money back. SO FRUSTRATING!!!

  40. Ashley

    I was making reasonable progress for a about a week but after switching off the computer for a couple of hours yesterday progress has been snail like and I’m having great difficulty connecting to Crashplan central. This happened yesterday and now today again after leaving the computer switched off for an hour or so I’m waiting hours simply to reconnect. It’s not good.

  41. Just to be fair, I got a quick response from CrashPlan support this time. Followed their instructions, rebuilt my local cache for CrashPlan Central and now I can access again my previous backups. That said, it’s not normal for their indexing mechanism to get corrupted out of the blue… Judging from this last glitch and the previous ones, it seems like they are moving files among different storage spaces and the desktop client cannot follow with the necessary adjustments of local indices.

  42. Just a quick update: everything with Crashplan seems to be working fine now; I keep a pretty close eye on it, as you can imagine, but I’m fully backed up, and it now only runs 12-9 am and is able to keep up with no problems, so far as I can tell. So, fingers crossed, it looks like I’m out of the woods.

    At this point I’m almost more worried about their apparent lack of responsiveness than I am about the technical side (though obviously I have my concerns there as well). Still haven’t been credited the promised 3 months of additional membership, but the fact that they haven’t replied to a direct request for comment is very worrying. If they go under because of bad communications my backup goes poof (again).

    I’ll be sticking with them for the moment, but I’m not sure what I’d tell a friend looking for a backup service (and I’m constantly preaching to my friends about the importance of transparent online backup). So to the Crashplan people I’m sure are reading this: you really need to start responding. I’d guess you could have avoided 95% of the bad publicity and bad feelings with a strong, proactive response.

    1. Ashley

      The lack of response bothers me as well. It’s bad enough when they take a week or more to respond to a support question when there is a problem but I’m really perplexed why they wouldn’t have come here to try and deal with some of the issues raised.

      Frankly if they are having problems dealing with the volumes of traffic and signups I’d rather they raised the price by a few dollars for new customers to ensure it made good business sense going forwards.

  43. Hey CrashPlan people

    I am a bit annoyed at my current online back up provider, and during research for a new, your company came out in the top 3.

    …not anymore.

    To be reasonable, I have tried many services, and no matter the price, there have been problems, and when it comes to my files, I need someone I can really trust. This company doesnt seem to exist. The better looking ones all got hidden rules about deleting files.

    If you are looking to back up files, well, buy 2 external hard discs, leave one at a friend/family member/your job, switch the 2 every now and then. That seems to be the only safe route.

    Thanks for your post, Jeffrey.

    1. While most people are talking about problems with the “cloud” side of things, we need to remember that with CP we do have the “backup to many” option and don’t have to resort to swapping offline drives every once in a while etc (not there is anything wrong with that).

      Not everyone is going to be able to take advantage of these options, however they are there and available.

      In my case I have multiple sources (sites, ie friends, family) backup to my NAS (and others where they also have one), as well as “friend” to “friend”. Obviously you need to prioritise your sets and not backup the world when doing so but for important things like documents and photo’s it works really well.

  44. I fear CrashPlan just lost my data, I’m waiting for support to respond.

    However, I feel no backup service can be counted on 100%, there is too much room for error to cause data loss. I had formerly used another solution, and the developer discovered several years in that he had an error in the code.

    For these reason, I do lots of different types of backup, and cross my fingers. Even if CrashPlan did lose my data, I still may stick with them, depending on the explanation and resolution.

    1. CrashPlan did not lose my data, instead it was just the interface that made it look that way, as the application was doing a complete resync of the data. They should definitely improve the UI to differentiate back-up vs. sync, but oh well…

  45. Simcha

    It seems to me that the core of all of these problems is that crashplan does not have redundant storage of backed up data.

    Other (much more expensive) services maintain multiple copies of your backup distributed across multiple data centers.

    Crashplan maintains a single copy of your data. As a result, it something happens to that copy, your backed up data is lost. And since crashplan doesn’t seem to maintain a backup of their storage array (not surprising given its size), data that is lost is permanently lost.

    A related question: does crashplan operate more than one data center? If not, one major fire will permanently take out all of their customers’ data.

    1. Wow, I didn’t realize CrashPlan did not have redundancy, but you seem to be right. I guess its (another) reason to have redundancy in each individual’s backup plan, not just trust one source to handle it all.

      Also, if you read the details of this data loss, it appears data corruption was the culprit, and it’s not clear if redundancy would have been any help in this case (data could be identically corrupt on several servers). There are lots of ways software error or user error can delete or screw up data on all servers.

    2. Simcha

      You are correct that having redundant data storage doesn’t help if you write bad data to the system. For example, if a bug in the CrashPlan client software were to send bad data to CrashPlan’s servers, this bad data would be replicated (if replication were supported) and therefor recovery would not be possible from any of the copies.

      However, the situation described above is quite different. As stated in CrashPlan’s announcement: “a storage node in one of our Minneapolis data centers experienced a hardware failure. While typically a non-event, additional human error escalated this failure into backup archive corruption”

      Since the data loss was triggered by a hardware failure (and then exacerbated by human error), having a second copy of the data stored elsewhere would have allowed a full recovery.

      A bit of conjecture based on CrashPlan’s description of the source of the problem: Their description of the problem sounds as if they had a drive failure in a RAID array. Normally this is a non-issue, since there is no data loss at this point and replacing the drive will restore redundancy.

      Until the failed drive is replaced and the RAID has had a chance to rebuild, the data stored in the RAID remains vulnerable. The failure of another drive in the array prior to the replacement drive being rebuilt will cause all data in the RAID array to be lost.

      If CrashPlan’s “human error” involved replacing one of the good drives of the RAID array instead of replacing the failed drive, this would be the same as having a second drive fail before the RAID has rebuilt its redundancy. The result of this human error would be the loss of all data in the RAID, instead of the RAID rebuilding its data redundancy using the failed drive’s replacement.

      If this conjecture is correct, total data loss was unavoidable since CrashPlan does not have redundant storage and apparently does not backup their data servers.

      As with all purchases, you get what you pay for. Since the fees charged by CrashPlan for their unlimited plans won’t even cover the cost of the hard drives needed to store a heavy user’s data, it is understandable that they don’t provide redundant storage (although it would be nice if they offered this as an extra cost option).

      While I feel that they should still perform periodic backups of their data servers, the volume of data requiring backup makes this expensive (but potentially less expensive than true redundancy).

      So, unless CrashPlan eventually offers redundant cloud storage for an extra fee, we shouldn’t rely exclusively on their copy of our data. Of course, this remains true even if they do offer redundant storage, even though the odds of failure are vastly reduced.

      In their defense, CrashPlan’s software is unique, since it supports multiple ways of performing off-site backups – we aren’t tied exclusively to their data center.

      The ideal backup plan should include a local, on-site backup, a backup to an off-site computer and a backup to the cloud.

      In the end, if any data were to be completely lost, much of the blame has to be placed on the user who failed to design a proper, multi-level backup strategy. At most, the loss of data stored in the cloud is a (albeit major) contributing factor to the total loss of data.

    3. That makes sense, Simcha!

  46. Ashley

    Crashplan has been working better for me lately and it finally finished the initial backup after nearly two months running virtually 24/7. I was actually encouraged to see that they had taken on a new senior executive and I believe they had a big cash injection at the start of the year, so perhaps it will all be OK. These reports of lost data however, are definitely matters of concern.

    1. Thank you for this piece of positive news. I’ve just been looking for an online backup solution myself for my photo collection (about 160GB presently, and only going to get bigger – RAW files and whatnot!).

      Ideally I would use Amazon S3 given the huge redundancy they offer but I found the price prohibitive given it is personal data (maybe I will store a small subset of near and dear finished albums on there still). If I was running a professional business I would do it without a second thought, but as a private consumer it can add up quickly.

      I had narrowed it down to Backblaze and Crashplan. My favourite was easily Backblaze due to the high level of openess and honesty present within the company (recent reddit AMAs for example), but the application just didn’t offer the functionality I was after. I only wanted to backup my photos and nothing else (I have a Windows Home Server backing up our PCs, I can live without some of our other data in a major emergency but losing all our photos would be heartbreaking). I live in New Zealand and bandwidth caps are a very limiting factor for me. In fact up until about a week ago online backup was simply out of the question, so it is absolutely critical I have the granularity to backup only what I explicitly want.

      This left me with Crashplan, which I initially felt pretty good about until seeing this post and comments. They do seem like they have put a lot of work into their product and are willing to help the end user. I can understand that support is stretched thin at the moment due to rapid growth. I hope they’ve learnt some valuable lessons from these events and can only hope they come out better for it.

    2. You should know that Backblaze also allows you to select exactly which folders you want backed up, although by default it is set to back up your entire “user” folder, excluding only applications, system, os, etc.

    3. Yeah it does but it is based around exclusions instead of inclusions which can make it quite tricky to work with. For example if I want to backup C:\Users\Brad\Pictures I would need to exclude all the other folders inside C:\Users\Brad\, rather than simply including the one folder as I can with Crashplan.

      I can understand their reasoning as far as playing it safe with the average user but I deal with corporate backups on a day to day basis so I’m happy to take responsibility for what is backed up. Unfortunately I just can’t get that level of granularity with Backblaze :(

    4. I agree with Brad, I tried a service that only allowed file selection by exclusion, and found some of what I wanted to do to be nearly impossible.

    5. kadajawi

      If you have limited bandwidth then forget about Backblaze. They do not do proper deduplication or delta updates… meaning if one bit in a <10 MB file is changed, the whole file has to be uploaded again. If it is bigger than that it sometimes doesn't need to upload everything, but still much more than you would have to with CrashPlan.

  47. Good Day everyone,
    Firstly let me thank everyone for all the various responses on this blog it has given me allot to think about.

    I just had a few quick questions that some of you may be able to help me with.

    I am based Hong Kong and have about 2tb of data to backup, I am more then happy to wait 2-3 months for it to get backed up… that doesn’t worry me.
    I have been reading about crashplan and backblaze as they seem to be the best options for me. I don’t think either have servers based in Asia, does any one know if they do ?

    Of course I am a bit worried about experiencing issues like some of you have had regarding losing your backups as I would hate that to happen to me as my seed time is so so long esp being out here in hong kong.

    Is anyone else using either backblaze or crashplan from asia with a large amount of data to backup?

    Actually I have a few friends all with 2-6tb of data that are also considering using one of these providers.

    Thanks for your help and once again a great blog here for people for finding out information.

    Cheers,
    Mike

    1. I live in Taiwan and have been using Crashplan for about 1.5months. My experience was generally good, but you need to be really, really patient if you want to upload that much data.

      It doesn’t really matter where their servers are located, because they have apparently been throttling my upload speed based on how much I have uploaded. It’s been gradually slowing down from more than 3Mbits/s at the start, to below 500Kbits/s after about 500GBs of data uploaded. At this rate I can’t see them backing up all my 1TB of data within half a year.

  48. Trusting your data files to a backup program written in Java is akin to building a skyscraper on a foundation made of tinker toys. Backblaze is a native application, written in C for both Windows and the Mac. I’ve been using it for 2 years on both, and couldn’t be happier with it.

    1. kadajawi

      The Backblaze software isn’t great either though. “Warning: Backblaze has stopped working properly on this computer! Your bzfileids.dat file is too large, please contact Backblaze customer support.” The result is that I have to upload _everything_ again. And I am on a slow connection, that would take at least 3 months. Also I have been with the service for 3 1/2 years, and never, ever, stopped uploading. It was never complete, due to meta data changes, screw ups in their software, etc.
      However I do love their support (very helpful and responsive) and their openness…

  49. I just switched to CP due to its features and so far so good. However, the low price was a real concern. Doesn’t it strike you that they should be charging sufficient to actually pay for the hardware they need? Since CP is the only visible/credible company without storage limits, their customer base must be skewed towards those with high storage needs. The only possible outcomes are:
    – increasing problems/company goes under;
    – imposition of data volume caps; and/or
    – significant price increase
    Practically all other online backup companies have done one or more of the above in the last year or two. The sooner they choose one or both of the lower options the sooner we’ll have confidence in them. However, I worry that they rely on their rapid growth and multi-year pre-pays to support their operations (ponzi scheme). As soon as growth stops, option 1 may be inevitable without a massive external investment (Accel Partners, this is where you come in).

    1. rove said “…CP is the only visible/credible company without storage limits…”

      What about Backblaze?