The database is too fragile for (supposedly) long-term data

⁉️ Ask us anything!
#1

Hello all,

First, I’ll start with something positive: the app itself feels very polished and professional looking, and when it works, full-text search on my visited pages has been a godsend during my research sessions!

However, it hasn’t been all too peachy with Memex for me, and I’m still evaluating if it meets my needs because of that. I spent a couple of evenings turning all my old better-onetab (a practically abandoned extension) groups into Memex collections, but after a brownout, I had lost it all… or well, I would have lost it all, were it not for an external backup (completely external to Memex and a backup of my entire Chrome profile altogether) which I had lucked out on. I was a bit too shaken to dig out any error logs so I just contented myself with overwriting my old profile and waiting for the best. And indeed, all my data, save for a few groups I had made in the last couple hours, was still there.

In my opinion, charging for automatic backups when the app is still in such an unstable state is a bit preposterous. The user experience, save for the (unfortunately necessary, as far as I can tell) friction of downloading an extra app for backups has been pretty great, but none of that is any good if Memex ends up “forgetting” everything after “bumping its head” a little bit, and although it seems like it’s not a straightforward problem to solve given similar feedback you’ve received, I do think it’s absolutely vital to figure out going forward.

#2

If I may chime in, this is mostly a communication problem.

I had also lost my data once, not because of a bug in memex, but because I hadn’t installed the “Memex backup helper” app. The word “backup” in the app’s name is a bit misleading. The app is where you actually store your memex data, simply because the data stored in the extension are never supposed to be permanent. The browser could bail out the storage with no warning. Using the word “backup” for the app leads one to think that it’s a … backup. It’s not needed unless something goes wrong, which is not true. :slight_smile:

I would probably think of a different way to name the app without the word “backup,” and communicate how essential it is.

Like, maybe put a red badge on the extension’s icon until some storage mechanism is chosen (cloud or local) so the user knows their data won’t be persisted unless they configure their storage mechanism.

#3

@EntropiaFox

Such a bummer, we are right now in the process of making the database more stable.
What do you mean with “after a brownout”? Can you explain exactly what happened?

We are actually currently in a conversation to make the auto backup part of the free tier.

@AriaMinaei

I would probably think of a different way to name the app without the word “backup,” and communicate how essential it is.

Mhh actually it should not be essential to use it, theoretically. But the browser is a bit unreliable at times. We are right now in the mode of diving into some more leads we got recently around data ejection issues.
But until then it makes sense to communicate it as an essential component. Will take care of that for the next update.

#4

Sure, I can try to explain a little bit better. Power flickered while I was working (on a desktop PC with no UPS) which made the computer restart itself, and when I logged back in, I had lost the entire Memex database: first time I went to check the dashboard I was met by just a plain white screen, but then I restarted Chrome and I was able to get the extension working properly again, it was as if I just had a new database, none of my bookmarks were present or anything.

Instead of fiddling around too much with things (as I was in a hurry) I just decided to restore my entire Chrome profile from an external backup (I have incremental backups to an external harddrive with UrBackup every two hours or so) so I do apologize if there was any error log I could have submitted there which could have given a bit more of insight about what happened!

This would be greatly appreciated and I do believe it should come as basic functionality, as not everyone will have set up an external backup solution nor it should be deemed necessary to reliably use Memex as it stands right now.

If you wish to still have some degree of functionality tied to the paid tier related to backups, perhaps it could be customizing the interval on which backups are made while the free tier is locked to backups every 15 minutes. While at it, there should probably be a more visible warning in case the user isn’t running the backup helper.

#5

Whoa!!! This is a huge miscommunication… I had assumed that all my data is stored in the cloud by Memex and would follow me around to different computers. I just changed computers and wondered where all my data had gone. It was stored in the browser extension?!!

I had assumed that the download extension and google drive, etc were all ways that were provided for me to get my data out of Memex if I wanted it later… basically an export and additional layer of reliability.

What I am understanding from this thread is that storage of user data is TEMPORARY UNLESS YOU TAKE ADDITIONAL STEPS… yes I’m screaming. That is an extremely important bit of the mental model, and 100% at-odds with the whole idea of capturing valuable research data.

So yes new features and including in free plan are good, but the key here is the miscommunication: users MUST realize that their valuable research information is held only in temporary storage unless they connect Google drive, a local folder, or the local database. “Backup” is not the right word, the concept is closer to “Temporary” vs “Persistent” storage.

1 Like
#6

@Eric1 the app will live up to your definition of its name (backup) so long as you choose a folder that is synced to a remote server.

I consider backup a progressive attempt at increasing resilience of the data. There is no permanent backup. Eventually, after their cloud services are bought by China, users will need to pay for their data to get it back, and they won’t be able to afford this as their currency will be too weak :joy_cat:

So practically, having a local ‘backup’ in your filesystem is better than having it in the browser’s, highly temperemental, IndexedDB. But I agree, it is not good enough to be termed a complete backup. For that you should also encrypt them (this also seems like a backup in terms of increasing security of your data), sync the files to a remote server. You could just export them to a Cloud synced folder. Ideally though, this remote server/cloud should be one you own if you are not using your own encryption. You could have a server that is running say RAID5 for further redundancy in case of harddisk failure. Some open source tools to manage this are: Duplicity, rsync and syncthing, bittorrent share. There are others. On mac Arq Backup looks quite good. You could sync them in IPFS (a really robust solution so long as you pin it and have multiple copies on your devices), or copy them to an external harddrives/flash drives which you keep in EMF protected sleeves or cases (e.g. a steel lockbox - the poor man’s faraday cage).

So I disagree that backup is not the right word. So long as you choose a folder that is auto-synced to other devices or the cloud for increased redundancy, the data can be considered persistent and backed-up.

@BlackForestBoi I might say that it would be a good idea to further emphasise the necessity of ensuring the redundancy of backups (in human language) similar to what Polar have done with their backup service. Exporting to local files seems like a very necessary free feature. However, exactly like Polar, automated encrypted cloud or inter-device syncing can definitely be offered as a paid addon, similar to how you’re doing it now. Keep up the great work!

polar_business_model
polar_business_model.png2688×1992 485 KB

#7

Thanks @AUM for chiming in.

Indeed, we consider it a backup in the true sense of the word as the the indexedDB storage is working separately (and ideally very reliably)

However I echo the sentiment of making those local backups 100% free, and potentially add the ability to run a cloud backup and charge for it, and be more explicit about the need to backup data.

4 Likes
#8

Hello there! I greatly appreciate the agreement that, yes, this kind of feature really should be included as part of the “free” plan, just checking in here to see when it’d be planned for or if there are any showstoppers preventing it at the moment because, as I mentioned in my initial post, I do think it’s an incredibly important feature while you sort out the issues with browser database frailty.

#9

We recently fixed the bug that caused data loss on version upgrades in very rare cases, which were reported here. So we are 80% confident that it won’t happen again. (and hasn’t happened since to the best of our knowledge)
That was about 1,5 months ago, shortly after my last comment here.

We had another conversation in the team and came to the conclusion for now to keep it a premium feature as it’s with sync the only one that provides us revenue right now.
We also think, that if the tool is really useful to you you may want to pay and support its development, especially since we have product values that make it 10x harder to build software (offline first) and 10x harder to raise money (we have an investor profit cap and can’t be sold).

If it is not even worth 3€ to people, then it’s probably not the right software anyway.