Database not loading / sync, backup not working (Firefox, macOS)

🐞Bug Reports
#1
  1. Which browser, operating system and versions of those are you using?

Firefox 79.0, macOS 10.15.6

  1. What are the steps you took that lead to the bug?

Recently I wanted to access my database and noticed that it is stuck on loading. Last successful backup went through 5 days ago, currently it gets stuck on “Preparing uploads” and even cancelling gets stuck on “Finishing current batch…” Also I don’t see my phone in sync

Debugging the extension yields some information, the console is full of errors "UpgradeError Not yet support for changing primary key":

Uncaught (in promise) 
_e: Error
columnNumber: 1183152
fileName: "moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js"
lineNumber: 1
stack: "Y@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1183152\nr@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1184712\nQt</e.prototype.create@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1218849\ne@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1196970\ne/<@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1197312\ndt/<@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1195302\nWe@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1189718\nXe@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1190202\n$e/<@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1190569\n"
<prototype>: Object { … }
_stack: "OpenFailedError: UpgradeError Not yet support for changing primary key\nQt</e.prototype.create@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1218849\ne@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1196970\ne/<@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1197312\ndt/<@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1195302\nWe@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1189718\nXe@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1190202\n$e/<@moz-extension://911f7c5a-0426-8040-a89b-1a9d88cd1043/background.js:1:1190569\n"
inner: {…}
_e: Error: 
inner: null
message: "Not yet support for changing primary key"
name: "UpgradeError"
stack: ""
<prototype>: Object { stack: "", … }
message: "UpgradeError Not yet support for changing primary key"
name: "OpenFailedError"
stack: ""
<prototype>: Object { stack: "", … }

image
image.png1432×618 1.1 MB

It looks like this error is probably related to Dexie.js which, I assume, is a backend you use for Storex. I haven’t found this error in relation to Memex hence I report this bug (sorry if it’s a duplicate).

Also given the last time of successful backup, I suspect this issue appeared after upgrade to Firefox 79. In the release notes there is a mention of storage.sync changes but I don’t think it is related to this issue.

#2

Looks like the error is being reported to your Sentry account so you should have the necessary details already. :wink:

#3

Indeed we are already aware of this bug. Happened with a schema upgrade recently and some users are affected. Fix is already underway and will go out asap.

Thanks for the report!

1 Like
#4

Hey jvn. Thanks for the report!

You noted that your phone was not in sync after this.
What version of the mobile app do you run? As in platform and version? Mainly interested to know if you’re on Android and are a part of our open beta program.

Cheers!

#5

Hi! I have it on Android 10, app version is 1.6.2 and I am enrolled in beta versions on Play Store.

Mind that I still see the device as paired on the phone, I don’t see it in the browser.

#6

When you say “don’t see it in the browser” what do you mean? Its not on that list?

That may be because database access is impaired right now.

However, another few questions:

  1. Which version are you on in the extension
  2. Do you have auto-update enabled?
  3. Which version did you have last?
#7

Precisely. In the browser I see “No device paired yet”:

Screenshot 2020-08-06 at 10.39.08.png856×482 28.4 KB

But on the phone I see:

Screenshot_20200806-103705
Screenshot_20200806-103705.jpg1080×1697 45.5 KB

That is my suspicion as well.

  1. Version 2.3.0, last update July 31, 2020.
  2. Yes I do.
  3. Unfortunately I don’t know. I have updates enabled all the time, so I assume I had the latest version before 2.3.0. I have noticed the issue only 2 days ago but the automatic backup failed 6 days ago (from today). That corresponds with the date of the last update.
    Perhaps is there some way to get the version of the database through the JS console?
#8

Thanks for all the info so far! I have a few more questions:

  1. Do you remember roughly when you first installed this extension?
  2. Have you had the extension disabled at all recently? If so, for how long was it disabled?
  3. What is the version of your underlying Dexie DB: you can copy this command in the BG script console to get that:
    storageMan.backend.dexie.verno
  4. What is your underyling Dexie DB schema? You can run:
    storageMan.backend.dexie._versions[storageMan.backend.dexie.verno - 1]._cfg.storesSource

I’m hoping we can get this reproducible and figure out how to solve it soon.

#9

I have one more bit of code for you to try out in the BG script console. It’s a bit more involved, so I put it in this gist which you should just be able to copy and paste in and receive output for upon pressing enter.

The output you get will also be a bit more complicated than the previous ones, so great if you could copy and paste it all here in a code block or in a gist.

Let me know if you have any issues

#10

Thanks for looking into it!

Wow, that’d be long. I’ve looked into extensions.json in my profile directory and the "installDate" for the extension is 1541158778000 which corresponds to November 2nd 2018.

Not recently, perhaps for a few months last year but that’d be over an year from now.

21

{
  "bookmarks": "url, time",
  "tags": "[name+url], url, name",
  "pages": "url, hostname, domain, *urlTerms, *titleTerms, *terms",
  "visits": "[time+url], url",
  "favIcons": "hostname",
  "directLinks": "url, *_comment_terms, createdWhen, *_body_terms, pageUrl, *_pageTitle_terms",
  "customLists": "id, createdAt, isNestable, isDeletable, name",
  "pageListEntries": "[listId+pageUrl], pageUrl, listId",
  "eventLog": "[time+type], type, time",
  "notifications": "id",
  "annotations": "url, *_comment_terms, lastEdited, createdWhen, *_body_terms, *_pageTitle_terms, pageUrl",
  "backupChanges": "timestamp, collection",
  "annotListEntries": "[listId+url], url, listId",
  "annotBookmarks": "url, createdAt",
  "socialPosts": "++id, userId, createdAt, serviceId, *_text_terms",
  "socialUsers": "++id, username, name, serviceId",
  "socialBookmarks": "++id, postId, createdAt",
  "socialTags": "++id, postId, name",
  "socialPostListEntries": "++id, postId, listId",
  "pageFetchBacklog": "++id, createdAt",
  "clientSyncLogEntry": "[deviceId+createdOn], needsIntegration, sharedOn, [collection+pk], createdOn",
  "syncDeviceInfo": "deviceId",
  "templates": "id, isFavourite, code, title",
  "customListDescriptions": "listId",
  "pageListEntryDescriptions": "[listId+pageUrl]",
  "sharedListMetadata": "localId",
  "contentSharingAction": "++id, createdWhen",
  "readablePageArchives": "url, lastEdited, createdWhen"
}

Sure, your script seems to work fine. Here’s the output:

By the way, if you need any more information or want to borrow my browser for direct debugging, I am available for call or DM at Worldbrain’s Slack (as jnv) – today at least until 5 pm CEST, so feel free to contact me there.

#11

@poltak @BlackForestBoi Hi, any update on this? I kinda miss access to my data and I will be renewing my subscription soon, so…

Unless there’s further investigation I can help you with, would it be possible to somehow restore access to the database? Will reinstalling the extension and restore from Google Drive fix the issue? Or should I rather downgrade to v2.2 first?

1 Like
#12

I will put up recent findings here so it doesn’t get lost in Slack’s history. :wink:

Bug Reproduction

I have installed Memex when it was around version 0.10 back in November 2018. Shortly after that the database schema has changed:

So my database carries on the older version of the schema with auto increment on customLists table. This haven’t been an issue until Memex version 2.3.0. It looks like that the issue is caused by the underlying DB library Dexie.js.

I have reproduced the issue with fresh profile, and made a build of Memex v0.10.3 from October 2018 (which seems to be the last release before that schema change). You can find this version in my fork: https://github.com/jnv/Memex/tree/repro-0.10.3
I needed to adjust a missing dependency and remove Sentry reporting, but otherwise it should be the same as the original. You can find the ZIP with built extension in release assets: https://github.com/jnv/Memex/releases/tag/v0.10.3

It can be installed to Firefox Developer Edition (80.0b6), you just need to disable extension signatures check:

  1. Go to about:config (and void the warranty)
  2. Change xpinstall.signatures.required to false
  3. Go to about:addons
  4. Install the extension.zip, you can just drag & drop the file to the browser window.

Now the old version is installed. You can update the extension to the latest version (on cog button click Check for Updates) and Firefox should give you an option to update the extension.

After the restart the Dashboard shows just a spinner and I am getting the same UpgradeError Not yet support for changing primary key message in console (about:devtools-toolbox?type=extension&id=info%40worldbrain.io). And the customLists table has auto increment:

image.png1154×212 16.8 KB

Now for the next step I will try a reinstallation and restoring backup.

#13

So the good news is that restoring the data from backup effectively solves the issue.

The bad news is – unrelated to this bug – I was unable to restore my backup from Google Drive completely, because every time the server returns error 500 at some point causing the whole restoration process to fail. But looking at the GoogleDriveClient's code, it should be easy to add further retry attempts if the error is on server’s side. I will try to prepare a PR.