1. The problem
Lately it seems that many note managers with "infinite" tree structure are choosing a block model (where each paragraph is an entry in the DB), instead of a document or file model.
| Blocks | Documents |
|---|---|
| Notion Workflowy Remnote Dynalist Roam Research |
Evernote Obsidian Bear app |
If you find any errors in the table, please let me know.
We have been developing an app very similar to Notion for 8 months now, also using the block model, but we are considering making a radical change and switching to the document model. The structure of our blocks in MongoDB currently looks like this:
_id: "61fd3ede7f6d2cc7a53ca669"
children: Array
0: "61fd3ee87f6d2cc7a53ca66b"
1: "61fd3ef37f6d2cc7a53ca671"
2: "61fd3ef77f6d2cc7a53ca673"
backlinks: Array
type: "bullet"
parentPage: Array
_id: "61fd3ede7f6e2ccra53ca664"
userParent: "german-jablo"
permisionParent: "edit, comment, read"
parentParagraph: "61fd3ede7f6d2cc7a53ca668"
content: "<p>This is a paragraph</p>"
isCollapsed: false
createdAt: 2022-02-04T14:57:34.280 00:00
updatedAt: 2022-02-04T14:57:59.585 00:00
Many pages talk about the differences of both approaches (example) although in a very vague way, so we decided to open this thread to find a more scientific answer to the question.
Features of our app
| app | Blocks that can be opened as documents | Blocks that can collapse or expand their children |
|---|---|---|
| Notion | Page type blocks | Toggle type blocks |
| Workflowy | All | All |
| Evernote | Documents | None |
| Our app | Page type blocks | All others |
Our app has two types of "blocks". The page type (which, like in Notion, can be inserted into any note and generate a document "inside" the current document), and the rest of the blocks, which are equivalent to the "toggle" block type in Notion (i.e. they can be collapsed or their nested children can be expanded).
2. What we have tried
In trying to answer our question (which DB model would work best for our application), we've realized that the answer is probably "it depends". Perhaps both models have strengths or weaknesses in different types of operations or situations. That is why we formulated this comparison table describing how we believe the performance of both models would be for each of these operations.
| Operation | Blocks | Documents | Apparent Winner |
|---|---|---|---|
| Fetch the contents of a page | Find all paragraphs in the DB. | Search the document in the DB. | Document |
| Render the content of a page** | Build the tree from the paragraphs recursively. You can omit the children of paragraphs whose isCollapsed property is true | Render the document | Document |
| Update the content of a paragraph in the DB | Only the modified paragraph is rewritten | The whole document is rewritten | Block |
| Alternatives for rendering very large documents * | Blocks can be fetched or rendered as you scroll (as Workflowy does), or as you expand child paragraphs that were collapsed. | I thought that Grifds could achieve similar behavior, breaking the document into smaller chunks and bringing them in piecemeal, but it doesn't support updating an individual chunk, or even the entire document. It could also corrupt an HTML by splitting it into binary format. | Block |
| Import or paste content | In addition to converting the clipboard to HTML and/or sanitizing it, you must set up paragraphs with tree structure recursively. Note: Roam Research e.g. supports importing in json format, but generally users do not handle this format beforehand. | Only convert the clipboard to HTML and/or sanitize the clipboard | Document |
| Copy content** | Clipboard must be sanitized and/or transformed | Correct by default** | Document |
| Real Time Collaboration | At document level, could use some tree-based (Json) library like Automerge, or combine with some CRDT library for paragraph level. | Could use tinymce solution. | Tie? Both seem to have their advantages and disadvantages. |
*Render very large documents: Most users probably do not use notes larger than 250 kb (considering that multimedia files are referenced in a separate collection). Still in the document model the question arises: how can we load, render or edit large documents in manageable chunks? One idea we came up with is to split HTML documents that reach large dimensions into portions of a certain size in kb, instead of splitting them into paragraphs. (It would be like a kind of Gridfs that allows you to modify the file in parts.) Could this be a good idea?
**Should the DOM be nested? In order to be able to collapse or expand nested child paragraphs, note managers with a block model structure the DOM in a nested way (paragraphs are in divs, inside their parent divs, etc.). However, an alternative in the document model could be that when the user presses tab, only that block (HTML tag such as <p> or <li>) is assigned an attribute with a number less than or equal to 1, representing the nesting levels relative to the previous block. This way when you press tab to nest or shift-tab to un-nest, you only have to modify one attribute of an HTML element instead of many elements; and the DOM stays simple, without having nested blocks.
Our conclusions
We believe that for each of the rows in the comparison table, benchmarks could be done measuring the performance of both models. Other people have done something similar here and here, comparing the performance of note managers using both models. The problem with those tests, is that it is difficult to draw an accurate conclusion about the goodness of both models. Obsidian uses documents locally, so you don't have to sync notes. Roam Research is a very new and poorly optimized app. Standard Notes encrypts notes locally. In other words, it's not always apples to apples.
And even if tests could be done, we believe that the answer may even depend on how each user uses the application. Suppose user A usually organizes his notes in long documents using paragraph nesting (to collapse or expand them). On the other hand user B usually organizes his notes by creating new documents within documents. It is likely that a block model based manager would work better for user A while a document based one would work better for B.
So, we tried to push our doubt as far as we could, but we are still not sure of the answer Which of the two models do you think would offer better performance for our app and why?
CodePudding user response：
Looks like you've done your homework. Database modeling is sometimes a bit of an art rather than a science. I think that with both models, you can achieve good performance if you optimize them well. So I would recommend that you go for the one that requires less work. Since you've been working on the block model for 8 months already, that's probably the best option for you.
CodePudding user response：
Love how thorough you thought about the design of your application. I just want to add some suggestions you might look:
- Obsidian is using a hybrid approach. It started document-based, but now supports block links and embeds while still being super-fast. From all programs tested in my benchmark you linked above, Obsidian was the fastest.
- One of the most important functions of a notetaking tool is effectively searching the probably thousands of notes. I created an amazingly simple test (the "Spaghetti Parmesan"-test) where all block-based approaches currently fail. It is about searching for two ingredients (spaghetti and parmesan) in a recipe. When both ingredients are in different blocks, all common block-based applications ultimately fail to find the recipe. You can read more about this here. I also tried to start a discussion with some of the authors on Twitter but failed to get any serious results. If you want to continue the block-based approach, you might start designing a search algorithm that can handle search terms spread about blocks (if you haven't yet). I tried to outline an algorithm in the thread linked above, but not sure if it will really work with hundreds of thousands of blocks.