[Discuss] Syncing between Notes and MarkDown

[windingwind]

Hi,

These days I've received many requests about syncing/importing markdown files to a Zotero note, like #143 #141. This has been put aside for a long time and I'm planning to support the two-path syncing using a note2md(we already have)+md2note conversion.

The syncing system would automatically compare the note and MD file's versions and their modification time. The older one will be updated if it keeps unchanged since the last syncing. If both are modified, the user will be asked to merge them and choose which modification will be accepted using a git-diff style window.

Previous discussions about note2md syncing are here: #45

Welcome any suggestions!

[windingwind]

To avoid possible conflicions caused by the annotation/citation, there will be extra hidden text in the exported markdown file. Like this:

<!-- bn:begin:ABXZ1469 -->
comment (pdf link) (item link)
<!-- bn:end:ABXZ1469 -->

And the markdown block wrapped by these decorators will be regarded as placeholders while syncing.
Different MD reader/editors may have different behaviors for these decorators. I tested on Typora and Obsidian:

• Typora displays them in yellow and they do not appear after export.
• Obsidian displays them in grey in edit mode and they do not appear in preview mode.

[windingwind]

This is not in the 0.8.0 release. Annotations are exported in HTML.
No extra decorators.

[IpeTek]

Hi,
I like to use Zettlr as my (Zettelkasten) editor. Could you please check how the sync works with that.

Also, Zettlr uses a totally app-agnostic/future proof way of linking between pages: Double brackets with an ID of the type %Y%M%D%h%m%s, [[20201116150658]]. Would it be difficult to compile a Better-notes template that would include this identifier in every note? With the feature in place I could use both apps seamlessly together.

I just found Better-n. A great invention that makes Zotero so much better. Thank you!

[windingwind]

Hi, @IpeTek

1. BN’s sync is currently and will work in an app-agnostic way (rely on the Zotero App because the notes are stored in the Zotero database). The markdown format can be opened in any software; the links between notes are the relative path of the markdown files.

2. As far as I know, Zetllr does not actually sync to Zotero. It uses the exported JSON file to get citations and the BBT plugin of Zotero does the syncing of this file. So not sure what you want with Zettlr, the sync, or something else?

This BN plugin tries to do something different: sync the note itself, rather than the citation information. This allows users to take notes while reading papers in Zotero more easily and sync them to their familiar note software later.

3. Compiling a template of the current time is easy:

Template Name: [Text] Time

<p>${((date)=>`[[${date.getFullYear()}${String(date.getMonth()).padStart(2, '0')}${String(date.getDate()).padStart(2, '0')}${String(date.getHours()).padStart(2, '0')}${String(date.getSeconds()).padStart(2, '0')}]]`)(new Date())}</p>

The result:
[[202208242210]]

[davidoskky]

The citations/annotations in note cannot be modified, while their corrsponding MD segments might be modified or deleted

The only idea that comes to mind is to write the citations in a yaml header and then reference them in the note as [1] [2] [3].
In this way one can move around references and even delete them but you'll still have something in the Zotero note.

Not a really good way to handle this, but maybe it can give you an additional idea on how to handle it.

[windingwind]

@davidoskky

Thanks for the suggestion. Actually the citation information is stored in each note item as a string, having it stored in the markdown is a good idea.

[windingwind]

We are almost there:

I proudly announce that we now have a two-path syncing of Zotero note and MarkDown!

Currently, it's still a pre-release and I decide to test it for at least one week before we could make it a formal release. The pre-release version will not be updated automatically. Please install manually from https://github.com/windingwind/zotero-better-notes/releases/tag/0.8.0-1

How to Sync

To add a note to syncing list, export it to an MD file:

You can manage the syncing list from the menu->File->Better Notes Sync Manager.

The notes in the syncing list, along with their corresponding MD files, are automatically updated, with content, formats, and image attachments.

To stop syncing, select items in the sync manager and click Remove.

How it Works

I use the rehype and remark to build a syntax tree from MD/note and parse them so that they can be converted to another format lossless.

The metadata of annotations is stored as raw HTML in MarkDown so that they can be converted back to notes. The metadata of images in the note is saved to the image alt ![IMG META](IMG URL). Do not touch them.

About Diff-Merge

If you happen to modify both the note and MD file, the plugin will try to analyze the changes and merge them automatically (like what Git does). However, it is possible that you are required to merge them manually.

Pick changes that you need from the left pane and the result will be rendered in the right pane. The changes on the raw note are shown in the center pane.

To confirm, click Finish. To skip, click Skip. To cancel the syncing of the note, click Unsync.

Note that if you skip any merge requests and the auto-sync is on, you may be required to do this again later.

Have fun with the new feature and please let me know any suggestions/bugs!

[windingwind]

The 0.8.x is still a pre-release, so please be careful not to sync any important data. After it's stable and passes the test, we could trust this new feature more.

[bepolymathe]

Seems to work fine for me 👍🏼

[IpeTek]

Congratulations! Hopefully will have time to test.

[windingwind]

About YAML header:

There will be a new template [ExportMDFileHeader] to generate this header. Currently, I add the note's parent item name, collection names, and tags.

The version, library ID, and item key is controlled by the plugin and cannot be removed.

[windingwind]

During the 5 pre-release versions, no severe bug is reported.
I think it's stable enough to be released. Now we are moving to the 0.8.x.
The syncing feature of earlier versions will no longer be maintained.

https://github.com/windingwind/zotero-better-notes/releases

[IpeTek]

Works perfectly! What a wonderful tool.

[ycwang-hello]

Awesome!!

[DedZago]

Thanks for adding this wonderful feature, I might have a suggestion to add.

Since my workflow involves using neovim for fast typesetting of math, I would greatly benefit from having an "Open in External Editor" button for a synced Zotero note (and, maybe, also a keyboard shortcut) to open the corresponding .md file in a predefined external editor?
Much like Zotero allows an Open PDFs using option in the Edit > Preferences > General tab.

Thank you very much.

[windingwind]

Yes. Thanks for your suggestion!

[DedZago]

Also, do you know whether it would be possible to batch export/import notes for portability across devices on cloud sync platforms?
I have found that I needed to manually export ~50 notes one by one in order to use it as a note-taking workflow on external files, and I expect I would have to to it again each time I log in from another device.

Perhaps setting a path for finding synced notes might resolve portability issues?

[windingwind]

Select multiple note items in library and click 'export notes and subnotes' in right-click menu to export in batch.

Batch importing is not available now, but it's planned.

[citadel-lewis]

Hi, how do I change the MD file name template so that the file name for a note is not changed at all? So say I've got a note called "This is a test note" it would be exported as This is a test note.md . At the moment I've changed the template from:

${(noteItem.getNoteTitle ? noteItem.getNoteTitle().replace(/[/\\?%*:|"<> ]/g, "-") + "-" : "")}${noteItem.key}.md

to

${(noteItem.getNoteTitle ? noteItem.getNoteTitle().replace(/[/\\?%*:|"<> ]/g, "-") + "-" : "")}.md

which gives me This-is-a-test-note-.md ... I can't figure out what to remove just to have the file note the same as the item note (but maybe keeping the substitution of '-' for problematic characters).