The Dat Project is a promising way to share “folders” of data — which is an attractive option for sharing research data and/or source code for more reproducible research.
I’ve been excited about the Dat project for a while, and lately I’ve been thinking a lot about the pros and cons of several approaches to data sharing/collaboration tools. I admire what the folks at the Dat project are doing, but I worry that their focus has recently moved away from research and toward “a peer-to-peer web”. While that isn’t inherently bad, it does mean that their priorities no longer align so perfectly with what researchers need.
So here is my own wishlist for what I wish Dat could/would do differently to become a perfect tool for sharing research datasets (and maybe code). I’m writing it down mainly to help myself make sense of the trade-offs between the plethora of alternatives.
Offline Staging
The way a Dat is currently constructed, if you are writing changes into the “live” dat archive, those changes are immediately propagated to anyone else who is watching that archive. You could make a copy of the archive, try out your “experimental” changes in that copy, then “commit” to the changes by copying them back into the live archive when you are sure you are ready. The problem is that now you have to use twice as much space locally, and always be careful that you are working on the “sandbox” version, not the “live” one.
You could also work on the local version, then use dat sync to commit and propagate the changes to at least one peer (maybe a pinning service like Hashbase), then go back offline and do more work. Such a workflow would look like this:
$ mkdir mydat && cd mydat
$ dat init
$ # Now, add files to the Dat.
$ dat sync # this also shares!
dat v13.10.0
dat://ec930b[...]
Sharing dat: 4 files (180 B)
0 connections | Download 0 B/s Upload 0 B/s
Watching for file updates
Ctrl+C to Exit
Now you can’t make any changes in the Dat without having them be “live” instantly. So, you could <CTRL+C> to take down the dat, and maybe someone else is seeding it – or not.
Then you change some things, and do dat sync again when you are ready to make it live…
What I really would prefer is to basically “steal” the workflow of Git. Initialize, work, stage, commit, push, repeat.
$ mkdir mydat && cd mydat
$ dat init
$ # Now, add your data...
$ dat add . # Add everything to the managed archive
Then, use a “commit” to mark the changes that are ready to be shared with peers:
$ dat commit -m "initial data version"
$ # When you run `sync`, only committed changes are actually synced:
$ dat sync # A `dat push` to a pinning peer would be nice as well
When you make more changes, you “stage” them with add and then commit them to make them “live”. In my imagining, the dat sync operation starts seeding your committed changes immediately, and runs in the background to make your local machine an (almost) always-on peer. Then you can continue working — none of your local changes become “visible” on remote copies of your Archive until you’ve staged and committed them. There could be a dat status that would quickly tell you what version is “live” and what you’ve changed but not committed.
A Real Revert-able History
Dat archives maintain a history of every action that has taken place, but it isn’t a very useful history. Here is an example:
$ dat log
1 [put] /dat.json 39 B (1 block)
2 [put] /dat.json 157 B (1 block)
3 [put] /file1 0 B (0 blocks)
4 [put] /file2 0 B (0 blocks)
5 [put] /file3 0 B (0 blocks)
6 [put] /file2 23 B (1 block)
Log synced with network
Archive has 6 changes (puts: +6, dels: -0)
Current Size: 180 B
Total Size:
- Metadata 336 B
- Content 219 B
Blocks:
- Metadata 7
- Content 3
I can see that there are six revisions of the information in this Dat, but I have no idea what the significance of any of those revisions might be. This is where commit messages like the ones in Git would come in handy. Or, in lieu of that, something akin to the concept of a Git tag to mark milestones. How about this:
$ dat log
1 > dat initialization...
[put] /dat.json 39 B (1 block)
[put] /dat.json 157 B (1 block)
2 > adding placeholders for files 1 - 3
[put] /file1 0 B (0 blocks)
[put] /file2 0 B (0 blocks)
[put] /file3 0 B (0 blocks)
3 > adding results to file2 from experiment 1
[put] /file2 23 B (1 block)
Log synced with network
Archive has 3 commits, 6 changes (puts: +6, dels: -0)
Current Size: 180 B
Total Size:
- Metadata 336 B
- Content 219 B
Blocks:
- Metadata 7
- Content 3
Untracked files:
- file4
- MessyNotes.md
Changes not synced:
- [put] /file1 32 B (1 blocks)
Now, let’s say I wanted to go back to just before the results were entered from experiment 1 (commit #2 above). A command might look like this:
$ dat checkout 2
Archive at commit 2: "adding placeholders for files 1 - 3"
Now, to be fair, the Dat documentation does claim that you can revert the history (so it is there in the protocol), but the example shown on the linked page will not really work (you have to use the URL of the file with the “?version=X” query string, not the Dat archive itself). Also, why does this only work on HTTP, not on a dat:// link? It just feels like either an unfinished feature, or a feature that was abandoned after the project’s focus shifted. Since it already partially works, maybe it just needs some love to get it fully implemented.
Peer-to-Peer Git
The more I think about it, the more I’m convinced that what I actually want is a fork of Git that would do just a few things differently:
- It would be nice to have the peer-to-peer capability of Dat.
- Large binary files would need to be a first-class citizen.
- The interface should to be (much) less daunting for non programmers.
- CLI: Great, but make the commands more obvious.
- GUI: An (optional) graphical interface out-of-the-box is probably a good idea for collaborators who never venture out of the GUI world of Word, Excel, and occasionally R-Studio.
Now, the question is: What is the shortest path to this? Would it be easier to “adapt” Git to be more like Dat, or adapt Dat to be more like Git? I’m not sure, but every time I sit down and write about it, I get a little closer to clarity I think…