Categories
Blog

Distributed findings

Recently I stumpled upon two interesting projects that I thought was worth sharing.

The first is Reticulum, an encrypted network that allows one two create adhoc private mesh networks over both internet and wireless links.

The second is the Eg-walker paper detailing an efficient crdt algorithm for text. What I found interesting is that some of the algorithms seems more general and this might be applicable in other situations with concurrency. The paper also details a lot of tricks used to spead an the implementation. The paper is based on diamond types, so more a write up of an existing system rather than a new algorithm that is not really used anywhere yet.

Categories
Blog

SQL links

SQL sometimes feel a bit like black magic. How the query engine figures out what plan to execute, why is this query slow etc. The best resource I have come across is the book Indexing beyond the basics. Its available as a PDF download or for reading online. It gives a bit of easy to remember theory and lots of great examples.

Some more random links I stumples upon recently:

And lastly ALWAYS remember to

Categories
Blog

A work diary

Acording to wikipedia, diaries are almost 2000 years old. And they still work just as well today! At work I have been keeping a simple diary in the form of a giant org document. It’s a practice I have been doing for 9 years and would not be without. While org mode can do a million things, I almost use none of the functionality. Instead the main function I use is the ability to search anything instantly. I use it as for todos, for saving SQL queries, code snippets, for links and in general for things I need to find again. I have a key binding to enable F9 to insert the current date and then I use tags to split things into topics.

(define-key global-map (kbd "<f9>")
'(lambda () (interactive)
  (when (eq major-mode 'org-mode)
    (org-insert-time-stamp nil t nil)
    (insert " ")
  )))

The only extra module I have installed is org-bullets.

I store the file in git and just do commits regularly. Over time I split out the oldest entries into seperate files, keeping only the latest 3 years in the main file.

Categories
Blog

Raspberry pi music player

Ever since the raspberry pi zero 2 was released I knew it would be good as a always on machine at home. It uses around 0.4Wh idle power which amounts to around 5kWh per year at 0.6 Wh average. Or 1-2 euros and around 500g co2 per year.

I have been digitizing my music collection. I do still enjoy and mostly listen to full albums and having the ability to listen to any of hundreds of those with a click is amazing. I still keep my CDs around, they look like this:

Today you can store the whole library of hundres of albums on a single micro SD card for less than 50 euros for 256GB. I was able to use my old external stero link amp together with sennheiser HD 650 headphones. The clarity is great, but there isn’t a whole lot of bass. The final setup looks like this:

I’m using the open source moode audio for playing the audio from the PI using a browser. Not super happy about the interface, but I managed to get it into a state where I mainly just click on an album I want to hear and it plays. It did take a whole to get covers for the whole collection, but that that is done I havn’t touched the system for around a month.

Categories
Blog

P2P applications

User autonomy is at the heart of P2P applications. This means that the data should be written locally first and later exchanged with peers. For this data to be safely exchanged, the data should be signed by a key in the users control and ideally be encrypted as well. The aggregate data is what makes up a full application. There is no single source of truth as in a centralized application instead this concept is fluid and changes over time as the application grows.

To make this more concrete lets look at specific protocol for writing P2P applications know as Scuttlebutt. In Scuttlebutt each peer has their own feed with their own key pair where they write messages. These messages can be follow and block messages of other peers allowing a social network to be formed or it can be application specific messages. Because it is generally non-trivial to have a direct connected between two peers on the internet, a intermediate peer with a public IP is often used. This can either be a pub that relays messages between peers, meaning the peer don’t have to be online at the same time, or could be a room server where peers can exchange message directly using the room server as an E2E encrypted tunneling service.

Peers can run applications by either installing them on their computers or phones (such as manyverse) or can run directly in the browser with no installation required using ssb-browser-core. Development of the database (ssb-db2, jitdb) that runs on all of these platforms and the second generation room server was sponsored by the European Union through a grant from NGI pointer. For more detailed information about that project included videos see this.

Recently I have been creating a tech demo showcasing private groups and all of the technology mentioned above in a single application. The application is a good starting point for writing other applications using the SSB stack and allow it to run directly in the browser. You can try an online demo and try it for yourself. Note this is a demo and thus not very polished.

Categories
Blog

Butts in the Browser Fosdem talk

Over the weekend I gave a talk on running secure scuttlebutt directly in the browser. The talk gives an overview of ssb in general, how it is different from federated services. It touches on some of the work we have been doing for a European NGI pointer grant including jitdb. The browser app uses local storage, indexed db and Chrome FS. WASM for crypto and web sockets for communication with other nodes. Slides are available here and video available here.

Categories
Blog

GITEA and cors

I was looking to use git as a storage backend for a project I was working on, as one can run git in the browser now using isomorphic git. The app needed to run from my machine directly for easier development, which meant that the remote git backend has to support CORS. Luckily the gitea project allows one to quite easily host your own git reposities with an interface very similar to github.

The following 3 extra options was need in the app.ini file for CORS to ne working:

[repository]
ACCESS_CONTROL_ALLOW_ORIGIN = *

[cors]
ENABLED = true
ALLOW_CREDENTIALS = true

Categories
Blog

How to setup a PUB for SSB browser

SSB-browser as mentioned in my last blog post runs a SSB client in the browser. In order to communicate with other peer it needs a pub to relay messages through. Furthermore the pub acts as a super node allowing the browser peer to do partial replication without necessarily needing to store the full feeds of peers.

We don’t really gain very much compared to the classical client-server setup if we rely too much on especially a single pub. The browser client can of course run fully off-line, but still when it needs to communicate messages with other peers it needs some intermediary.

In this post I will show how to setup a pub that can be used together with ssb browser. I will also show how this can be used to build a totally seperate network from the main scuttlebutt network for local groups and communities.

First of all one needs to install the basic ssb-server. On my pub I use ssb-minimal-pub-server. The good thing is that this already comes with everything we need for peer invites. So we just need to install the before mentioned ssb-partial-replication module into ~/.ssb/node_modules/. Easiest is just to git clone and npm install it from the directory. It is also recommended to install ssb-tunnel but this is not required.

Lastly we need to configure ssb-server to use the modules and to enable web sockets so the browser will be able to connect.

{
"connections": {
"incoming": {
"net": [{ "port": 8008, "host": "::", "scope": "public", "transform": "shs", "external": ["between-two-worlds.dk"] }],
"ws": [{ "port": 8989, "host": "::", "scope": "public", "transform": "shs", "external": ["between-two-worlds.dk"], "key": "/etc/letsencrypt/live/between-two-worlds.dk/privkey.pem", "cert": "/etc/letsencrypt/live/between-two-worlds.dk/cert.pem" }]
},
"outgoing": {
"net": [{ "transform": "shs" }],
"tunnel": [{ "transform": "shs" }]
}
},
"logging": {
"level": "info"
},
"plugins": {
"ssb-device-address": true,
"ssb-identities": true,
"ssb-peer-invites": true,
"ssb-tunnel": true,
"ssb-partial-replication": true
},
}

As one can see this uses letsencrypt, but that is not strictly needed.

Lastly we need to start up the server. I run it using the following bash script that will make sure the pub keeps running even if it should crash.

#!/bin/bash
while true; do
sbot server --port 8008 >> log.txt 2>&1
sleep 30
done

This is everything you need to run your own scuttlebutt pub that supports browser clients.

Separate network

If one wants to run a totally different network than the main scuttlebutt network one needs to use a different caps key. This means that messages from this network will never appear on the main network. This can generated using using node:

crypto.randomBytes(32).toString('base64')

And can be added to the config file on the pub using:

"caps": {
"shs": "72I4EC/gZUuNffHEwooHob4hhtXHAj0HE5xKM3njRSg="
}

SSB-browser-core needs a little tweak to use the same caps. Change browser.js to:

require('./core').init(dir, { 
  caps: { 
   shs: '72I4EC/gZUuNffHEwooHob4hhtXHAj0HE5xKM3njRSg=' 
  }
})

It might also be a good idea to change hops to 2 in the client. In order to bootstrap the network, generate an invite or follow the first user from the command line of the pub in order for messages to start propagating.

sbot publish --type contact --contact '@feedId=.ed25519' --following

That is all you need to run your seperate network 🙂

Categories
Blog

Secure scuttlebutt in a browser

For roughly the last year I have been working on bringing secure scuttlebutt to the browser. The result of this is ssb-browser-core.

Secure scuttlebutt allows one to write an unforgeable append-only log of messages and to exchange these messages between nodes in a p2p network without a central authority. With this, one can build event sourcing systems.

The first secure scuttlebutt implementation (there are also Go and rust implementations now) is a normal node application and thus requires a lot of things that are not readily available in a browser such as a filesystem and crypto. Filesystem api is provides by random-access-web that for chrome uses the file system api which is quite fast. Crypto is provided by libsodium.js which uses wasm and is roughly 90% the speed of running it natively. The last part is network, he we rely on web sockets which gives us a long running connection to a server somewhere. With epidemic broadcast trees to exchange messages, one can get the latest messages for several hundred feeds (probably also thousands) in a reliable and low latency manor.

With this, one is able to run secure scuttlebutt in any environment where the browser works. This includes Chrome, Firefox, Safari and on android and iOS. By running as a progressive web app it is possible to side step the normal walled garden of the dominant mobile operating systems. Jacob also managed to get it running as a web extension.

The main application for showing what can be built on top of these primitives is a social network called scuttlebutt. In order to test the functionality of the core, I wrote a demo application with most of the basic functionality for having conversations. Other applications that has been built on top of the protocol includes gatherings, chess, git, books.

I did a walkthrough of how it works for a few people in the SSB community. You can watch the video below.

This weekend I’ll be at the University in Basel to discuss SSB and related p2p technologies.

Categories
Blog

Bornhack talk

I recently gave a talk at Bornhack with the title anarchitecture. The talk is available here and was recorded so you can watch it here.