Announcing js-IPFS 0.53.0

Alex Potsides
Announcing js-IPFS 0.53.0
#js-ipfs #breaking change
Share this item:

# 🔦 Highlights

full-duplex streaming browser APIs and the latest libp2p

js-IPFS@0.53.0 has launched into the stratosphere with an experimental gRPC (opens new window)-over-websockets server that means the http client has true full-duplex streaming for the ipfs.add API

# ↔️ gRPC over websockets

In the beginning, go-IPFS shipped with an HTTP API (opens new window), which js-IPFS also implemented (opens new window) with the aim of being cross-compatible.

This HTTP API allows you to orchestrate a locally running node from a language or environment that may not be able to run a fully-fledged IPFS node, or it may be preferable to have a single system-wide node shared between multiple applications.

The go-IPFS HTTP API supports streaming responses, and returns the message from any error that occurs during a streaming response as an HTTP trailer (opens new window).

When the request is made from the browser or from node via the node-fetch (opens new window) module, HTTP trailers are not available (opens new window). Bugs are open against FireFox (opens new window) and Chromium (opens new window) to address this but they are unlikely to be fixed in any reasonable time frame, partially due to a security concern (opens new window) around allowing trailers to change the status of a response after that response has been sent.

The effect of all this is that if you are reading a streaming response from the server, and an error occurs on the server, from the client’s point of view the stream just stops (opens new window) with no error message, which is less than ideal.

That’s the response end, from the request end, we’d like to be able to send requests of arbitrary length and start processing the response before the request has finished sending (think pubsub, or giving the user server-driven progress notifications when adding large or multiple files).

The Fetch API allows for sending request bodies as WhatWG ReadableStreams (opens new window) but no browser currently supports this (opens new window) (Chrome recently announced an intent to ship (opens new window) this feature which is great news, but FireFox so far has made little progress (opens new window) so we’re not there yet).

If ever implemented widely, this will allow us to stream uploads and downloads, but there’s no guarantee it’ll be full-duplex (opens new window) (e.g., both ends streaming at the same time).

HTTP/2 supports full-duplex streaming, but currently (opens new window) it requires ALPN (opens new window), which requires TLS (opens new window), which requires certificates, which is overkill if you’re only communicating over the loopback interface.

Where does this leave us?

We want full-duplex streaming over the API; we want the safety of typed inputs and outputs.

gRPC (opens new window) gives us the typing, (yay!) and there is gRPC-web (opens new window) which is designed to work over HTTP, but it does not support bi-directional streaming, and while there are plans afoot (opens new window), there is nothing we can use today.

There is a way we can do bi-directional, full-duplex streaming today: WebSockets (opens new window).

The gRPC-web (opens new window) team have no plans (opens new window) to publish even an experimental spec for gRPC-web-over-websockets due to concerns about compatibility with existing HTTP infrastructure - proxies and the like, all of which are not relevant to our use-case.

The engineering team at improbable.io (opens new window) have published a golang implementation of gRPC-web (opens new window) that wraps a gRPC (opens new window) server but also provides a websocket transport (opens new window) with HTTP fallback.

js-IPFS@0.53.0 ships with a JavaScript port (opens new window) of this gRPC-web (opens new window)-over-websockets server. Phew!

To use it we’ve shipped a new client called ipfs-client (opens new window) (hat tip to @brosenan (opens new window) for very kindly donating the module name) - a fully Core API (opens new window) compatible client. It combines the ipfs-http-client (opens new window) and the new ipfs-grpc-client (opens new window), using gRPC (opens new window) for implemented methods and falling back to HTTP for methods that have not been ported to gRPC (opens new window) yet.

const createClient = require('ipfs-client')

const client = createClient({
  grpc: '/ipv4/127.0.0.1/tcp/5003/ws',
  http: '/ipv4/127.0.0.1/tcp/5002/http',
})

const id = await client.id()

A few caveats:

  1. This only works against js-IPFS for now
  2. It only works over the loopback address (not localhost) as loopback is considered a secure context (opens new window), otherwise it would require TLS (opens new window)
  3. Only the ipfs.add API has full-duplex streaming enabled for this release - expect a lot more in future!

There’s a new example called ipfs-client-add-files (opens new window) that you can use as a sandbox to experiment with this new client.

If you fire it up, you’ll see progress and file import events appearing before the upload request has completed, something that’s not been possible with the ipfs-http-client until now.

What’s next? We’re going to extend the gRPC (opens new window) implementation to all streaming methods and get a server into go-IPFS so you can use the same client with that implementation too.

The full-duplex streaming capability opens up all sorts of interesting possibilities, for example, opening up the full libp2p API over HTTP instead of the limited subset that we currently support.

Speaking of libp2p:

# ☎️ libp2p@0.30.x

The 0.53.0 release of js-IPFS ships with libp2p@0.30.x which gives us TypeScript type definitions, Auto Relay support, and
improved Peer advertising and dialer mechanics.

Expect a blog post here soon with a deep dive, but in the mean time see the libp2p@0.30.x release notes (opens new window) for more details.

# ✨New features

# 🔨 Breaking changes

  • The websocket transport will only dial DNS+WSS addresses - see https://github.com/libp2p/js-libp2p-websockets/releases/tag/v0.15.0

# 🕷️ Bug fixes

# 🗺️ What’s next?

Check out the js-IPFS Project Roadmap (opens new window) which contains headline features organised in the order we hope them to land.

Only large features are called out in the roadmap, expect lots of small bugfix releases between the roadmapped items!

# 😍 Huge thank you to everyone that made this release possible

# 🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don’t know how? Well, there are a few places you can get started:

# ⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works, and what you can do with it is at discuss.ipfs.tech (opens new window). We are also available at the #ipfs channel on Freenode.