File System (WNFS)

Working with the ODD SDK File System (WNFS)

The Web Native File System (WNFS) is a file system built on top of the InterPlanetary Linked Data model (IPLD). Each ODD user has their own WNFS, and apps store user files and data in it.

Each file system has a public tree and a private tree, much like your macOS, Windows, or Linux desktop file system. The public tree is "live" and publicly accessible on the Internet. The private tree is encrypted so that only the owner can see the contents.

All information (links, data, metadata, etc.) in the private tree is encrypted. Decryption keys are stored so that access to a given directory grants access to all of its subdirectories.

WNFS is structured and functions similarly to a Unix-style file system, with one notable exception: it's a Directed Acyclic Graph (DAG), meaning that a given child can have more than one parent (think symlinks but without the "sym").

Permissions

An ODD app assumes full access to the file system unless given a permissions object in the configuration, then it assumes authorization to parts of the filesystem will be granted when requesting capabilities from another app.

Paths

WNFS uses directory and file paths built from path segments by path functions.

const { RootBranch } = odd.path

// Creates a directory path equivalent to "public/some/directory/"
const publicDirectoryPath = odd.path.directory("public", "some", "directory")

// Creates a directory path equivalent to "private/some/directory/"
const privateDirectoryPath = odd.path.directory("private", "some", "directory")

// Creates a file path equivalent to "public/some/file"
const publicFilePath = odd.path.file("public", "some", "file")

// Creates a file path equivalent to "private/some/file"
const privateFilePath = odd.path.file("private", "some", "file")

// Paths can also use RootBranch.Public and RootBranch.Private constants
const anotherPublicPath = odd.path.file(RootBranch.Public, "another", "file")
const anotherPrivatePath = odd.path.file(RootBranch.Private, "another", "file")

All WNFS operations expect paths created by path functions. See the path API documentation for more path utility functions.

Path Objects. The path functions create objects like { directory: ["public", "some", "directory"] } or { file: ["public", "some", "file"] }. We recommend you use path functions because they validate paths to make sure they are well-formed.

File System Interface

WNFS exposes a POSIX-style interface:

  • write: write to a file

  • read: read from a file

  • exists: check if a file or directory exists

  • ls: list a directory

  • mkdir: create a directory

  • mv: move a file or directory

  • rm: remove a file or directory

// App info is the namespace used when initializing the ODD SDK program
const appInfo = { creator: "Nullsoft", name: "Winamp" }

// After retrieving a session or loading the file system manually
const fs = session.fs // or program.loadFileSystem(username)

// List the user's public files
await fs.ls(odd.path.directory("public"))

// List the user's private files that belong to a specific app
await fs.ls(odd.path.appData(appInfo))

// Create a sub directory and write to a file
await fs.write(
  odd.path.appData(appInfo, odd.path.file("Sub Directory", "hello.txt")),
  new TextEncoder().encode("👋")
)

// Persist changes and announce them to your other devices
await fs.publish()

// Read from a file
const content = new TextDecoder().decode(
  await fs.read(
    odd.path.appData(appInfo, odd.path.file("Sub Directory", "hello.txt"))
  )
)

Publish

The publish function synchronizes your file system with the Fission API and IPFS. WNFS does not publish changes automatically because it is more practical to batch changes in some cases. For example, a large data set is better published once than over multiple calls to publish.

Returns: CID the updated root CID for the file system.

Remember to publish! If you do not call publish after making changes, user data will not be persisted to WNFS.

API Summary

Methods

Methods for interacting with the filesystem all use absolute paths.

Paths created by path functions have a FilePath or DirectoryPath type. Methods with a DistinctivePath param accept either a FilePath or a DirectoryPath.

The FileContentthat WNFS can store includes FileContentRaw, Blob, string, number, and boolean. FileContentRaw is Uint8Array. In addition, the private file system can store Objects.

write

Writes to a file at a given path.

Params:

  • path: FilePath required

  • content: FileContentrequired

Returns: CID the updated root CID for the file system

Example:

const content = new TextEncoder().encode("hello world")

// create a file called "file" at "public/path/to/a/"
await fs.write(
  odd.path.file("public", "path", "to", "a", "file"),
  content
)

read

Reads from a file at a given path.

Params:

  • path: FilePath required

Returns: FileContent

Example:

const content = await fs.read(odd.path.file("public", "some", "file"))

exists

Checks if there is anything located at a given path.

Params:

  • path: DistinctivePath required

Returns: boolean

Example:

const bool = await fs.exists(odd.path.file("private", "some", "file"))

get

Retrieves the node at the given path, either a File or Tree object

Params:

  • path: DistinctivePath required

Returns: Tree | File | null

Example:

const node = await fs.get(odd.path.directory("public", "some", "directory"))

ls

Returns a list of links at a given directory path

Params:

  • path: DirectoryPath required

Returns: { [name: string]: Link } Object with the file name as the key and its Link as the value.

Example:

// public directory
const publicPath = odd.path.directory("public", "some", "directory")
const publicLinksObject = await fs.ls(publicPath)

// private directory
const privatePath = odd.path.directory("private", "some", "directory")
const privateLinksObject = await fs.ls(privatePath)

// convert private links object to a list
const links = Object.entries(privateLinksObject)

// working with links
const data = await Promise.all(links.map(([name, _]) => {
  return fs.read(
    odd.path.file("private", "some", "directory", name)
  )
}))

mkdir

Creates a directory at the given path

Params:

  • path: DirectoryPath required

Returns: CID the updated root CID for the file system

Example:

// create a directory called "directory" at "public/some/"
const updatedCID = await fs.mkdir(odd.path.directory("public", "some", "directory"))

mv

Move a directory or file from one path to another.

Params:

  • from: DistinctivePath required

  • to: DistinctivePath required

Returns: CID the updated root CID for the file system

Example:

const fromPath = odd.path.file("public", "doc.md")
const toPath = odd.path.file("private", "Documents", "notes.md")
const updatedCID = await fs.mv(fromPath, toPath)

rm

Removes a file or directory at a given path.

Params:

  • path: DistinctivePath required

Returns: CID the updated root CID for the file system

Example:

const updatedCID = await fs.rm(odd.path.file("private", "some", "file"))

Versioning

Each file and directory has a history property, which you can use to get an earlier version of that item. We use the delta variable as the order index, primarily because the timestamps can be slightly out of sequence due to device inconsistencies.

const articlePath = odd.path.file("private", "Blog Posts", "article.md")
const file = await fs.get(articlePath)

file.history.list()
// { delta: -1, timestamp: 1606236743 }
// { delta: -2, timestamp: 1606236532 }

// List more than (by default) 5 versions
file.history.list(10)

// Get the previous version
file.history.back()

// Go back two versions
const delta = -2
file.history.back(delta)

// Get a version strictly before a timestamp
// The first version (delta -2) is prior to
// the second version (delta -1) timestamp
file.history.prior(1606236743)

Requesting many versions with file.history.list can be slow. The acceptable delay will depend on your application.

File system utilities

An ODD program includes a set of utility functions for working with the file system:

export type FileSystemShortHands = {
  addPublicExchangeKey: (fs: FileSystem) => Promise<void>
  addSampleData: (fs: FileSystem) => Promise<void>
  hasPublicExchangeKey: (fs: FileSystem) => Promise<boolean>
  load: (username: string) => Promise<FileSystem>
  recover: (params: RecoverFileSystemParams) => Promise<{ success: boolean }>
}

  • addPublicExchangeKey. Stores the public part of the exchange key in the DID format in the /public/.well-known/exchange/DID_GOES_HERE/ directory of the user's file system.

  • addSampleData. Add sample directories and a file to the file system for quick experimentation.

  • hasPublicExchangeKey. Checks if the public exchange key was added in the well-known location.

  • load. Loads the user's file system.

  • recover. Recovers a file system.

See Sharing Private Data for more information on addPublicExchangeKey and hasPublicExchangeKey.

Loading the file system

The file system is loaded by the ODD SDK at program initialization, but there are times when you may want to delay loading it. For example, you may want to load the file system in a Worker.

You can load a file system with load function:

const fs = await program.fileSystem.load(username)

Recovering a file system

File system recovery is a utility for recovering a user's data when their account cannot be fully recovered. Your app must store a user's username and file system encryption key in a recovery kit or secure storage such as a password manager or iCloud.

A recovery kit might be a text file that looks like:

username: llama
key: SmnsBR3krWxhNm+tnUDX+2pm3gnyXnnBNxEhXnt4jp0=

The username is available on program.session. You can access the encryption key with the following:

import * as uint8arrays from 'uint8arrays'
import { retrieve } from '@oddjs/odd/common/root-key'

const accountDID = await program.accountDID(username)
const key = await retrieve({ crypto, accountDID })

// Base64 encode the key to store it as a string
const encodedKey = uint8arrays.toString(key, 'base64pad')

The recover function works by assigning the file system to a new user:

import * as uint8arrays from 'uint8arrays'

// Convert the key to a Uint8Array
const key = uint8arrays.fromString(encodedKey, 'base64pad')

const { success } = program.fileSystem.recover({ newUsername, oldUsername, key })

The recover function will automatically register newUsername. You must check that the new username is valid and available before calling recover with it.

File system event listeners

An ODD program emits events on local changes to the file system and when the file system is published.

program.on("fileSystem:local-change", ({ path, root }) => {
  console.log("The file system has changed locally 🔔")
  console.log("Changed path:", path)
  console.log("New data root CID:", root)
})

program.on("fileSystem:publish", ({ root }) => {
  console.log("The file system has been published 🚀")
  console.log("New data root CID:", root)
}

File system behavior

Conflict Resolution

A user may access their file system from more than one device. When multiple devices write to the same resource, the last write wins. We will improve this approach in a future release of the ODD SDK and WNFS. See https://github.com/wnfs-wg/roadmap/issues/12 for more details.

Offline and slow network connections

When a device goes offline or has a slow connection, it will continue to persist changes locally in the browser. The ODD SDK syncs changes when the file system is loaded or when it detects a browser has come online.

Sync

A device may not be aware of changes that another device has made. To pull changes, reload the file system. We plan to add an interface to subscribe to file system changes across devices: https://github.com/oddsdk/ts-odd/issues/414.

PreviousComponentsNextSharing Private Data

Last updated