xrsh
/
xrsh
1
0
Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases Wiki Activity
Page: Home
Pages
Home network
7 Home
coderofsalvation edited this page 2024-01-04 19:18:56 +01:00
Table of Contents

XRSH (WebXR Shell)

modular WebXR components & multi-shell

the whole project and its expected outcome(s).

xrsh (xrshell) brings more unix workflow & philosophy to WebXR, and mitigates frequent laptop-vs-headset switching. It offers an alternative OS escape-hatch for commercial devices like Meta/Apple's recently released headsets. It does this by promoting the use of (interactive text) terminal/operating systems inside WebXR (=xrsh). Thanks to tech like nix, buildroot, and wasm, new packaging possibilities can be explored.

Technically, xrsh is just a bundle of freshly created re-usable FOSS WebXR components.

It provides a common filesystem interface for interacting with WebXR, by offering the well-known linux/unix toolchain: a commandline and filesystem to invoke, store, edit and run these WebXR utilities, regardless of their implementation. Think of it as a termux for the VR/AR headset browser.

xrsh makes it easier for:

  • users booting a light x86 (linux)-distro (ISO-file) inside the headset browser
  • proxy-ing a shell from the laptop to the headset
  • linux users to learn and use a wide range of linux AND XR utilities
  • new users to learn and use linux (or other OS'es)
  • access WebXR environments as a device (/dev/browsertab1/world e.g.)
  • livecode using terminal auto-completion for XR component (registries)

In short: it is important to offer an escape-hatch to endusers to go beyond the dumb-downed restrictive OS workflow of commercial WebXR headsets, simply by entering an URL.

outcomes:

  1. CODING: repository with WebXR components created for xrsh (used by #2)
  2. DEMO/DOCS: a working xrsh WebXR application (*) in a repository
  3. RESEARCH: accessibility research (components) and sharing to community

All is realized by using FOSS AFRAME/THREEjs framework (components) with an opensource license

Another overarching theme is mitigating the constant switching between laptop and headset (for textual input and debugging): a famous pain point for (web)XR developers. As well as implementing different ways (accessibility) of controlling an XR Shell (using speech & gestures e.g.).

milestones

The project has no past/present funding sources. The project contains 3 chapters with milestones:

1. CODING: repository with components created for xrsh (used by #2)
1.1. xrsh higher-order APP component (allowing portable AFRAME apps)
  1.1.1 app: windowmanager demo: auto-layout of (AFRAME) apps
  1.1.2 app component should allow requiring local/remote AFRAME/CSS/JS files dependencies
2. terminal component
2.1 VM-component: run ISO in terminal (using https://github.com/copy/v86)
2.2 VM-component: js gateway: `$ /dev/tab/voidlinux1/js < foo.js` runs javascript in browsertab
2.3 CodeMirror component: allow editing of files
2.4 remoteshell-mode: redbean component (proxy laptop stdin/stdout to AFRAME component)
2.5 multi browsertab mode to allow terminal(s)<->XR world interfacing/livecoding
2.6 VM-component: persist VM(s) to indexeddb (to survive webpage reload/offline) or dump-file
2.7 VM-component: build fast default ISO 
2.8 VM-component: WASI Mode: `$ ./nano.wasm` (https://runno.dev/wasi) 
2.9 VM-component: [research] X11 or VNC-support to default ISO
2.10 VM-component: [research] add nix package manager to default ISO
2.11 VM-component: [research] nix-package-to-iso or wasm (`bash-vim-and-nix.iso` e.g.)
2.12 VM-component: XR layers (bypass framebuffer resampling for improved reading)
2.13 VM-component: HTML5 File API gateway: mount directory onto filesystem
2.14 VM-component: wasm gateway: `$ /dev/tab/voidlinux1/wasm < foo.wasm` runs wasm in browsertab 
2.15 VM-component: html gateway: `$ /dev/tab/voidlinux1/html < index.html` overwrites DOM of certain tab
2.16 VM-component: browserconsole gateway: `tail -f /dev/tab/voidlinux1/js` e.g.


3 remotestorage component
3.1 add THREE-compatibility (https://three.org)
3.2 add AFRAME-compatibility: write THREE wrapper for use in https://aframe.io
3.3 add remotestorage to XRSH 2D (dom/html) ui
3.4 display connector using spatial ui THREE: (https://github.com/NikLever/CanvasUI e.g.)
3.5 display connector using spatial ui AFRAME using HTML-mesh


4. DEMOCODING/DOCS: a working xrsh XR application (*) in a repository
4.1 demo 3D asset viewing, navigation & linking (xrfragments component from xrfragment.org)
4.2 repo + crossplatform selfcontained executable (redbean.dev) + nix package
4.3 demo generic AFRAME launcher (1.1) for components (icon+description+help)
4.4 codemirror component demo
4.5 terminal component demo: i/o using canvas + https://github.com/xtermjs/xterm.js
4.6 terminal component demo: run custom ISO (https://github.com/copy/v86)
4.7 terminal component demo: ISO filesystem-javascript interface
4.8 remotestorage component demo: show remotestorage connectorstorage
4.9 remotestorage component demo: pull-request: add remotestorage-webxr.js to https://blog.searxr.me
4.10 remotestorage component demo: pull-request: add remotestorage-webxr.js to https://xrfragment.org
4.11 demo remotestorage: connect filesystem (mount?FuSE?)
4.12 demo remotestorage: edit file using linux editor (in VM terminal)
4.13 demo rclone interface
4.14 demo rclone interface mount filesystem 
4.15 demo searx component
4.16 packaging: nix/docker/yunohost


5. RESEARCH ( accessibility research (components) and community-sharing
5.1. publish components upstream to FoSS registries:
5.2. gesture-manager component: uGlyphs integration (https://lig-microglyph.imag.fr)
5.3. gesture-manager component: train on gesture (via uGlyph description and "test")
5.4. common task component(s) like copy-paste: objects can react on paste thru this.el.addEventListener('paste', ...) e.g.
5.5. speech component(s): (prompt-component which allows speaking 'foo' to trigger this.el.dispatchEvent('foo'))

projects or organisations relevant to this project

concerning WebXR (pain points):

concerning rclone & selfcontained crossplatform binaries

concerning remotestorage:

Leon looked into RS during the NLnet searx (https://blog.searxr.me) project, and got quite excited about it. Remotestorage.io offers a nice easy drop-in solution compared to bigger-scope initiatives like solidproject). Unfortunately the searx-project ran out of time before I could turn searxr into an RS app. A remotestorage WebXR-component would definately help endusers not having to roll their own storage-infrastructure.

existing or historical efforts.

There seemed to be no projects which allow (spatial) interop with (other) operating systems (in WebXR). From a on-headset development point of view, all efforts quickly fall into these categories:

These solutions are either proprietary, non-WebXR-compatible, or don't allow booting/interop with (offline) OS'es, or require hairy installing/configuring/hacking various pieces of software.

concerning remotestorage (RS) in XR

It seems there's no RS gui apps which work in webxr immersive mode or provide RS spatial filepickers. Perhaps, its absense can be explained by the fact, that on one side of the spectrum there are gui libraries for THREE & babylon. And on the other side of the spectrum is https://github.com/remotestorage/remotestorage.js. There isn't anything which integrates them together (an RS WebXR Filepicker e.g.). It would be circular reasoning to assume that 'nobody did it because nobody wants it'. The bottomline is that iright now, webxr developers don't have an easy way to create spatial RS apps (without having to worry about DIY gui-bindings).

Solidproject/nextcloud: these are quite big complex applications, with a much bigger scope and requirements (and no WebXR integration).

concerning rclone

Rclone gui (https://rclone.org/gui): is hidden behind a commandline-flag which supports an incomplete set of rclone API features. My first thought would be to extend this ui (which is html-only), however this project needs file i/o to persist state e.g. Extrapolating on opinions of rclone maintainers: implementing extra functionality would be considered 'out of scope' / non-KiSS. Perhaps a combination of the HTML-gui and some WebXR ui-elements (which trigger rclone JSON API-calls) could be a great stepping stone towards data-management in XR.

developer-only rclone cli- or library-only automators which use/demonstrate the rclone JSON API:

significant technical challenges you expect to solve during the project (if any?)

  1. A smart generic interface for mounting rclone/remotestorage/javascript filesystems to the ISO VM filesystem (FuSE might be overkill vs a shared IndexedDB). Some small tests need to be done, before settling on the final interface.

  2. Keeping components (remotestorage-webxr.js e.g.) a one-file drop-in javascript library, which supports/detects popular WebXR frameworks (while keeping the size small). This can probably be done by importing/lazyloading framework-specific code at runtime).

ecosystem of the project

how to engage with relevant actors and promote the outcomes:

  1. developer/researcher: Leon van Kammen (HU)
  2. developer/researcher: Fabien Benetou (FR)
  3. WebXR Frameworks: aframe.io threejs.org
  4. FoSS WebXR Registry: https://aframe.io/aframe-registry,2.1.2 (**)
  5. FoSS WebXR Registry: https://github.com/c-frame (**)
  6. ISO / VM-related: http://copy.sh/v86 (*)
  7. RS-storage: we will test with a free pod at 5apps.com (and can contact them for questions) along with resources on remotestorage.io and unhosted.org.
  8. selfcontained binary (cosmopolitan libc): Leon as direct contact on discord with its maintainer.
  9. testers: As members of https://thefutureoftext.org/team, we can have our members test it across various devices.
  • = Leon is in contact with the maintainer of the javascript VM project (http://copy.sh/v86/) which has consulted me for this project proposal. ** = Fabien is maintainer of 5 and we are both in contact with the maintainer of 4.

who should use it to make this project a succes?

Well first of all, anyone who wants to BUILD/CREATE (in) WebXR instead of consuming only. Being able to do daily work / development ON the headset (without requiring an extra laptop) would be awesome travel-wise and ewaste-wise. On the other hand, curious teenagers (which Fabien is giving workshops to) who are interested in building in XR instead of consuming in XR, while getting exposed to speaking/reading/writing interactive text to the terminal.

Also, any WebXR experience/enduser which involves userdata / datasovereignty. As WebXR is relatively a new standard / space, saving data in better (new) ways (using RS) makes sense. Additionally, hopefully it inspires existing webgl/webgpu game developer studios, to refactor/simplify their games into unhosted ones. Saving game state to RS would save them (and earth) a lots of unnecessary server-infrastructure(costs). Last but not least, every 'ah now I get why RS is cool (and saves us money)'-epiphany caused in the heads of any developer is a win. RS is more than 'just' a dropbox alternative.

eco-friendly considerations

The technologies used in this project promote a greener approach:

  • OS'es & remote shells in webXR mitigate the need for physical hardware = less e-waste
  • connecting cross-OS application ecosystems (linux utilities in webxr e.g.) = less software waste
  • bring new life to old/abandoned (phone) VR/AR headsets
  • run same binary on various platforms (at least x86_64 mac/linux/win) using cosmopolitan libc
  • mitigates server- and hostingvendor-complexity
  • less diskspace + less cpu + less webservices + less resources = less carbon & e-waste
  • opensource binary: executable is also zip-archive (when renamed to .zip), to manage lua-scripts & add custom ISO linux-images.