33 KiB
%%% Title = "XR Fragments" area = "Internet" workgroup = "Internet Engineering Task Force"
[seriesInfo] name = "XR-Fragments" value = "draft-XRFRAGMENTS-leonvankammen-00" stream = "IETF" status = "informational"
date = 2023-04-12T00:00:00Z
author initials="L.R." surname="van Kammen" fullname="L.R. van Kammen"
%%%
.# Abstract
This draft offers a specification for 4D URLs & navigation, to link 3D scenes and text together with- or without a network-connection.
The specification promotes spatial addressibility, sharing, navigation, query-ing and tagging interactive (text)objects across for (XR) Browsers.
XR Fragments allows us to enrich existing dataformats, by recursive use of existing proven technologies like URI Fragments and BibTags notation.
Almost every idea in this document is demonstrated at https://xrfragment.org
{mainmatter}
Introduction
How can we add more features to existing text & 3D scenes, without introducing new dataformats?
Historically, there's many attempts to create the ultimate markuplanguage or 3D fileformat.
Their lowest common denominator is: (co)authoring using plain text.
XR Fragments allows us to enrich/connect existing dataformats, by recursive use of existing technologies:
- addressibility and navigation of 3D scenes/objects: URI Fragments + src/href spatial metadata
- Interlinking text/3D by deriving a Word Graph (XRWG) from the scene (and augmenting text with bibs / BibTags appendices (see visual-meta e.g.)
NOTE: The chapters in this document are ordered from highlevel to lowlevel (technical) as much as possible
Core principle
XR Fragments strives to serve (nontechnical/fuzzy) humans first, and machine(implementations) later, by ensuring hasslefree text-vs-thought feedback loops.
This also means that the repair-ability of machine-matters should be human friendly too (not too complex).
"When a car breaks down, the ones without turbosupercharger are easier to fix"
Let's always focus on average humans: our fuzzy symbolical mind must be served first, before serving a greater categorized typesafe RDF hive mind).
Humans first, machines (AI) later.
Thererfore, XR Fragments does not look at XR (or the web) thru the lens of HTML.
XR Fragments itself is HTML-agnostic, though pseudo-XR Fragment browsers can be implemented on top of HTML/Javascript.
Conventions and Definitions
See appendix below in case certain terms are not clear.
List of URI Fragments
| fragment | type | example | info |
|---|---|---|---|
#pos |
vector3 | #pos=0.5,0,0 |
positions camera to xyz-coord 0.5,0,0 |
#rot |
vector3 | #rot=0,90,0 |
rotates camera to xyz-coord 0.5,0,0 |
#t |
vector2 | #t=500,1000 |
sets animation-loop range between frame 500 and 1000 |
#...... |
string | #.cubes #cube |
object(s) of interest (fragment-to-object-or-classname) |
xyz coordinates are similar to ones found in SVG Media Fragments
List of metadata for 3D nodes
| key | type | example (JSON) | info |
|---|---|---|---|
name |
string | "name": "cube" |
available in all 3D fileformats & scenes |
class |
string | "class": "cubes geo" |
available through custom property in 3D fileformats |
href |
string | "href": "b.gltf" |
available through custom property in 3D fileformats |
src |
string | "src": "#q=cube" |
available through custom property in 3D fileformats |
Popular compatible 3D fileformats: .gltf, .obj, .fbx, .usdz, .json (THREE.js), .dae and so on.
NOTE: XR Fragments are file-agnostic, which means that the metadata exist in programmatic 3D scene(nodes) too.
Navigating 3D
Here's an ascii representation of a 3D scene-graph which contains 3D objects ◻ and their metadata:
+--------------------------------------------------------+
| |
| index.gltf |
| │ |
| ├── ◻ buttonA |
| │ └ href: #pos=1,0,1&t=100,200 |
| │ |
| └── ◻ buttonB |
| └ href: other.fbx | <-- file-agnostic (can be .gltf .obj etc)
| |
+--------------------------------------------------------+
An XR Fragment-compatible browser viewing this scene, allows the end-user to interact with the buttonA and buttonB.
In case of buttonA the end-user will be teleported to another location and time in the current loaded scene, but buttonB will
replace the current scene with a new one, like other.fbx.
Embedding 3D content
Here's an ascii representation of a 3D scene-graph with 3D objects ◻ which embeds remote & local 3D objects ◻ with/out using queries:
+--------------------------------------------------------+ +-------------------------+
| | | |
| index.gltf | | ocean.com/aquarium.fbx |
| │ | | │ |
| ├── ◻ canvas | | └── ◻ fishbowl |
| │ └ src: painting.png | | ├─ ◻ bass |
| │ | | └─ ◻ tuna |
| ├── ◻ aquariumcube | | |
| │ └ src: ://rescue.com/fish.gltf#q=bass%20tuna | +-------------------------+
| │ |
| ├── ◻ bedroom |
| │ └ src: #q=canvas |
| │ |
| └── ◻ livingroom |
| └ src: #q=canvas |
| |
+--------------------------------------------------------+
An XR Fragment-compatible browser viewing this scene, lazy-loads and projects painting.png onto the (plane) object called canvas (which is copy-instanced in the bed and livingroom).
Also, after lazy-loading ocean.com/aquarium.gltf, only the queried objects bass and tuna will be instanced inside aquariumcube.
Resizing will be happen accordingly to its placeholder object aquariumcube, see chapter Scaling.
XR Fragment queries
Include, exclude, hide/shows objects using space-separated strings:
| example | outcome |
|---|---|
#q=cube |
show only object named cube (if any) |
#q=cube -ball_inside_cube |
show only object named cube but not child named ball_inside_cube |
#q=* -sky |
show everything except object named sky |
#q=-.language .english |
hide everything with class language, then show all class english objects |
#q=cube&rot=0,90,0 |
show only object cube and rotate the view |
#q=price:>2 price:<5 |
show only objects with custom property price with value between 2 and 5 |
It's simple but powerful syntax which allows css-like class/id-selectors with a searchengine prompt-style feeling:
- queries are showing/hiding objects only when defined as
srcvalue (prevents sharing of scene-tampered URL's). - queries are highlighting objects when defined in the top-Level (browser) URL (bar).
- search words like
cubeandfooin#q=cube fooare matched against 3D object names or custom metadata-key(values) - search words like
cubeandfooin#q=cube fooare matched against tags (BibTeX) inside plaintextsrcvalues like@cube{redcube, ...e.g. #equals#q=*- words starting with
.like.germanmatch class-metadata of 3D objects like"class":"german" - words starting with
.like.germanmatch class-metadata of (BibTeX) tags in XR Text objects like@german{KarlHeinz, ...e.g.
For example:
#q=.foois a shorthand for#q=class:foo, which will select objects with custom propertyclass:foo. Just a simple#q=cubewill simply select an object namedcube.
including/excluding
| operator | info |
|---|---|
* |
select all objects (only useful in src custom property) |
- |
removes/hides object(s) |
: |
indicates an object-embedded custom property key/value |
. |
alias for "class" :".foo" equals class:foo |
> < |
compare float or int number |
/ |
reference to root-scene. Useful in case of (preventing) showing/hiding objects in nested scenes (instanced by src) (*) |
* =
#q=-/cubehides objectcubeonly in the root-scene (not nestedcubeobjects)
#q=-cubehides both objectcubein the root-scene AND nestedskyboxobjects |
» example implementation » example 3D asset » discussion
Query Parser
Here's how to write a query parser:
- create an associative array/object to store query-arguments as objects
- detect object id's & properties
foo:1andfoo(reference regex:/^.*:[><=!]?/) - detect excluders like
-foo,-foo:1,-.foo,-/foo(reference regex:/^-/) - detect root selectors like
/foo(reference regex:/^[-]?\//) - detect class selectors like
.foo(reference regex:/^[-]?class$/) - detect number values like
foo:1(reference regex:/^[0-9\.]+$/) - expand aliases like
.foointoclass:foo - for every query token split string on
: - create an empty array
rules - then strip key-operator: convert "-foo" into "foo"
- add operator and value to rule-array
- therefore we we set
idtotrueorfalse(false=excluder-) - and we set
roottotrueorfalse(true=/root selector is present) - we convert key '/foo' into 'foo'
- finally we add the key/value to the store like
store.foo = {id:false,root:true}e.g.
An example query-parser (which compiles to many languages) can be found here
XR Fragment URI Grammar
reserved = gen-delims / sub-delims
gen-delims = "#" / "&"
sub-delims = "," / "="
Example:
://foo.com/my3d.gltf#pos=1,0,0&prio=-5&t=0,100
| Demo | Explanation |
|---|---|
pos=1,2,3 |
vector/coordinate argument e.g. |
pos=1,2,3&rot=0,90,0&q=.foo |
combinators |
Text in XR (tagging,linking to spatial objects)
How does XR Fragments interlink text with objects?
The XR Fragments does this by extracting a Word Graph (the XRWG) from the current scene, facilitated by Bib(s)Tex.
The (class)names end up in the Word Graph (XRWG), but what about text (inside an article e.g.)?
Ideally metadata must come with that text, but not obfuscate the text, or spawning another request to fetch it.
This is done by detecting Bib(s)Tex, without introducing a new language or fileformat
Why Bib(s)Tex? Because its seems to be the lowest common denominator for a human-curate-able XRWG (extendable by speech/scanner/writing/typing e.g, see further motivation here)
Hence:
- XR Fragments promotes (de)serializing a scene to the XRWG
- XR Fragments primes the XRWG, by collecting words from the
classand name-property of 3D objects. - XR Fragments primes the XRWG, by collecting words from optional metadata at the end of content of text (see default mimetype & Data URI)
- Bib's and BibTex are first class citizens for priming the XRWG with words (from XR text)
- Like Bibs, XR Fragments generalizes the BibTex author/title-semantics (
author{title}) into this points to that (this{that}) - The XRWG should be recalculated when textvalues (in
src) change - HTML/RDF/JSON is still great, but is beyond the XRWG-scope (they fit better in the application-layer)
- Applications don't have to be able to access the XRWG programmatically, as they can easily generate one themselves by traversing the scene-nodes.
- The XR Fragment focuses on fast and easy-to-generate end-user controllable word graphs (instead of complex implementations that try to defeat word ambiguity)
Example:
http://y.io/z.fbx | Derived XRWG (printed as BibTex)
----------------------------------------------------------------------------+--------------------------------------
| @house{castle,
+-[src: data:.....]----------------------+ +-[3D mesh]-+ | url = {https://y.io/z.fbx#castle}
| Chapter one | | / \ | | }
| | | / \ | | @baroque{castle,
| John built houses in baroque style. | | / \ | | url = {https://y.io/z.fbx#castle}
| | | |_____| | | }
| #john@baroque | +-----│-----+ | @baroque{john}
| | │ |
| | ├─ name: castle |
| | └─ class: house baroque |
+----------------------------------------+ |
[3D mesh ] |
| O ├─ name: john |
| /|\ | |
| / \ | |
+--------+ |
the
#john@baroque-bib associates both textJohnand objectnamejohn, with classbaroque
Another example:
http://y.io/z.fbx | Derived XRWG (printed as BibTex) ----------------------------------------------------------------------------+-------------------------------------- | +-[src: data:.....]----------------------+ +-[3D mesh]-+ | @house{castle, | Chapter one | | / \ | | url = {https://y.io/z.fbx#castle} | | | / \ | | } | John built houses in baroque style. | | / \ | | @baroque{castle, | | | |_____| | | url = {https://y.io/z.fbx#castle} | #john@baroque | +-----│-----+ | } | @baroque{john} | │ | @baroque{john} | | ├─ name: castle | | | └─ class: house baroque | +----------------------------------------+ | @house{baroque} [3D mesh ] | @todo{baroque} +-[remotestorage.io / localstorage]------+ | O + name: john | | #baroque@todo@house | | /|\ | | | ... | | / \ | | +----------------------------------------+ +--------+ |
> both `#john@baroque`-bib and BibTex `@baroque{john}` result in the same XRWG, however on top of that 2 classes (`house` and `todo`) are now associated with text/objectname/class 'baroque'.
As seen above, the XRWG can expand [bibs](https://github.com/coderofsalvation/hashtagbibs) (and the whole scene) to BibTeX.<br>
This allows hasslefree authoring and copy-paste of associations **for and by humans**, but also makes these URLs possible:
| URL example | Result |
|---------------------------------------|---------------------------------------------------------------------------|
| `https://my.com/foo.gltf#.baroque` | highlights mesh `john`, 3D mesh `castle`, text `John built(..)` |
| `https://my.com/foo.gltf#john` | highlights mesh `john`, and the text `John built (..)` |
| `https://my.com/foo.gltf#house` | highlights mesh `castle`, and other objects with class `house` or `todo` |
> [hashtagbibs](https://github.com/coderofsalvation/hashtagbibs) potentially allow the enduser to annotate text/objects by **speaking/typing/scanning associations**, which the XR Browser saves to remotestorage (or localStorage per toplevel URL). As well as, referencing BibTags per URI later on: `https://y.io/z.fbx#@baroque@todo` e.g.
The XRWG allows XR Browsers to show/hide relationships in realtime at various levels:
* wordmatch **inside** `src` text
* wordmatch **inside** `href` text
* wordmatch object-names
* wordmatch object-classnames
Spatial wires can be rendered, words/objects can be highlighted/scaled etc.<br>
Some pointers for good UX (but not necessary to be XR Fragment compatible):
9. The XR Browser needs to adjust tag-scope based on the endusers needs/focus (infinite tagging only makes sense when environment is scaled down significantly)
10. The XR Browser should always allow the human to view/edit the metadata, by clicking 'toggle metadata' on the 'back' (contextmenu e.g.) of any XR text, anywhere anytime.
12. respect multi-line BiBTeX metadata in text because of [the core principle](#core-principle)
13. Default font (unless specified otherwise) is a modern monospace font, for maximized tabular expressiveness (see [the core principle](#core-principle)).
14. anti-pattern: hardcoupling an XR Browser with a mandatory **markup/scripting-language** which departs from onubtrusive plain text (HTML/VRML/Javascript) (see [the core principle](#core-principle))
15. anti-pattern: limiting human introspection, by abandoning plain text as first class citizen.
> The simplicity of appending metadata (and leveling the metadata-playfield between humans and machines) is also demonstrated by [visual-meta](https://visual-meta.info) in greater detail.
## Default Data URI mimetype
The `src`-values work as expected (respecting mime-types), however:
The XR Fragment specification bumps the traditional default browser-mimetype
`text/plain;charset=US-ASCII`
to a hashtagbib(tex)-friendly one:
`text/plain;charset=utf-8;bib=^@`
This indicates that:
* utf-8 is supported by default
* lines beginning with `@` will not be rendered verbatim by default ([read more](https://github.com/coderofsalvation/hashtagbibs#hashtagbib-mimetypes))
* the XRWG should expand bibs to BibTex occurring in text (`#contactjohn@todo@important` e.g.)
By doing so, the XR Browser (applications-layer) can interpret microformats ([visual-meta](https://visual-meta.info)
to connect text further with its environment ( setup links between textual/spatial objects automatically e.g.).
> for more info on this mimetype see [bibs](https://github.com/coderofsalvation/hashtagbibs)
Advantages:
* auto-expanding of [hashtagbibs](https://github.com/coderofsalvation/hashtagbibs) associations
* out-of-the-box (de)multiplex human text and metadata in one go (see [the core principle](#core-principle))
* no network-overhead for metadata (see [the core principle](#core-principle))
* ensuring high FPS: HTML/RDF historically is too 'requesty'/'parsy' for game studios
* rich send/receive/copy-paste everywhere by default, metadata being retained (see [the core principle](#core-principle))
* netto result: less webservices, therefore less servers, and overall better FPS in XR
> This significantly expands expressiveness and portability of human tagged text, by **postponing machine-concerns to the end of the human text** in contrast to literal interweaving of content and markupsymbols (or extra network requests, webservices e.g.).
For all other purposes, regular mimetypes can be used (but are not required by the spec).<br>
## URL and Data URI
+--------------------------------------------------------------+ +------------------------+
| | | author.com/article.txt |
| index.gltf | +------------------------+
| │ | | |
| ├── ◻ article_canvas | | Hello friends. |
| │ └ src: ://author.com/article.txt | | |
| │ | | @book{greatgatsby |
| └── ◻ note_canvas | | ... |
| └ src:data:welcome human\n@book{sunday...} | | } |
| | +------------------------+
| |
+--------------------------------------------------------------+
The enduser will only see `welcome human` and `Hello friends` rendered verbatim (see mimetype).
The beauty is that text in Data URI automatically promotes rich copy-paste (retaining metadata).
In both cases, the text gets rendered immediately (onto a plane geometry, hence the name '_canvas').
The XR Fragment-compatible browser can let the enduser access visual-meta(data)-fields after interacting with the object (contextmenu e.g.).
> additional tagging using [bibs](https://github.com/coderofsalvation/hashtagbibs): to tag spatial object `note_canvas` with 'todo', the enduser can type or speak `#note_canvas@todo`
## XR Text example parser
Here's an example XR Text (de)multiplexer in javascript, which supports inline bibs & bibtex:
xrtext = {
expandBibs: (text) => {
let bibs = { regex: /(#[a-zA-Z0-9_+@-]+(#)?)/g, tags: {}}
text.replace( bibs.regex , (m,k,v) => {
tok = m.substr(1).split("@")
match = tok.shift()
if( tok.length ) tok.map( (t) => bibs.tags[t] = @${t}{${match},\n} )
else if( match.substr(-1) == '#' )
bibs.tags[match] = @{${match.replace(/#/,'')}}
else bibs.tags[match] = @${match}{${match},\n}
})
return text.replace( bibs.regex, '') + Object.values(bibs.tags).join('\n')
},
decode: (str) => { // bibtex: ↓@ ↓<tag|tag{phrase,|{ruler}> ↓property ↓end let pat = [ /@/, /^\S+[,{}]/, /},/, /}/ ] let tags = [], text='', i=0, prop='' let lines = xrtext.expandBibs(str).replace(/\r?\n/g,'\n').split(/\n/) for( let i = 0; i < lines.length && !String(lines[i]).match( /^@/ ); i++ ) text += lines[i]+'\n'
bibtex = lines.join('\n').substr( text.length )
bibtex.split( pat[0] ).map( (t) => {
try{
let v = {}
if( !(t = t.trim()) ) return
if( tag = t.match( pat[1] ) ) tag = tag[0]
if( tag.match( /^{.*}$/ ) ) return tags.push({ruler:tag})
if( tag.match( /}$/ ) ) return tags.push({k: tag.replace(/}$/,''), v: {}})
t = t.substr( tag.length )
t.split( pat[2] )
.map( kv => {
if( !(kv = kv.trim()) || kv == "}" ) return
v[ kv.match(/\s?(\S+)\s?=/)[1] ] = kv.substr( kv.indexOf("{")+1 )
})
tags.push( { k:tag, v } )
}catch(e){ console.error(e) }
})
return {text, tags}
},
encode: (text,tags) => {
let str = text+"\n"
for( let i in tags ){
let item = tags[i]
if( item.ruler ){
str += @${item.ruler}\n
continue;
}
str += @${item.k}\n
for( let j in item.v ) str += ${j} = {${item.v[j]}}\n
str += }\n
}
return str
}
}
The above functions (de)multiplexe text/metadata, expands bibs, (de)serialize bibtex (and all fits more or less on one A4 paper)
> above can be used as a startingpoint for LLVM's to translate/steelman to a more formal form/language.
str = ` hello world here are some hashtagbibs followed by bibtex:
#world #hello@greeting #another-section#
@{some-section} @flap{ asdf = {23423} }`
var {tags,text} = xrtext.decode(str) // demultiplex text & bibtex tags.find( (t) => t.k == 'flap{' ).v.asdf = 1 // edit tag tags.push({ k:'bar{', v:{abc:123} }) // add tag console.log( xrtext.encode(text,tags) ) // multiplex text & bibtex back together
This expands to the following (hidden by default) BibTex appendix:
hello world here are some hashtagbibs followed by bibtex:
@{some-section} @flap{ asdf = {1} } @world{world, } @greeting{hello, } @{another-section} @bar{ abc = {123} }
> when an XR browser updates the human text, a quick scan for nonmatching tags (`@book{nonmatchingbook` e.g.) should be performed and prompt the enduser for deleting them.
# HYPER copy/paste
The previous example, offers something exciting compared to simple copy/paste of 3D objects or text.
XR Text according to the XR Fragment spec, allows HYPER-copy/paste: time, space and text interlinked.
Therefore, the enduser in an XR Fragment-compatible browser can copy/paste/share data in these ways:
1. time/space: 3D object (current animation-loop)
1. text: TeXt object (including BibTeX/visual-meta if any)
1. interlinked: Collected objects by visual-meta tag
# Security Considerations
Since XR Text contains metadata too, the user should be able to set up tagging-rules, so the copy-paste feature can :
* filter out sensitive data when copy/pasting (XR text with `class:secret` e.g.)
# IANA Considerations
This document has no IANA actions.
# Acknowledgments
* [NLNET](https://nlnet.nl)
* [Future of Text](https://futureoftext.org)
* [visual-meta.info](https://visual-meta.info)
# Appendix: Definitions
|definition | explanation |
|----------------------|--------------------------------------------------------------------------------------------------------------------------------------|
|human | a sentient being who thinks fuzzy, absorbs, and shares thought (by plain text, not markuplanguage) |
|scene | a (local/remote) 3D scene or 3D file (index.gltf e.g.) |
|3D object | an object inside a scene characterized by vertex-, face- and customproperty data. |
|metadata | custom properties of text, 3D Scene or Object(nodes), relevant to machines and a human minority (academics/developers) |
|XR fragment | URI Fragment with spatial hints like `#pos=0,0,0&t=1,100` e.g. |
|src | (HTML-piggybacked) metadata of a 3D object which instances content |
|href | (HTML-piggybacked) metadata of a 3D object which links to content |
|query | an URI Fragment-operator which queries object(s) from a scene like `#q=cube` |
|visual-meta | [visual-meta](https://visual.meta.info) data appended to text/books/papers which is indirectly visible/editable in XR. |
|requestless metadata | metadata which never spawns new requests (unlike RDF/HTML, which can cause framerate-dropping, hence not used a lot in games) |
|FPS | frames per second in spatial experiences (games,VR,AR e.g.), should be as high as possible |
|introspective | inward sensemaking ("I feel this belongs to that") |
|extrospective | outward sensemaking ("I'm fairly sure John is a person who lives in oklahoma") |
|`◻` | ascii representation of an 3D object/mesh |
|(un)obtrusive | obtrusive: wrapping human text/thought in XML/HTML/JSON obfuscates human text into a salad of machine-symbols and words |
|BibTeX | simple tagging/citing/referencing standard for plaintext |
|BibTag | a BibTeX tag |
|(hashtag)bibs | an easy to speak/type/scan tagging SDL ([see here](https://github.com/coderofsalvation/hashtagbibs) which expands to BibTex/JSON/XML |