V2.0•July 27th, 2025•Still Evolving

Technical Walkthrough

A comprehensive technical deep-dive into the Constrained Asset Framework implementation, architecture, and code examples. This document represents the current state of the technology and will continue to evolve as development progresses.

1. Natural Language Processing Module

Core Architecture

The NLP module transforms natural language prompts into structured scene specifications using a multi-stage pipeline with semantic understanding and entity extraction.

Stage 1: Tokenization

Input text is normalized and tokenized using BERT-based tokenizers for robust handling of various language patterns.

Stage 2: Entity Recognition

Named Entity Recognition (NER) identifies objects, attributes, and spatial relationships using fine-tuned models.

Stage 3: Scene Graph Generation

Constructs a hierarchical scene graph representing objects and their relationships in 3D space.

Code Example: Text Processing

class NLPProcessor:
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained('bert-base')
        self.ner_model = pipeline('ner', model='custom-3d-ner')
        self.scene_parser = SceneGraphParser()
    
    def process_prompt(self, text: str) -> SceneSpec:
        # Normalize and tokenize
        tokens = self.tokenizer(text, return_tensors='pt')
        
        # Extract entities
        entities = self.ner_model(text)
        objects = [e for e in entities 
                  if e['label'] == 'OBJECT']
        attributes = [e for e in entities 
                     if e['label'] == 'ATTRIBUTE']
        
        # Build scene specification
        scene_spec = SceneSpec()
        for obj in objects:
            scene_spec.add_object(
                type=obj['word'],
                attributes=self._match_attributes(
                    obj, attributes),
                position=self._infer_position(
                    obj, tokens),
                rotation=self._infer_rotation(
                    obj, tokens)  # [x, y, z] in degrees
            )
        
        return scene_spec

2. Asset Selection & Constraint Engine

Asset Library Architecture

Asset Metadata Structure

{
  "assetId": "VEND_001",
  "type": "vending_machine",
  "metadata": {
    "polyCount": 4500,
    "textureSize": [2048, 2048],
    "lodLevels": [4500, 2000, 500],
    "semanticTags": ["japanese", "modern", "beverage", "interactive"],
    "dimensions": {"width": 1.2, "height": 2.0, "depth": 0.8},
    "anchorPoints": [
      {"type": "ground", "position": [0, 0, 0]},
      {"type": "interaction", "position": [0.6, 1.0, 0.8]}
    ]
  },
  "performance": {
    "mobile": {"maxInstances": 3, "lodLevel": 2},
    "desktop": {"maxInstances": 10, "lodLevel": 0},
    "vr": {"maxInstances": 5, "lodLevel": 1}
  },
  "licensing": {
    "type": "CC0",
    "attribution": false,
    "commercial": true
  }
}

Constraint Types

Performance: Polygon budget, texture memory, draw calls
Semantic: Style consistency, theme matching, era appropriateness
Spatial: Collision bounds, placement rules, scale limits
Legal: License compatibility, content rating, regional restrictions

Selection Algorithm

def select_asset(request, constraints):
    candidates = asset_db.query(
        type=request.type,
        tags=request.tags
    )
    
    scored = []
    for asset in candidates:
        score = 0
        score += semantic_match(asset, request)
        score += performance_fit(asset, constraints)
        score += style_coherence(asset, scene)
        scored.append((score, asset))
    
    return max(scored, key=lambda x: x[0])[1]

Fuzzy Matching System

When exact matches aren't available, a fuzzy matching system finds the best alternatives using semantic embeddings and multi-factor scoring.

class FuzzyMatcher:
    def __init__(self):
        self.encoder = CLIPModel.from_pretrained('openai/clip-vit-base')
        self.index = faiss.IndexFlatL2(512)  # CLIP embedding dimension
        
    def find_similar(self, query: str, k: int = 5) -> List[Asset]:
        # Encode query to embedding
        query_embedding = self.encoder.encode_text(query)
        
        # Search in vector database
        distances, indices = self.index.search(query_embedding, k)
        
        # Apply additional filters
        results = []
        for idx, distance in zip(indices[0], distances[0]):
            asset = self.asset_db[idx]
            if self.validate_constraints(asset):
                results.append({
                    'asset': asset,
                    'similarity': 1 / (1 + distance),
                    'fallback_reason': self.get_fallback_reason(query, asset)
                })
        
        return results

3. Spatial Layout Generator

Physics-Based Placement

The spatial layout system ensures physically plausible object placement using a combination of physics simulation and rule-based constraints.

Placement Rules

{
  "rules": {
    "gravity": {
      "enabled": true,
      "direction": [0, -9.81, 0]
    },
    "collisions": {
      "mode": "prevent_overlap",
      "margin": 0.1
    },
    "surfaces": {
      "table": ["small_objects", "decorative"],
      "floor": ["furniture", "large_objects"],
      "wall": ["paintings", "signs", "mounted"]
    },
    "relationships": {
      "chair": {"near": ["table", "desk"]},
      "lamp": {"above": ["table"], "near": ["seating"]}
    }
  }
}

Hierarchical Scene Graph

class SceneNode:
    def __init__(self, asset_id: str):
        self.asset_id = asset_id
        self.transform = Transform()  # position, rotation, scale
        self.children = []
        self.constraints = []
        
    def add_child(self, node: 'SceneNode'):
        self.children.append(node)
        node.parent = self
        
    def update_transform(self):
        # Apply local transform
        world_transform = self.transform
        
        # Combine with parent transform
        if self.parent:
            world_transform = (
                self.parent.world_transform * 
                self.transform
            )
            
        # Apply constraints
        for constraint in self.constraints:
            world_transform = constraint.apply(
                world_transform
            )
            
        return world_transform

Transform Pipeline

1. Initial placement based on semantic hints
2. Physics simulation for gravity and collisions
3. Constraint satisfaction for spatial rules
4. Optimization for visual composition
5. Final validation and bounds checking

Collision Detection & Resolution

Broad Phase

Spatial hashing with dynamic octree for rapid AABB intersection tests.

O(n log n) average case complexity

Narrow Phase

GJK algorithm for precise convex hull collision detection with EPA for penetration depth.

Sub-millimeter precision at 60 FPS

4. Runtime Rendering System

Scene Output Format

{
  "sceneId": "scene_tokyo_001",
  "timestamp": "2025-07-27T10:30:00Z",
  "objects": [
    {
      "id": "obj_001",
      "assetId": "VEND_001",
      "type": "vending_machine",
      "position": [2.3, 0, -1.0],
      "rotation": [0, 45, 0],      // Euler angles in degrees
      "scale": [1, 1, 1]
    },
    {
      "id": "obj_002",
      "assetId": "NEON_001",
      "type": "neon_sign",
      "position": [1.5, 3, 0],
      "rotation": [0, 180, 0],
      "scale": [1.2, 1.2, 1]
    },
    {
      "id": "obj_003",
      "assetId": "LAMP_003",
      "type": "street_lamp",
      "position": [-1.2, 0, 2.4],
      "rotation": [0, 90, 0],
      "scale": [1, 1, 1]
    }
  ],
  "environment": {
    "lighting": "neon_night",
    "weather": "rain",
    "time": "night"
  }
}

Progressive Loading Pipeline

Multi-Resolution Streaming

class ProgressiveLoader:
    def __init__(self, scene: Scene):
        self.scene = scene
        self.load_queue = PriorityQueue()
        self.cache = LRUCache(max_size=100 * 1024 * 1024)  # 100MB
        
    async def load_scene(self, viewport: Viewport):
        # Stage 1: Load bounding boxes (instant)
        for obj in self.scene.objects:
            self.render_bbox(obj)
            
        # Stage 2: Load low-res proxies (< 100ms)
        proxies = await self.load_proxies(self.scene.objects)
        self.render_proxies(proxies)
        
        # Stage 3: Progressive quality enhancement
        for obj in self.prioritize_visible(self.scene.objects, viewport):
            lod_level = self.calculate_lod(obj, viewport)
            
            # Load appropriate LOD
            if not self.cache.has(obj.id, lod_level):
                asset = await self.fetch_asset(obj.id, lod_level)
                self.cache.store(obj.id, lod_level, asset)
                
            self.render_object(self.cache.get(obj.id, lod_level))
            
            # Yield to maintain 60 FPS
            if self.frame_budget_exceeded():
                await self.next_frame()

Compression

• Draco geometry compression
• Basis universal textures
• GZIP for JSON metadata
• 70-90% size reduction

Caching Strategy

• CDN edge caching
• Browser IndexedDB
• Memory LRU cache
• Predictive preloading

Platform Optimization

• WebGL 2.0 / WebGPU
• Instanced rendering
• Frustum culling
• Occlusion queries

Performance Monitoring

class PerformanceMonitor:
    def __init__(self):
        self.metrics = {
            'fps': RollingAverage(60),
            'frame_time': RollingAverage(60),
            'draw_calls': 0,
            'triangles': 0,
            'texture_memory': 0
        }
        
    def adapt_quality(self):
        if self.metrics['fps'].avg < 30:
            self.reduce_quality()
        elif self.metrics['fps'].avg > 55:
            self.increase_quality()
            
    def reduce_quality(self):
        # Reduce shadow resolution
        self.shadow_resolution *= 0.5
        # Increase LOD bias
        self.lod_bias += 1.0
        # Disable post-processing
        self.post_processing = False

WebXR Integration

Native support for VR/AR experiences with automatic performance scaling and comfort optimizations.

VR Optimizations

• Foveated rendering with eye tracking
• Fixed foveated rendering fallback
• Instanced stereo rendering
• Asynchronous reprojection
• Dynamic resolution scaling

5. Implementation Examples

Complete Integration Example

JavaScript/TypeScript Implementation

import { SceneGenerator, Constraints, Platform } from '@kanighta/scene-generator';

// Initialize the generator
const generator = new SceneGenerator({
  apiKey: 'your-api-key',
  assetLibrary: 'https://cdn.kanighta.com/assets',
  platform: Platform.WEB
});

// Define platform constraints
const constraints = new Constraints({
  maxPolygons: 100000,
  maxTextureMemory: 256 * 1024 * 1024,  // 256MB
  maxDrawCalls: 100,
  targetFPS: 60,
  platform: {
    mobile: { maxPolygons: 50000 },
    vr: { minFPS: 90 }
  }
});

// Generate scene from prompt
async function generateScene(prompt: string) {
  try {
    // Process the prompt
    const scene = await generator.generate(prompt, constraints);
    
    // Get the scene as different formats
    const gltf = await scene.toGLTF();
    const usd = await scene.toUSD();
    const json = scene.toJSON();
    
    // Render in Three.js
    const threeScene = await scene.toThreeJS();
    renderer.render(threeScene, camera);
    
    // Enable real-time editing
    scene.on('objectSelected', (obj) => {
      console.log('Selected:', obj);
      // Show transform controls
    });
    
    scene.on('objectModified', (obj, changes) => {
      console.log('Modified:', obj, changes);
      // Auto-save to cloud
    });
    
  } catch (error) {
    console.error('Generation failed:', error);
    // Fallback to cached scene
    const fallback = await generator.getFallbackScene(prompt);
    return fallback;
  }
}

Unity Integration

using KanightaSceneGenerator;
using UnityEngine;
using System.Threading.Tasks;

public class SceneGeneratorUnity : MonoBehaviour
{
    private SceneGenerator generator;
    
    async void Start()
    {
        // Initialize with Unity-specific settings
        generator = new SceneGenerator(new Config
        {
            ApiKey = "your-api-key",
            Platform = Platform.Unity,
            RenderPipeline = RenderPipeline.URP
        });
        
        // Generate scene
        var scene = await generator.GenerateAsync(
            "Medieval castle throne room with torches"
        );
        
        // Instantiate in Unity
        foreach (var obj in scene.Objects)
        {
            var prefab = await AssetLoader.LoadAsync(obj.AssetId);
            var instance = Instantiate(prefab);
            
            instance.transform.position = obj.Position.ToUnity();
            instance.transform.rotation = Quaternion.Euler(obj.Rotation.ToUnity()); // Convert [x,y,z] degrees to quaternion
            instance.transform.localScale = obj.Scale.ToUnity();
            
            // Add physics if needed
            if (obj.Physics.Enabled)
            {
                var rb = instance.AddComponent<Rigidbody>();
                rb.mass = obj.Physics.Mass;
                rb.useGravity = obj.Physics.UseGravity;
            }
        }
    }
}

React Component

import React, { useState, useEffect } from 'react';
import { Canvas } from '@react-three/fiber';
import { SceneGenerator } from '@kanighta/scene-generator';
import { OrbitControls, Environment } from '@react-three/drei';

export function GeneratedScene({ prompt }) {
  const [scene, setScene] = useState(null);
  const [loading, setLoading] = useState(true);
  
  useEffect(() => {
    const generator = new SceneGenerator({
      apiKey: process.env.REACT_APP_KANIGHTA_API_KEY
    });
    
    generator.generate(prompt)
      .then(generatedScene => {
        setScene(generatedScene);
        setLoading(false);
      })
      .catch(error => {
        console.error('Failed to generate scene:', error);
        setLoading(false);
      });
  }, [prompt]);
  
  if (loading) return <div>Generating scene...</div>;
  if (!scene) return <div>Failed to generate scene</div>;
  
  return (
    <Canvas camera={{ position: [0, 5, 10] }}>
      <ambientLight intensity={0.5} />
      <directionalLight position={[10, 10, 5]} />
      
      {scene.objects.map((obj, index) => (
        <SceneObject 
          key={index} 
          data={obj}
          position={obj.position}  // [x, y, z]
          rotation={[obj.rotation[0] * Math.PI/180, obj.rotation[1] * Math.PI/180, obj.rotation[2] * Math.PI/180]}  // Convert degrees to radians
        />
      ))}
      
      <Environment preset={scene.environment} />
      <OrbitControls />
    </Canvas>
  );
}

6. Performance Optimizations

Rendering Pipeline

Draw Call Batching

Automatic instancing for repeated objects with GPU instancing support.

1000 objects → 10 draw calls

Texture Atlasing

Dynamic texture atlas generation reduces texture switches.

50 textures → 2 atlas textures

Level of Detail

Automatic LOD selection based on screen coverage and performance.

Distance-based + performance-adaptive

Memory Management

Caching Hierarchy

1. GPU Memory: Active textures and meshes
2. RAM Cache: Recently used assets
3. IndexedDB: Persistent browser storage
4. CDN Cache: Edge server distribution
5. Origin: Master asset repository

Predictive Loading

// ML-based prediction
const predictor = new AssetPredictor();
predictor.train(userHistory);

const likely = predictor.predict(currentScene);
for (const assetId of likely) {
  cache.preload(assetId, Priority.LOW);
}

Platform-Specific Optimizations

Mobile

• Reduced polygon counts
• Compressed textures
• Simplified shaders
• Touch-optimized controls

Desktop

• Full quality assets
• Advanced shaders
• Real-time shadows
• Post-processing effects

VR/AR

• 90+ FPS target
• Foveated rendering
• Stereo instancing
• Comfort settings

Cloud

• Server-side rendering
• Video streaming
• Unlimited quality
• Ray tracing support

🚧 Working Prototype Under Development

A fully functional prototype implementing all the technical components described in this walkthrough is currently under active development.

Expected Availability: Q3/Q4 2025

Back to Overview