In my last post on Augmented Reality 101, i had shown a sample of how the FLARToolkit outputs using Flash. Here i would like to share the code i had written for that sample  so that u know what are all the modules that u need to import from the various packages, (from FLAR and Papervision) and what are all the classes that need to be defined to get this to work. You will obviously need to replace the model file and the marker file to a custom one, rest of it is a good skeleton code to build something for AR.

The below code may require you to have the FLEX SDK 3.0 Library – Download from HERE.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

package {
 
	import flash.display.Bitmap;
	import flash.display.BitmapData;
	import flash.display.PixelSnapping;
	import flash.display.Sprite;
	import flash.media.Camera;
	import flash.media.Video;
	import flash.events.Event;
	import flash.utils.ByteArray;
 
	//FLAR IMPORT
	import org.libspark.flartoolkit.core.FLARCode;
	import org.libspark.flartoolkit.core.param.FLARParam;
	import org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;
	import org.libspark.flartoolkit.detector.FLARSingleMarkerDetector;
 
	//PAPERVISION IMPORT SCENE GLOBALS
	import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;
	import org.libspark.flartoolkit.support.pv3d.FLARBaseNode;
	import org.libspark.flartoolkit.support.pv3d.FLARCamera3D;
	import org.papervision3d.render.LazyRenderEngine;
	import org.papervision3d.scenes.Scene3D;
	import org.papervision3d.view.Viewport3D;
 
	//PAPERVISION IMPORT OBJECTS GLOBALS
	import org.papervision3d.objects.parsers.DAE;
	import org.papervision3d.lights.PointLight3D;
	import org.papervision3d.materials.WireframeMaterial;
	import org.papervision3d.materials.utils.MaterialsList;
	import org.papervision3d.materials.ColorMaterial;
	import org.papervision3d.materials.shadematerials.FlatShadeMaterial;
	import org.papervision3d.materials.shadematerials.PhongMaterial;
	import org.papervision3d.events.FileLoadEvent;
 
	//INITIALIZE SWF PARAMS
	[SWF(width="640", height="480", frameRate="30", backgroundColor="#FFFFFF")]
 
	public class ArApp extends Sprite
	{
		//EMBED MARKER FILE (NEEDS FLEX SDK 3)
		[Embed(source="Data/crunkMarker16.pat", mimeType="application/octet-stream")]
		private var pattern:Class;
 
		//EMBED CAMERA PARAMS FILE (NEEDS FLEX SDK 3)
		[Embed(source="Data/camera_para.dat", mimeType="application/octet-stream")]
		private var params:Class;
 
		//FLAR VARS
		protected var Param:FLARParam;
		protected var Code:FLARCode;
		protected var raster:FLARRgbRaster_BitmapData;
		protected var detector:FLARSingleMarkerDetector;
		protected var resultMat:FLARTransMatResult = new FLARTransMatResult();
 
		//WEBCAM DATA VARS
		protected var webcamBox:Camera;
		protected var video:Video;
		protected var capture:BitmapData;
 
		//PAPERVISION VARS
		protected var viewport:Viewport3D;
		protected var camera3d:FLARCamera3D;
		protected var scene:Scene3D;
		protected var renderer:LazyRenderEngine;
		protected var markerNode:FLARBaseNode;
 
		//EXTERNAL MODEL VARS
		private var ext_model:DAE;
 
		public function ArApp() {
 
			doArApp();
			addEventListener(Event.ENTER_FRAME, loop);
		}
 
		private function doArApp():void
		{
 
		// FLAR CONSTRUCTOR
		Param = new FLARParam();
		Param.loadARParam(new params() as ByteArray);
		Code = new FLARCode(16, 16);
		Code.loadARPatt(new pattern());
 
		//WEBCAM DATA
		webcamBox = Camera.getCamera();
			if (!webcamBox) {
				trace ('No webcam!!!!');
				throw new Error('No webcam!!!!');
 
			}
 
		webcamBox.setMode(640, 480, 30);
		video = new Video(640, 480);
		video.attachCamera(webcamBox);
		addChild(video);
 
		//BITMAP DATA
		capture = new BitmapData(640, 480);
		capture.draw(video);
		raster = new FLARRgbRaster_BitmapData(capture);
		detector = new FLARSingleMarkerDetector(Param, Code, 80);
 
		//PV3D DATA
		camera3d = new FLARCamera3D(Param);
		scene = new Scene3D();
		markerNode = scene.addChild(new FLARBaseNode()) as FLARBaseNode;
		viewport = new Viewport3D();
		renderer = new LazyRenderEngine(scene, camera3d, viewport);
 
		//OBJECT BASED VARS - LIGHTS, MATERIALS, MODEL.
		var light:PointLight3D = new PointLight3D();
		light.x = 0;
		light.y = 1000;
		light.z = -1000;
 
		var ar_materials:MaterialsList = new MaterialsList({all: new PhongMaterial(light,0xffffff,0x000000,0.3)});
 
		ext_model = new DAE();
		ext_model.load('model/dastardly2.dae',ar_materials);
 
		//CUSTOM OBJECT TRANSFORM
		ext_model.scale = 10;
		ext_model.rotationX = 90;
		markerNode.addChild(ext_model);
		addChild(viewport);
 
		}
 
		private function loop(e:Event):void {
			capture.draw(video);
 
			var detected:Boolean = false;
			try {
				detected = detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5;
			} catch (e:Error) {}
 
			if (detected) {
				detector.getTransformMatrix(resultMat); //GET
				markerNode.setTransformMatrix(resultMat); //SET
				markerNode.visible = true; //GO
			} else {
				markerNode.visible = false; //REMOVE IF MARKER NO IN FRAME
			}
			renderer.render(); //RENDER THE SCENE
		}
 
	}
}

Augmented Reality Test.
1. Download the Marker File Here.. crunkMarker2.png

2. To Watch Lores Model Version CLICK HERE(Requires a Webcam) .
This lores model is 16798 Triangles… (1.02MB – Please give it time to Load).

3. To Watch Hires Model Version CLICK HERE(Requires a Webcam).
This hires model is 68412 Triangles… (5.06MB – Please give it time to Load)

——————————————————————————————————————————————

Recently i have been seeing lots of hype happening around the AR (Augmented Reality) Tech… so as i was looking around the web i came across this amazing Action-script Class library made by Saqoosha called FLARToolKit

History : FLARToolKit is an AS3 ported version of NyARToolkit, which is a Java ported version of ARToolKit.
ARToolKit was originally developed by Dr. Hirokazu Kato, and its ongoing development is being supported by the Human Interface Technology Laboratory (HIT Lab) at the University of Washington, HIT Lab NZ at the University of Canterbury, New Zealand, and ARToolworks, Inc, Seattle.

FLARToolKit recognize the marker from input image. and calculate its orientation and position in 3D world which equals to doing all the hardwork as far as AR Technology is concerned.

The FLARToolKit currently works with a variety of Flash 3D Engines like Alternativa3D, Away3D, Papervision3D, and Sandy3D etc. my personal choice here is papervision 3d since i have used it before and through what ive seen is just plainly amazing :)

So for this example i have used…

Flash as the main program here loaded with the FLARToolKit Library,
Papervision3D 2.0 for the viewport rendering, lighting and 3d geometry display.
Blender for making Geometry and exporting Collada (dae) files.
ARToolkit Marker Gen for the custom marker generation.

——————————————————————————————————————————————

The FLAR theory process follows some very simple steps.

1. Flash Displays Webcam Data – Since u cant do AR without a Webcam.. the first step is to get ur webcam feed in Flash  through the Camera.getCamera();

2. Applies Camera Correction – FLARToolKit  comes with a binary camera parameters file which it loads to correct lens distortion from your webcam. Calib_camera2 is the software used to generate this file and it is existent from the original ARToolKit. Most examples u see online state that the default camera_para.dat that comes with the Toolkit works for everything.

3. Finding Matching Marker – The Marker is a .pat (pattern file) that FLAR will continuously be scanning for. The pattern file consists of four 16×48 matrices, one for each direction and 48 is for RGB color information. There is an Air app created by Saqoosha for easy marker creation here..

4. Marker is Detected – As the marker detector basically basically scan through the bitmapData from the web cam feed and comparing it to the provided marker pattern from the marker file. On marker detection, FlarToolkit will will built a transformation matrix around the marker and create a Flar Base Node to sit on top of it.

5. FLAR Base Node - The FLAR base node is just a ground plane of which all the geometry u want to show will be created, The transformation matrix will hold all the data required to properly orient ur 3d data in comparison to marker orientation.

6. 3D Render Engine – And finally the visual part , at this level with a bunch of papervision classes , like viewport3d and lights, we can finally display the desired geometry.

——————————————————————————————————————————————

here are screenshots of the actually model, with color applied :)

Background Information
________________________________________________________________________________________________________________

All CGI images are made of fundamental building blocks called pixels. a pixel is the smallest breakdown of an image, and can only be made of one one color. When an image is rendered, the renderer needs to assign a single color to each pixel of the overall image depending on what object that pixel represents. This standard process brings up some problem as to what we call aliasing. where thin lines or diagonal lines are present in an image. To overcome this entire thing, we use a process called Anti-Aliasing. Here each pixel is multi-sampled i.e each pixel is broken down into multiples smaller parts which equals to more color information and then all the samples are averaged out using a Anti-alias filter, so as to obtain the final one color per pixel value.

________________________________________________________________________

Sampling Methods
________________________________________________________________________

Fixed Sampling - Uses a fixed number of samples per pixel when processing an image.
Adaptive Sampling - The number of samples used per pixel varies depending on the contrast of your scene. Either vary by a fixed distance
Custom Sampling - Custom Sampling allows you to tune the Min Sample Level and Max Sample Level independently, while retaining true adaptive sampling.

________________________________________________________________________

Anti-aliasing Filters
________________________________________________________________________

Box Filter – A pretty Balanced Speed/Quality Filter. Gets you the fastest results, Due to it having a Step Curve the sample contributions are always taken from the center of the pixel.

Area Filter – A little more advanced than the Box Filter in the case that it blur the image slighty.

Triangle/Tent Filter – Overall SlowerSpeed/Better Quality than the box filter. In this case samples at the center have maximum contribution while samples further away from the center have a leanear falloff to the contribution.

Gaussian Filter – Produces Best Quality/Slowest Speed. Gauss uses a curved fall-off for determining sample contributions. The contribution falloff from the center to the edge of the pixel is smooth. Gauss requires a higher filter size than normal due to which the images get a little blurry.

Mitchell & Lanczos – Both are variants to the Gauss, but they tend to increase the overall contrast and the sharpness of the image. A good allround filter that gives reasonable sharpness

Quadratic B-Spline Filter - Produces Good Quality at speeds faster than the Gauss Filter. Compared to the Gauss filter the contribution from the center of the pixel does not peak but archieves a more quadratic curve.

CatmullRom Filter - Speed Same as Mitchell and Lanczos  but Catmull-Rom filter, gives the most sharpening. the contribution curve travels also in negative to give best overall contribution.

Sinc Filter - i dont know how this works.. if someone find out please let me know

I recently discovered this not so new software called geoControl which has just come out with version 2.. and has added a bunch of new features. I have seen everything in the software yet. especially the new isolines features, but for the general aspect of terrain generation was quite strong,

Below is an examples that i rendered of the output of geoControl, very basic stuff.. made from a DEM map which i converted to terragen TER file , and then got into geocontrol2 to render, after tweaking various options inside the software i managed to get a lot of different variations.. i wish the software was more interactive on render, but right now its like u do a bunch of slider changes and then hit render, even u rotate the model. i tried 1024 resolution and was very in satified with the results, maybe its made for some other purpose.  All the below images are with a 2048 texture/displacement  resolution. which takes about 10-15 mins to render.

Somehow the thumbnails look blur.  Click on the images for bigger sharper preview.

MORE LINKS
——————————————

GeoControl 2 Offical Website – http://www.geocontrol2.com/
GeoControl 2 Features -  http://www.geocontrol2.com/e_geocontrol_geocontrol2.htm

I finally managed to get my hands on a trial version of mudbox today morning, and decided to do some stuff so that i could see whats new, or wheater it was worth the hype. Well for the first thing, it doesnt look very different from the last version other than the name Autodesk and some new tabs, well for one thing autodesk has managed to keep the same simplicity of use that the original mudbox gave, but there are still a lot of things i would like to see in it, other than going into maya everytime u need a small change.

So, after about 6hrs of dooling, here is what i came up with

• Well, the new mudbox definately has a speed upgrade, in both sculpting and viewport interactivity
• my final mesh was to 7 Levels that is abt 75,29,656 Polygons or 7 Million Polygons, but yet sculpting was very interactive, was gettin abt 37FPS in the viewport,
• this new version of mudbox uses the power of next generation graphics cards to show realtime shadows and reflections, so it can turn ur majorly slow down ur machine when the use any of these facny new options,
• i did not seem to find a button to render the current image, that is why all the above images are screeshots 0_0.
• One of the weird things i noticed quite late was, the saved file size, seems to be much bigger now, i wonder what all it saves, cause the last saved .mud file takes about 819MB, and i have already saved around 15files before i realized this, so today mudbox project takes around 6.5GB of HDD space for me.
• I personally feel they could have done something better of the default shader, (the one that u see in the render) cause its really a weird shader.
• I did not go to far out in checking all the new features, somehow i got quite irritated with a lot of missing features realted to Modelling and Sculpting itself.
• Mudbox really needs a posing solution, it quite a problem to get ur files from another software and then see it in mudbox and then suddenly u dont like it , and u need to repeat the process.

For those who dont know what HLSL (High Level Shader Language) is its a propreoraty shading language made by Micrsoft for use with Direct3D API.

Autodesk FBX (FilmBox) format is currently the standard format for game assests, including most games now developed for the XBOX 360.

Note : Since i originally come from using 3d artist tools, i will be refrencing the common simplicities that are taken for granted. other than starting out with a blank pad in the game programming world :)

• Before we get into shading we need something to shade.. namely a model, and its not as simple as saying file>Open.. in the game programming world, new objects need to be drawn onto the viewable space, which includes itterating through every seperate piece of geometry that a single character is made of. generally speaking, when u need to draw something in 2d or 3d, u obviously need something to draw it on u cant draw in thin air, so u need to build the space on which u can draw. to archieve this we need 3 matrices namely the world, view, and projection matrix.

World Matrix - Is basically defining the origin of the worldspace, a point from which everyother object can form a relation, or be relative too.

View Matrix -  is the position of the camera in 3d space i.e XYZ values, the camera target and a camera up vector which defines the up director for the camera.

Projection Matrix - is the behavior of a world matrix to the view matrix, simply speaking its the calcuation between the world as seen through the camera, which could be orthagonal or perspective, properties like the field of view, aspect ratio ( viewport width / viewport height ), near and Far ClipPlaneDIstance.

Gettin these three things in place will let you successfully load up anykind of model for basic shading purposes.

• A Background on shading in the computer world, is that all graphical data is vector based data, meaning that if this data was to be rendered as is, u would get a huge block of continiously flowing numbers, Yes exactly like the matrix, expections that it would be green.. so to see this data in the form that we humans find more apealling, that is through pictures, this vecotor needs to be converted to pixel based information, a process nowdays commonly known as rasterization, in doing so the processor read every vector data and calculates the equivalent pixel based value, to show the final shaded model.

Now we can get down to the actual HLSL ShadingModel, a standard file structure consists of 4 major parts the Input(Incoming Vertex Data), Vertex Shader Functions, Output, Pixel Shader Functions, Techniques.

Input(Incoming Vertex Data) – This is where all the incoming vertex data is stored for futther calculation, this includies data from other textures, models, cameras, matrixes etc. it is basically a storage block ( Buffer ) for migrating data.

Vertex Shader Functions – This is where all the various methods for processing the input data reside, with a combination of these functions, a variety of results can be archieved, like calculating object normal space, or normals in relation to light, basically all kind of vector data. 

Output(Outgoing Vertex Data)  - This is again a buffer for vertex data, the only difference is that this data has already been worked on by the vertex shader functions, and is now ready to go to the next level which is processing by the pixel shader class.

Pixel Shader Functions – This is where all the calculated vector data, is re-calculated with pixel based logic, so that the realtime renderer can finally get the needed values to display the required object in worldspace.

Techniques - A technique is a function where a specific vector and pixel rendering function can be put together as a combination of different varities of rendering. like if u had an object that required normal vector rendering and another didnt not require that logic, then u could write 2 techniques, where 1 would have a function for calculating normal data and the other would be for stuff that didnt need normal calculations.

• Below is the DirectX API to more define the ShadingModel Pipeline.

The Microsoft Direct3D 10 API defines a process to convert a group of vertices, textures, buffers, and state into an image on the screen. This process is described as a rendering pipeline with several distinct stages.

The different stages of the Direct3D 10 pipeline are:

1. Input Assembler: Reads in vertex data from an application supplied vertex buffer and feeds them down the pipeline.
2. Vertex Shader: Performs operations on a single vertex at a time, such as transformations, skinning, or lighting.
3. Geometry Shader: Processes entire primitives such as triangles, points, or lines. Given a primitive, this stage discards it, or generates one or more new primitives.
4. Stream Output: Can write out the previous stage’s results to memory. This is useful to recirculate data back into the pipeline.
5. Rasterizer: Converts primitives into pixels, feeding these pixels into the pixel shader. The Rasterizer may also perform other tasks such as clipping what is not visible, or interpolating vertex data into per-pixel data.
6. Pixel Shader: Determines the final pixel colour to be written to the render target and can also calculate a depth value to be written to the depth buffer.
7. Output Merger: Merges various types of output data (pixel shader values, alpha blending, depth/stencil…) to build the final result.

• Now getin down to  some shader logic,

Standard Shader – The major re-quiste for texutures  is UV which is available to us in the FBX model, but that information needs to be imported into the Shader Logic, that is go by itterating through the texutre co-ordinates and wraping it around the vector data and continously transorming that with the skin transform data, so as to always keep the data live and updated with new current snapshots of the model. Then that vecor data needs to be sent to the pixel shader, so that the vector is then compared with the texture file, to produced the write colors in the write space.

Diffuse Shading / Lighting - Diffuse shading is based on a principle that a surface is brightest when its facing the light and it tapers into darkness as is faces away from the light, basically if we normalize a vector, we end up keeping its direction, but the magnitude gets set to a value of 1, and this makes it a unit vector, the dot product of two unit vectors eqauls to the cos of the angle between the two vectors, —  dot( unit vector , unit vector ) = cos ( angle )  —    so with this we can determine if the surface is facing the light based on the result of the dot product, a value of 1 means that it is dirrectly facing the light and a value of -1 is that its in total darkness. so 1 is the brightest and -1 is the darkest area. the cos(angle) is a floating point value.

Normal Map Shading – Normal Mapping works on the principle that we do not need to use the normals from the vertices of a mesh, like in a generic vertex shader, in this case we can use the normals information stoed in an image. Normals contain XYZ information, and a normal map stores theses XYZ co-ordinates in the form of RGB color data, so in a normal shader, it basically overwrites the low number of normals derieved from the mesh, which a higher number which is then derieved from the normal map. The normal map can be generated from a high resolution object and can be used to shade a low detail mesh. normal maps are calculated in tangent space. 

Specular Shading – For specularity, light is reflected from a surface to the camera or eye, it is always dependent on these two factors, whihc is the reflected light and the placement of the camera, in shader terms this would be the light reflection direction and the view(eye) vector, calulating reflected light can be a very intensive method, since it is depended on various bounce levels, so James F. Blinn (1977) devised something called as a Half Vector. THe Half Vector is the vector which is between the light and the eye, and the angle between the vector and the normal, is the same as the angle between the reflected light and the eye. This process is much quicker to calculate. For more information on Half vecotor u can see Models of light reflection for computer synthesized pictures by  James F. Blinn. but thats not enough, speuclar depends on how smooth of rough an object really is, and thats where the specular exponent calculation comes in.

Reflection Shading – For reflection, we need a reflection vector, which can be archieved by calculating what the camera is seeing based of the normals. This then will reflect a specified texture data in form of a cube map, so as to fake the surroundings of the world, in realtime. On a more complex level these maps are also  gnenerated in realtime so as to continously be updated with the environment that the object resides in.

Simple Shadows - For basic shadows, we need to make a shadow matrix that will draw a flattened version fo the object on a ground plane, the object can then be shaded with a constant color, so that it can look like a shadow on a ground plane. This can be quite an overkill at times, since it is basically making an instance of the character, and it still needs to calculate another set of vertices for the shadow.

Stencil Buffer Shadows – The Flexibility of the stencil mesh is not to draw the shadow as a flattened mesh but to draw its shape like a sillothettes. the shadow matrix will still calculate a flattened version fo the object, but when the shadow needs to be draw, it will go throught a stencil mode, in which the pre-processor will check to see if there is overlapping of shadow pixels or no, so as to always keep only one value to a shadow pixel at any given time. This is also a better option for alpha blending, since there is no overlapping of information, it would not tend to jitter in realtime.