The Study: There and Back Again

GPU

The GPU or graphics processing unit the portion of the computer we care the most about. It has a programmable pipeline, which we can use to alter how it processes data. It is highly parallel meaning it can process multiple tasks at the same time very quickly.

Graphics Pipeline

Vertex Array -> Vertex Shader -> Triangle Assemble -> Rasterization -> Fragment Shader -> Testing and Blending -> Framebuffer

Vertex Array

This is essential an whats used to hold an array of data with values that are used to set positions of vertex points.

Vertex Shader

The vertex shader processes the vertex stream given to it one vertex at a time. When it is doing this it has no access to any of the other vertices.

Triangle Assemble

The triangle assembly is where the triangles that make up each object are formed.

Rasterization

The process where the image described in vector graphics format (2D image) representation of the scene is converted into a per pixel image and correct pixel values are determined; this is then displayed on the visual display unit.

Fragment Shader

The fragment shader is where the color is put on an object as well as where post processing occurs, including blurring.

Framebuffer

The Framebuffer is where a texture of the scene is stored so it may be drawn on a fullscreen quad for post processing and to display our final scene.

Immediate Mode

Immediate mode is slow and depreciated, it is better to use shaders or vbos for transformations. Immediate mode is what we used first term.

Channels

Channels that we care about store data in a range of 0 to 255. These represent the RGB colours. This is how we store texture for our games but we can store other information as well such as normal maps, bump maps or displacement maps.

VBO

VBO or vertex buffer object is a way of drawing a three dimensional object. This method is known as retained mode. The vbo stores in itself the vertices, normals and uvs and is able to be drawn with a single command. This is faster than drawing in immediate mode and will save space and time.

Shaders

Shaders are small pieces of code that can be executed by the graphics hardware allowing you to “cheat” for speed.

Languages

Shaders are done in their own languages, the three main ones we care about are HLSL, Cg and GLSL. These are classified as high level languages.

Vertex Shader

The vertex shader manipulates the vertices. Each vertex has its own defining qualities such as color, texture, position and normals. Most of the time vertex shaders just pass through the texture coordinates but it can also be used to alter the vertex in displacement mapping, mesh skinning and particle systems.

Fragment Shader

The fragment shader takes in data and outputs a color for each fragment. They are used in shadow mapping as well as bump mapping.

Vertex Skinning

Vertex skinning is when you pass in a bone matrix to a shader that specifies the transformations of each bone, it then passes in weights in as a texture unit as well as the index as well. This type of shader rotates the vertex accordingly to match the bone making is like a skin.

Lighting

Lighting is used in games to create mood and emotion. Without it our games look fake instead of real or believable. All light is additive and you must add together your different forms of light before applying it to your scene for proper lighting.

Ambient Lighting

Ambient lighting is the amount of non-directional lighting in a scene. It is calculated through K * Illumination Ambient where k is a value between one and zero.

Diffuse Lighting

Diffuse lighting is the light reflected off an object that we do not directly see. It follows the three lamberts laws which are as follows.

The area on the surface is illuminated at a perpendicular angle by light falling on it from a point source is proportional to the inverse distance between the surface and the source.

If the rays meet the surface at an angle then the illuminance is proportional to the cosine of the angle with the normal.

The intensity of the light decreased exponentially with distance as it travels through an absorbing medium.

This makes our formula for finding diffuse light equal to KIcos0 where K is our light constant between one and zero, I is our intensity of our light source and theta is the angle between the light source and the normal, which simplifies to (Normal dot LIght)

Specular Lighting

Specular lighting is the light that reflects off an object that we see directly. We calculate it through the dot product of the ray of reflection and the viewers angle. as KIcos0 or KL(reflection dot viewer).

Toon Shading

Toon shading works as a clamp where you would have multiple values. Once you calculate your objects light, if it is between certain values you set it to the value you want, this makes it so that you could have only 5 different intensities of shadows.

Multipass

The Idea behind multiple passes is to save them in an fbo or frame buffer object as a texture and then composite them at the end.

Post Processing

Post processing is a full screen effect in which you take an image of your final view and send it through a shader as a texture, you then edit that texture to get the view you so desire.

Different types of post processing techniques include: Blur, HDR/Bloom, Depth of Field.

Convolution Kernel Filter

A convolution kernel filter is a method of blurring an image to do this you would pull weighted values from a pixel in a texture as well as the pixels around it. For instance if you had the following two:

1	1	1
3	2	0
2	1	2

5	6	4
2	3	3
4	7	2

Then the final pixel value of the center would be:

1x5 + 1x6 + 1x4 + 3x2 + 2x3 + 0x3 + 2x4 + 1x7 + 2x2 = 5+6+4+6+6+0+8+7+4 = 46

The sum of all the element should equal one though so we need to normalize this value by dividing by the sum of all element.

There are two main mode of blurring, box blur and gaussian blur. In box blur all pixels have the same weight where in gaussian blur, pixels that are further away have a lower weight. For gaussian blur the larger the window the larger the blur but for box blur the blur is the same regardless of image size.

Gaussian blur is nice because it can be split up into two one dimensional passes, making it faster.

Edge Detection

Edge detection is also a filter of sorts. It is a kernel that looks for large change on the x or y axis to find out where edges are.

HDR Bloom

HDR bloom is a four step process. The first part is to render your scene to an offscreen framebuffer. Next you highlight the bright areas of your map and save it to another texture. Third you blur this new texture. The final step is to composite the blurred texture and the starting scene texture together to create your final scene. The process should look like this:

Global Illumination

Radiosity

Radiosity is power from an area in a given direction. It is also the outgoing power per unit area due to emission or reflection over a hemisphere of directions.

Radiant emitted flux density is the unit for light emission

.

With radiosity we need to trace the rays of of light as they reflect in our scene to create soft shadows, this though is too difficult and therefore

Occlusion

Occlusion is when your scene is shaded through other objects in the scene instead of full shadows. It is found using the formula max(0.0, dot(N,V)*(1.0/(1.0+d)) where d is the distance to the occludee from the occluder. to find the occluders sample around the current pixel, then rotate them by forty five degrees, ninety degrees and reflect around a random normal texture.

Deferred Shading

Deferred shading is the separation of drawing geometry and lighting calculations into different passes. This allows us to not only give lights certain areas or influence but to also allow use to have more lights without slowing down of processes.

This has a few drawbacks though, for instance our shading must still be done in a separate pass, adding to extra processor time, as well as not being able to process transparent objects or perform anti aliasing properly.

To do this we pass all our objects through our buffer outputting their depth, normals and colours which we then pass to lighting pass, we then composite our lighting pass and geometry together to get our final image.

To do this we need a geometry buffer, which take an input of shapes or points and outputs other shapes, points or images.

Shadow Mapping

Shadow mapping is a process that adds shadows to a scene with objects based off one or more light sources projecting onto the object. In order to draw a scene with shadows we need to do at least three passes.

In the first pass we create a depth map. This is done through rendering the scene from the lights point of view, we then take this view and save all the z values of the scene as a texture output into an fbo.

The second pass is re rendering the scene from the cameras view and applying the depth texture from the cameras location. Anything we see on our cameras view that has a greater depth value on the cameras view than our depth map becomes shaded, we then save this scene in an fbo.

The final step is to render out scene texture to a full screen quad then putting in in view for the player.

Using shadow mapping in real time can be fairly difficult when rendering the scene with multiple objects and light sources due to memory constraints, which becomes a major problem with anti-aliasing, but one method developed to overcome this issue called Cascaded Shadow Mapping which provides a higher resolution of the depth texture near the viewer and a lower resolution texture for farther away.

In order to perform this method the camera view is split by the frustum and creating a depth map for each partition.

The method can be broken down into two parts:

1.) For every lights frustum, render the scene depth from the lights point of view

2.) Render the scene from the cameras point of view, depending on the fragments z-value pick the correct shadow map to render.

A frustum would be a section where we move the light view to to get a new map.

Depth of Field

This is an optical effect that is used in games as a tool for cut scenes by shifting the focus of the player to what the game developer wants the audience to focus on, or as an effect for game play by creating the illusion of depth in game because if you focus on an object in game that should be far away yet the whole scene is in perfect focus takes away a feeling of immersion.

How do we use Depth of Field in gaming, well in OpenGL we can perform this post-process effect in two passes using GLSL.

The first pass when rendering the scene you store the depth of every vertex, calculating the amount of blur per fragment.

During the second pass you apply the per fragment blur based off the values from the last pass.

Motion Blur

This is the appearance of a streaking effect produced by a quickly moving object image being captured in motion by a camera, or the camera moving rapidly while focusing on a target, the effect produces a more natural appearance to a scene which can give a more immersive effect as this is a quality we would see in a real life scenario.

This can be done by saving the last couple images of our scene as textures, them blending them together with our new scene. The areas that do not move stay the same while the areas that do are blurred, giving the appearance of blurred motion.

Depth

Depth of Field

Depth of field is defined as the space in which we can see objects clearly; objects outside this depth appear blurry to our eyes, whether they are closer or further. Depth of field is fundamental to have for photo-realistic rendering.

To do depth of field you need to have alpha information to work with. To do depth of field you need to have alpha information to work with.

The first step is to pre-blur the image through downsizing. After this you use a kernel as well as the depth information to approximate where your circle of confusion is. After this you blend between the original and pre blurred image in this region.

This means that we need the blurriness as well as the depth of each pixel. To get this we pass three different types of information from the camera to the shader: the Focal Plane, the Near Plane and the Far Plane. The camera stores depth information between -1 and 1. To blur we need an absolute value instead of this relative value; to get this we merely half the value and add 0.5. This allows us to pass through blurriness and depth through the same channel. Even though we are doing this we can still do alpha blending in two passes. First you render the rgb with blending enabled for the first pass, after that you use the output of the your computation of depth blur to render to your destination.

Then now you blur you back image. You then blur blend between the original image and the blurred image based on the depth to find how blurry the image is supposed to be.

Most blurring techniques cause leaking of sharp objects into the background. To fix this we use the depth buffer to compare samples and remove ones that contribute to this “leaking”.

Portals

Lately I have been working on a portal system, trying to replicate the one in portal. There are three cameras in total, the main view camera and a camera attached to each portal.

There is a three pass system. For the first pass I start off by drawing the scene. I then create a texture out of the FBO from the first portals view.

The second pass is much like the same but bound the texture from the first pass to the second portals mesh before drawing the scene. Then I create a FBO texture from the second portals view.

During my third pass I have both of the textures bound the opposite portals, the first portals view is bound to the second and the seconds view to the first. I then render my scene to an FBO.

The fourth pass is to create a full screen quad and put the view FBO on top so that it can be post processed.

For the future I plane to add in the glow and ripple effects in. Both of these would create two more passes for the program.

For glow I plan to make two different border textures which I will blend between in order to get the glow to move. Then I will overlay this created texture on top of the texture created for the portals. This will require me to send in two additional textures and the a time variable that is between 0-1 to use for blending.

For my ripple effect I plan to do something similar with normal maps.

Teleportation is set up as a basic box collision check that is checked on both portals. Going through the portals is set up as teleportation for now as I am unsure how to do the gradual teleportation.

While this is not done yet I plan to work on it while I do other things and will post about my progress in the future.

Game Jam

A few weeks ago I took part in a game jam again. The theme of the game jam was interference and interactivity. There I had the chance to work with the Leap Motion. I worked with two others to create the game Ballin’. The basis of the game is to escort a marble through a maze and around pitfalls while not being able to see all but you near surroundings. The catch is that the maze is controlled by the Leap Motion. To play you would hover your hand over the Leap Motion and the board would mimic the movements of your hand. While it is not yet finished, we have actually acquired a Leap Motion of our own and plan to make and perhaps release the game on our own time. 

Post Processing

Post Processing

This week I was working on post processing shaders.

HDR/Bloom

The first one I worked on was bloom. We talked about bloom at an earlier time but lets recap it. Bloom is a glow effect overlay that is put on an image in order to reproduce the view of real world cameras. It is a computer graphics technique that is used in video games, tech demos as well as animated movies. Bloom is the process of “extending” the light from the borders of “bright” areas of an image. This makes the light brighter and more of a real world effect instead of seeming unsettling.

The theory of bloom is simple, in real life a lens can never focus perfectly. Even the most expensive lenses will distort the light somewhat. Normally we do not see this, but if the light is very bright, like the sun for instance, the light will go out of its natural place and extend its reach to other portions of the image. This effect is barely noticeable when there are two bright places next to each other, but when there is a bright and a dark place next to each other we see this effect.

In HDR we can reproduce this effect by altering the image with a Gaussian blur kernel. The distortion (or bleeding) from this kernel is effected by the brightness of the light in our scene. Since this is a post processing effect, it is done after the scene is initially rendered and there are two different ways to do this. The first way is the simplest: You start by rendering the scene to a FBO. The next step is to highlight your bright areas using tone mapping and save this as a second texture. You then apply Gaussian blur to this new texture. Finally you add together your original image and the blurred image for an image with bloom. The second way to do this is to add in an extra step. Before you do the highlights on your next texture you shrink it down and stretch it out to get a pixilated version of your original image. The more you shrink then stretch the image the more distorted your bleed effect will be. The second method is how we implemented it during my class.

Black and White

Another post processing technique I did was to make my image completely greyscale. This is very simple to do using  the following code:

float relativeLuminance(in vec3 rgb)

{

       return (0.2126*rgb.r + 0.7152 * rgb.g + 0.0722*rgb.b);

}

That alone allowed for only a basic greyscale image, I wanted to add in a cell shading effect therefore I used the following to create a cartoony, greyscale image.

float celShading(in vec3 pos, in vec3 norm)

{

       vec3 N = normalize(norm);

       vec3 L = normalize(lightPos - pos);

       float Lambert = max(0.0, dot(N, L));

       Lambert = texture(qMapTex, vec2(Lambert, 0.0)).r;

       return Lambert;

}

This gives the scene a similar effect as the following:

And He Said: Let There be Shade

This week the topic I will be covering will be shadows

Radiosity

Direct illumination is not realistic but radiosity is more. With radiosity you are not cutting off at shadows directly but with it shadows do not have sharp edges. The edges are soft and blurry which gives it a feel or realistic and ambient light. Windows and other edges will gain light as it can now be lit through reflected light instead of relying on direct contribution.

Add power to everything for reflection and set the power to zero except for the light sources, you then sort everything in order of how much power it has. Then you thrust some of the power to all the triangles you see, rinse and repeat.

Shadows

Shadows are used as a form of visual information that allows us to discern the position of objects at a distance as well as how the light is a scene operates. From shadows we can discern the strength and direction of light.

Shadows also give realism to a scene. If there are no shadows then a scene will look off. Not only this but shadows give us an idea of the space an object occupies, if something does not have a shadow, or if the shadow is far away then it looks like it is floating, but if the object is connected to its shadow then it looks like it is on the ground.

Shadow Mapping

With shadow mapping you want to render the scene twice. The first time you render the scene it is from the lights point of view. From this we can get the depth values of all the objects from the light point of view. This will create what is called a shadow map. You then render the scene from the cameras view. The newly created shadow map is then projected onto the scene after being compared to objects depth transformed to the lights position. Areas that are further away from the light than indicated by the mapping are then shaded.

This can create a few problems, the first problem being aliasing. Since all you are doing is overlaying a texture the edges could be jagged. 

We could always raise the texture resolution but there are other ways to fix this issue. The first way would be to compare multiple samples of the same map and then filter them. The second way would be to smooth the edges by using the UV offset of the texture to weight the mapping. If we pull ourselves off the grid system then we can use non-uniform sampling. While error is still there we can randomize our samples. Our offsets can be coded in; we can store two per vector for optimum efficiency. You have to be careful because constantly changing you offsets will give you undesirable results. You can pre-compute your values in the screen and based on your already aligned textures.

Depth Masking

There is always a problem though. If an object is behind another one, we still calculate the lighting and shading, even if we cannot see it. This leads to a large amount of unnecessary calculations. In this kind of scene we try not to use edge mapping, instead we use depth mapping, where, if there is an edge, we do a minimum and maximum depths for the region, this prevents us from having to do shading for multiple objects that we cannot see.

Shadow Silhouettes

Silhouette mapping is a way of increasing the qualities of our shadows without having to increase the size of our textures. The map contains new positions for the centers of each texel. To do this there is a three step process, first we start out by rendering our shadow map, after that we render our silhouette map. We offset our mapping centers in order to represent our edges better. Finally we do our lighting pass. Through the silhouette we chose which shadow map texel to pull data from.

To do this we find the location of the current pixel. We then grab the data of its neighbouring pixels as well. Using this data we find out which area quadrant the sub location is.

There is one large limitation for silhouette mapping. There cannot be overlapping silhouettes, doing so creates artifacts. Despite these limitations silhouette mapping provides a much higher quality than regular shadow mapping.

Bias

Bias is an increase of the depth values of your shadow map. You use this when you get incorrect depth values while shading you scene, a problem caused by the limited precision of depth maps as well as differences in the sampling rates of shadow maps and the sampling rate of the scene. If you have to little bias then you get artifacts along the orders of you shadow but if you give too much bias the shadows will become disjointed from the body itself.

A bias has two components, numeric and geometric. The numeric component is simply the shadow map component. The geometric component is the fact that when a shadow map is applied to an area of a scene, they only represent a single depth value, this means that a low resolution map, or a highly sloped scene will affect precision.

To understand bias we need to understand depth textures. These are what shadow maps use and, are themselves, essentially shadow maps. These maps hold data from the scene that holds the depth of each piece of geometry. Through rendering into the depth map we are able to not only save memory, but render any sized depth texture. These take care or memory issues and slope based bias without any additional cost. We use depth textures for depth of field, semi-transparent objects, lens flares and fog.

As you can see above, the light is hitting the plane at an angle. This causes shadow disconnect. What we want instead is a flattened, untangled shadow map. For this we need to know how the depth of the terrain changes based on the texture coordinates.

To do this we need to get the derivative of the texture coordinates based on the screen, or view, coordinates. This creates a transformation matrix which transforms information in the screen space to the texture space, this allows for proper mapping.

Blurred Design

Week 4

Well, its week four and here we are, this is where the difficulty ensues. Well, at least for me. This week we started to learn about blurring.

Convolution Kernel Filtering

Kernel filtering is a way of processing images through a “filter”. Through it you take a sample of the pixel and the surrounding pixels from its texture, then you use the sum of these pixels based on their weight. This new pixel is placed on to a new texture.

When you need to move over pixels you increase you pull location by 1/textureWidth and when you reach the end of a row you need to increase the height by 1/textureHeight.

More information: http://www.aforgenet.com/framework/features/convolution_filters.html

Blurring

Box Filter – A filter\ where every pixel is weighted the same.

 Left- regular image, Right – Box Blurred

Gaussian Blurring - Where the middle has a higher weight and the surrounding ones have a lower weighting; the bigger the sample window, the stronger the blur.

Edge Detection

Design a convolution kernel that favours things with a high change. Output values where left right values are different. For example the following is a sobel filter.

-1	0	1
-2	0	2
-1	0	2

-1	-2	-1
0	0	0
1	2	2

The top graph detects for edges on the x axis while the bottom graph detects for edges on the y axis. This gathers data for when the color of an object drastically changes letting us find when there is an edge. Then at this edge we can add in extra light. Normally you would do this in the lighting and shading but you do not have access to your neighbors in a fragment or vertex shader, therefore you need to do this in a second pass after the whole scene is saved as an image.

HDR, Bloom and Frame Processing

HDR and bloom are done though a post processing effect. It is done after you have your geometry and lighting.

Following the above steps you should end up with three separate textures which you merge together into the final frame. This data is processed in layers where you start with the base image and add the data together into a new image. This is done after rendering. To display your final image you would save everything in you back buffer, then make it into a texture. Then you make a quad the size of the screen and attach this texture to it. Through this process you can do everything through shaders. By sending the vertex information you can interpolate the UVs of the quad and send them to a pixel shader; this shader is called for every pixel. This means that this whole process can go into one function. If we want to make the image smaller than we just need to draw it at a percentage of its size.