GPU
The GPU or graphics processing unit the portion of the computer we care the most about. It has a programmable pipeline, which we can use to alter how it processes data. It is highly parallel meaning it can process multiple tasks at the same time very quickly.
Graphics Pipeline
Vertex Array -> Vertex Shader -> Triangle Assemble -> Rasterization -> Fragment Shader -> Testing and Blending -> Framebuffer
Vertex Array
This is essential an whats used to hold an array of data with values that are used to set positions of vertex points.
Vertex Shader
The vertex shader processes the vertex stream given to it one vertex at a time. When it is doing this it has no access to any of the other vertices.
Triangle Assemble
The triangle assembly is where the triangles that make up each object are formed.
Rasterization
The process where the image described in vector graphics format (2D image) representation of the scene is converted into a per pixel image and correct pixel values are determined; this is then displayed on the visual display unit.
Fragment Shader
The fragment shader is where the color is put on an object as well as where post processing occurs, including blurring.
Framebuffer
The Framebuffer is where a texture of the scene is stored so it may be drawn on a fullscreen quad for post processing and to display our final scene.
Immediate Mode
Immediate mode is slow and depreciated, it is better to use shaders or vbos for transformations. Immediate mode is what we used first term.
Channels
Channels that we care about store data in a range of 0 to 255. These represent the RGB colours. This is how we store texture for our games but we can store other information as well such as normal maps, bump maps or displacement maps.
VBO
VBO or vertex buffer object is a way of drawing a three dimensional object. This method is known as retained mode. The vbo stores in itself the vertices, normals and uvs and is able to be drawn with a single command. This is faster than drawing in immediate mode and will save space and time.
Shaders
Shaders are small pieces of code that can be executed by the graphics hardware allowing you to “cheat” for speed.
Languages
Shaders are done in their own languages, the three main ones we care about are HLSL, Cg and GLSL. These are classified as high level languages.
Vertex Shader
The vertex shader manipulates the vertices. Each vertex has its own defining qualities such as color, texture, position and normals. Most of the time vertex shaders just pass through the texture coordinates but it can also be used to alter the vertex in displacement mapping, mesh skinning and particle systems.
Fragment Shader
The fragment shader takes in data and outputs a color for each fragment. They are used in shadow mapping as well as bump mapping.
Vertex Skinning
Vertex skinning is when you pass in a bone matrix to a shader that specifies the transformations of each bone, it then passes in weights in as a texture unit as well as the index as well. This type of shader rotates the vertex accordingly to match the bone making is like a skin.
Lighting
Lighting is used in games to create mood and emotion. Without it our games look fake instead of real or believable. All light is additive and you must add together your different forms of light before applying it to your scene for proper lighting.
Ambient Lighting
Ambient lighting is the amount of non-directional lighting in a scene. It is calculated through K * Illumination Ambient where k is a value between one and zero.
Diffuse Lighting
Diffuse lighting is the light reflected off an object that we do not directly see. It follows the three lamberts laws which are as follows.
The area on the surface is illuminated at a perpendicular angle by light falling on it from a point source is proportional to the inverse distance between the surface and the source.
If the rays meet the surface at an angle then the illuminance is proportional to the cosine of the angle with the normal.
The intensity of the light decreased exponentially with distance as it travels through an absorbing medium.
This makes our formula for finding diffuse light equal to KIcos0 where K is our light constant between one and zero, I is our intensity of our light source and theta is the angle between the light source and the normal, which simplifies to (Normal dot LIght)
Specular Lighting
Specular lighting is the light that reflects off an object that we see directly. We calculate it through the dot product of the ray of reflection and the viewers angle. as KIcos0 or KL(reflection dot viewer).
Toon Shading
Toon shading works as a clamp where you would have multiple values. Once you calculate your objects light, if it is between certain values you set it to the value you want, this makes it so that you could have only 5 different intensities of shadows.
Multipass
The Idea behind multiple passes is to save them in an fbo or frame buffer object as a texture and then composite them at the end.
Post Processing
Post processing is a full screen effect in which you take an image of your final view and send it through a shader as a texture, you then edit that texture to get the view you so desire.
Different types of post processing techniques include: Blur, HDR/Bloom, Depth of Field.
Convolution Kernel Filter
A convolution kernel filter is a method of blurring an image to do this you would pull weighted values from a pixel in a texture as well as the pixels around it. For instance if you had the following two:
1
|
1
|
1
|
3
|
2
|
0
|
2
|
1
|
2
|
5
|
6
|
4
|
2
|
3
|
3
|
4
|
7
|
2
|
Then the final pixel value of the center would be:
1x5 + 1x6 + 1x4 + 3x2 + 2x3 + 0x3 + 2x4 + 1x7 + 2x2 = 5+6+4+6+6+0+8+7+4 = 46
The sum of all the element should equal one though so we need to normalize this value by dividing by the sum of all element.
There are two main mode of blurring, box blur and gaussian blur. In box blur all pixels have the same weight where in gaussian blur, pixels that are further away have a lower weight. For gaussian blur the larger the window the larger the blur but for box blur the blur is the same regardless of image size.
Gaussian blur is nice because it can be split up into two one dimensional passes, making it faster.
Edge Detection
Edge detection is also a filter of sorts. It is a kernel that looks for large change on the x or y axis to find out where edges are.
HDR Bloom
HDR bloom is a four step process. The first part is to render your scene to an offscreen framebuffer. Next you highlight the bright areas of your map and save it to another texture. Third you blur this new texture. The final step is to composite the blurred texture and the starting scene texture together to create your final scene. The process should look like this:
Global Illumination
Radiosity
Radiosity is power from an area in a given direction. It is also the outgoing power per unit area due to emission or reflection over a hemisphere of directions.
Radiant emitted flux density is the unit for light emission
.
With radiosity we need to trace the rays of of light as they reflect in our scene to create soft shadows, this though is too difficult and therefore
Occlusion
Occlusion is when your scene is shaded through other objects in the scene instead of full shadows. It is found using the formula max(0.0, dot(N,V)*(1.0/(1.0+d)) where d is the distance to the occludee from the occluder. to find the occluders sample around the current pixel, then rotate them by forty five degrees, ninety degrees and reflect around a random normal texture.
Deferred Shading
Deferred shading is the separation of drawing geometry and lighting calculations into different passes. This allows us to not only give lights certain areas or influence but to also allow use to have more lights without slowing down of processes.
This has a few drawbacks though, for instance our shading must still be done in a separate pass, adding to extra processor time, as well as not being able to process transparent objects or perform anti aliasing properly.
To do this we pass all our objects through our buffer outputting their depth, normals and colours which we then pass to lighting pass, we then composite our lighting pass and geometry together to get our final image.
To do this we need a geometry buffer, which take an input of shapes or points and outputs other shapes, points or images.
Shadow Mapping
Shadow mapping is a process that adds shadows to a scene with objects based off one or more light sources projecting onto the object. In order to draw a scene with shadows we need to do at least three passes.
In the first pass we create a depth map. This is done through rendering the scene from the lights point of view, we then take this view and save all the z values of the scene as a texture output into an fbo.
The second pass is re rendering the scene from the cameras view and applying the depth texture from the cameras location. Anything we see on our cameras view that has a greater depth value on the cameras view than our depth map becomes shaded, we then save this scene in an fbo.
The final step is to render out scene texture to a full screen quad then putting in in view for the player.
Using shadow mapping in real time can be fairly difficult when rendering the scene with multiple objects and light sources due to memory constraints, which becomes a major problem with anti-aliasing, but one method developed to overcome this issue called Cascaded Shadow Mapping which provides a higher resolution of the depth texture near the viewer and a lower resolution texture for farther away.
In order to perform this method the camera view is split by the frustum and creating a depth map for each partition.
The method can be broken down into two parts:
1.) For every lights frustum, render the scene depth from the lights point of view
2.) Render the scene from the cameras point of view, depending on the fragments z-value pick the correct shadow map to render.
A frustum would be a section where we move the light view to to get a new map.
Depth of Field
This is an optical effect that is used in games as a tool for cut scenes by shifting the focus of the player to what the game developer wants the audience to focus on, or as an effect for game play by creating the illusion of depth in game because if you focus on an object in game that should be far away yet the whole scene is in perfect focus takes away a feeling of immersion.
How do we use Depth of Field in gaming, well in OpenGL we can perform this post-process effect in two passes using GLSL.
The first pass when rendering the scene you store the depth of every vertex, calculating the amount of blur per fragment.
During the second pass you apply the per fragment blur based off the values from the last pass.
Motion Blur
This is the appearance of a streaking effect produced by a quickly moving object image being captured in motion by a camera, or the camera moving rapidly while focusing on a target, the effect produces a more natural appearance to a scene which can give a more immersive effect as this is a quality we would see in a real life scenario.
This can be done by saving the last couple images of our scene as textures, them blending them together with our new scene. The areas that do not move stay the same while the areas that do are blurred, giving the appearance of blurred motion.
