Javascript Canvas Pixels

In this tutorial we are going to explore manipulating Canvas pixels in Javascript Haxe.

First complete the tutorial on drawing a Red circle on the Canvas. I will assume you have setup your code and directory system following this tutorial.

Adding an Image


For simplicity I am going to try using a Googleplus profile image ( mine ), actually a painting of mine.
Now I am not sure if the path will change if it does I will try uploading elsewhere.

To grab pixels we need an Image div element with an image in it. We could create an Image Div with createElement, like we did with the Canvas, but unfortunately that does not seem to allow us listen for when the image has loaded. So instead we need to set it up untyped and use __js__ to call new Image() in raw javascript.
To position the image div we use the style left and top attribues and position it 'absolute'. Most websites place elements on screen using relative positioning and css styling, but when using javascript for interactive graphics I tend to position everything absolutely, it's a more interactive flash approach, it all depends on what your creating, a blog or an interactive game. We can add this method to our program and swap the drawCircle call for a createImage call.

    var image: js.Image;
    
    function createImage()
    {
        var img = 'https://lh4.googleusercontent.com/-469wLcc5jKY/AAAAAAAAAAI/AAAAAAAAALA/WCsaqYAgqd0/s250-c-k/photo.jpg'; 
        image = untyped __js__("new Image()");
        image.style.left = '0px';
        image.style.top = '0px';
        
            image.style.position = "absolute";
        image.src = img;
        body.appendChild( image );
    }

You should now see an image of me.

Drawing an Image onto the Canvas


To put the image on our canvas we just need to wait for the image to load ( using onLoad callback ) and then drawImage, from the Image Div to the Canvas Div. The google image we are using is 250x250 pixels. Since we no longer want to append the Image to the body we need to comment out this line. If we left the image on the screen we would not see to canvas image we are drawing below.
    var image: js.Image;
    function createImage()
    {
        var img = 'https://lh4.googleusercontent.com/-469wLcc5jKY/AAAAAAAAAAI/AAAAAAAAALA/WCsaqYAgqd0/s250-c-k/photo.jpg'; 
        image = untyped __js__("new Image()");
        image.style.left = '0px';
        image.style.top = '0px';
        
        // add call back for when image is loaded.
        image.onload = copyAccross;        

        image.style.position = "absolute";
        image.src = img;
        //body.appendChild( image );
    }
    function copyAccross( e: js.Event ): Void
    {
        surface.drawImage( image, 0, 0, 250, 250 );
    }

Accessing the Canvas Pixels


To access the pixels of the canvas we can use getImageData on the CanvasRenderingContext2d. So we can grab the pixels and then draw them back to the same place using putImageData. In between we can manipulate the data as a CanvasPixelArray. So the code without any manipulation will be something along the lines of:
    var imageData = surface.getImageData( 0, 0, w, h );
    var dataIn: CanvasPixelArray = imageData.data;
    surface.putImageData( imageData, 0, 0 );

The structure of CanvasPixelArray


The structure of CanvasPixelArray is a bit like a haxe array or list since the information is just a array of Integers and although it is not quite a haxe array we can use ArrayAccess. Now the pixels are stored with an integer for Red, Green, Blue and Alpha and arranged like writting from left to right and then starting on the next line. So taking a 3x3 pixel image example.
a b c
d e f 
g h i

So the resulting CanvasPixelArray would be arranged as
abcdefghi

where each letter is a place holder for 4 Integers ( Red Green Blue Alpha ). So if we were to try to do a 3x3 pixel french flag blue white red, the actual output CanvasPixelArray might look more like..
[0,0,255,255,255,255,255,255,255,0,0,255,0,0,255,255,255,255,255,255,255,0,0,255,0,0,255,255,255,255,255,255,255,0,0,255];

Iterating through the pixels


The easy way I found to deal with this data is to loop width and height ways through the pixels like you do when reading a book. But within the loops we keep a counter index and multiplying by 4 we can find the red Integer for each pixel. Then from the red pixel we can offset through the other pixels color components ( Green, Blue and Alpha ). Lets take a look at this in action, in this code we experiment with adjusting the alpha value of the image ( case 3 ) to create a fun pattern based on position and a sine. The equation I am using for creating the new alpha is not important, we are only interested in seeing the structure of accessing and changing the pixels. Notice we are using Switch to return a color value depending on the color index.
public function imagePattern( context: CanvasRenderingContext2D, width: Int, height: Int )
{
    var imageData = context.getImageData( 0, 0, w, h );
    var data: CanvasPixelArray = imageData.data;
    // count is the current pixel number.
    var count = 0;
    // Row increment
    for( row in 0...width ){ 
        // Column increment
        for( col in 0...height ){
            // loop through the colors
            for( colorOffset in 0...4 ){
                var indexNumber = count*4 + colorOffset;
                var pixelComponentColor = data[ indexNumber ];
                data[ indexNumber ] = 
                switch( pixelComponentColor )
                {
                    case 3: 200 + Math.round( ( 255 - 200 )*Math.sin( ( (row*indexNumber)%width )/Math.PI/5 )  ) ;
                    case 0,1,2: pixelComponentColor;
                }
            }
            count++;
            }
    }
    context.putImageData( imageData, 0, 0 );
}

Using this in the original code is not hard to test the experiment by adjusting our copyAccross method...
    function copyAccross( e: js.Event ): Void
    {
            surface.drawImage( image, 0, 0, 250, 250 );
        imagePattern( surface, 250, 250 );
    }

Moving Pixels around


Often we want to move the pixels around, for instance flipping the image horizontally and vertically, you can do this in other ways with Canvas, but we are playing with pixels for more eventual power so we may want to do transformation that are not supported normally.
Now swapping pixels requires we don't overwrite the original values until we have read all the original values, otherwise we will end up with a reflection rather than a flip!
So we need to place the new pixels in a normal haxe Array<Int> then in a second set of loops copy the new values over, so first lets setup the structure.
    var imageData = context.getImageData( 0, 0, w, h );
    var dataIn: CanvasPixelArray = imageData.data;
    // new values
    var arr = new Array<Int>();
    var count = 0;
    for( row in 0...w ){
        for( col in 0...h ){
            for( color in 0...4 )
            {
                var k: Int = count*4 + color;
                arr[ k ] = dataIn[ k ];
            }
        }
        count++;
        }
    for( count in 0...(w*h) ){
        for( color in 0...4 ){
                var k = count*4 + color;
                dataIn[ k ] = arr[ k ];
            }
    }
    context.putImageData( imageData, 0, 0 );

Now at the moment this is not doing anything!! If we adjust the index when setting the temporary array, arr, then we can create either horizontal and vertical flips.
public function horiTransform( context: CanvasRenderingContext2D, w: Int, h: Int )
{
    var imageData = context.getImageData( 0, 0, w, h );
    var dataIn: CanvasPixelArray = imageData.data;
    // new values
    var arr = new Array<Int>();
    var count = 0;
    for( row in 0...w ){
        for( col in 0...h ){
            for( color in 0...4 )
            {
                var k: Int = count*4 + color;
                var kFlip: Int = (h*row- col)*4 + color;
                arr[ kFlip ] = dataIn[ k ];
            }
        }
        count++;
        }
    for( count in 0...(w*h) ){
        for( color in 0...4 ){
                var k = count*4 + color;
                dataIn[ k ] = arr[ k ];
            }
    }
    context.putImageData( imageData, 0, 0 );
}

To do the vertical flip we can just copy the method above, change its name and change the definition of kFlip so that it produces a vertical flip instead, may take some thinking but the effort is worth it ( well I have done if for you ).
var kFlip: Int = (h*(w-row)-col)*4 + color;

For flipping both horizontally and vertically...
var kFlip: Int = (h*(w-rew)- col)*4 + color;

Lets test this by adjusting our copyAccross method to use one of these transform methods instead.
    function copyAccross( e: js.Event ): Void
    {
            surface.drawImage( image, 0, 0, 250, 250 );
        horiTransform( surface, 250, 250 );
    }

Creating filters to achieve blur and sharpen


Now html5 has ways to apply filters to images, but it's nice to explore how we can create our own pixel based versions. A filter often works as a matrix operation, you effectively take the weighted nearest neighbours pixels to change the current pixel.

Blur


For instance to blur an image we can use a few different filter matrices.
[ 1/9, 1/9, 1/9
, 1/9, 1/9, 1/9
, 1/9, 1/9, 1/9 ];

[ 0., 1/5, 0.
, 1/5, 1/5, 1/5
, 0., 1/5, 0.];

Sharpen


[ -1., -1., -1.
, -1., 9., -1.
, -1., -1., -1. ];

Edge enhance


[ 0., 0., -2.
, 0., 2., 0.
, 1, 0., 0. ];

Edge Detection / Emboss ( sends image fairly black with edges in color )


similar to sharpen but lower middle value.
[ -1., -1., -1.
, -1., 8., -1.
, -1., -1., -1. ];

Brightness


We just increase the central value by a multiple.
[.0,.0,.0
,.0,3,.0
,.0,.0,.0];

Darkness


To make the image darker we can just divide the centre by a multiple.
[ .0, .0, .0
, .0, 1/3, .0
, .0, .0, .0 ];

Directional Blur


Directional blur really needs larger matrices to achieve much effect, but we can apply rather subtle ones. Diagonal examples.
[ 0, 0, 1/2,
0, 0, 0
, 1/2, 0, 0 ];

[ 1/2, 0, 0
, 0, 0, 0
, 0, 0, 1/2];

To apply the matrices we need to get the 8 neighbors of a pixel for the calculation, the differece between a value on the row above is going to be the width of the image, so we can build an array of the pixel indexes, but we need to make sure that a pixel has 8 neighbors, so lets just ignore the edge pixels by using an if statement.

    // pixel index values
    var ks: Array<Int> = new Array<Int>();
    if( ( i > 0 ) && ( i < w - 1 ) && ( j > 0 ) && ( j < h - 1 ) )
    {
        // row above
        ks.push( count - w - 1 );
        ks.push( count - w );
        ks.push( count - w + 1 );
        // current row ( left and right of current )
        ks.push( count - 1 );
        ks.push( count );
        ks.push( count + 1 );
        // row below
        ks.push( count + w - 1 );
        ks.push( count + w );
        ks.push( count + w + 1 );
    }

So using these pixel indexes we can loop through and calculate our matrix of neighbor weightings adding them to create a new value.
// loop colors
for( color in 0...4 ){
    var out = 0.0;
    var k: Int;
    // loop through the 9 pixel values ( for each color )
    for( aK in 0...9 )
    {   
          k = ( ks[ aK ] )*4 + col;
        // add the weighted pixel color component value to the sum for that color.
          out += (dataIn[ k ]) * filta[col][ aK ];
    }
    k = count*4 + col;
    // save the sum of the new pixel value in a temporary array.
    arr[ k ] = Math.round( out );
}

Since we are ignoring a one pixel border we need to add some code to push the original value into our new array for these cases. But essentially this is all we need to do however we will need to pass a filter for each color channel.
 filta: Array<Array<Float>>

Ok so lets look at the final function for applying a filter.
    public function imageFilter( context: CanvasRenderingContext2D
                        , w: Int, h: Int, filta: Array<Array<Float>> )
    {
        var imageData = context.getImageData( 0, 0, w, h );
        var dataIn: CanvasPixelArray = imageData.data;
        var arr: Array<Int> = new Array<Int>();
        var count = 0;
        for( i in 0...w )
        {
            for( j in 0...h )
            {
                
                // if 1 pixel in so that 3x3 grid...
                var ks: Array<Int> = new Array<Int>();
                if( ( i > 0 ) && ( i < w - 1 ) && ( j > 0 ) && ( j < h - 1 ) )
                {
                    ks.push( count - w - 1 );
                    ks.push( count - w );
                    ks.push( count - w + 1 );
                    ks.push( count - 1 );
                    ks.push( count );
                    ks.push( count + 1 );
                    ks.push( count + w - 1 );
                    ks.push( count + w );
                    ks.push( count + w + 1 );
                }
                
                for( col in 0...4 )
                {
                    var out = 0.0;
                    var k: Int;
                    
                    if( ks.length > 1 )
                    {
                        for( aK in 0...9 )
                        {   
                            k = (ks[ aK ])*4 + col;
                            out += (dataIn[ k ]) * filta[col][ aK ];
                        }
                        k = count*4 + col;
                        arr[ k ] = Math.round( out );
                    } 
                    else 
                    { // on edges just use value as is...
                        k = count*4 + col;
                        arr[ k ] = dataIn[ k ];
                    }
                    
                }
                count++;
            }
        }
        
        count = 0;
        for( i in 0...(w*h) )
        {
            
            for( col in 0...4 )
            {
                var k = i*4 + col;
                dataIn[ k ] = arr[ k ];
            }
            
        }
        context.putImageData( imageData, 0, 0 );
    }
    
}

Lets try these filters out by adjusting our copyAccross code.
    function copyAccross( e: js.Event ): Void
    {
            surface.drawImage( image, 0, 0, 250, 250 );
        // unit filter ( since no edge filter is needed for alpha ).
        var one = [ .0, .0, .0, .0, 1, .0, .0, .0, .0];
        // apply an edge filter
        var edge = [ 0., 0., -2., 0., 2., 0., 1, 0., 0. ];
        imageFilter( surface, 250, 250, [ edge, edge, edge, one );
        // ok lets change brightness
        var double = [.0,.0,.0,.0,2,.0,.0,.0,.0];
        imageFilter( surface, 250, 250, [ double, double, double, one ] );
    }

Advanced Color Manipulation


Now for complex color manipulation we may want to switch the red and blue component value so thinking in pretend code we can write something like..
red pixel =  0 x red pixel  + 0 x green pixel + blue pixel + 0 x alpha pixel
blue pixel = 0 x red pixel + 0 x green pixel +  blue pixel + 0 x alpha pixel

So if we imagine we can have four times as many matrices ( this is maybe overkill as it also means we can sharpen on one channel and blur on another color channel ) and with only a small adjustment we can achieve the power of the advanced color adjustment panel in flashIDE. So we modify the filter in to be of type Array<Array<Array<Float>>> and we just quadruple up on the out sum...

k = (ks[ aK ])*4 + 0;
out += (dataIn[ k ]) * filta[col][0][ aK ];
k = (ks[ aK ])*4 + 1;
out += (dataIn[ k ]) * filta[col][1][ aK ];
k = (ks[ aK ])*4 + 2;
out += (dataIn[ k ]) * filta[col][2][ aK ];
k = (ks[ aK ])*4 + 3;
out += (dataIn[ k ]) * filta[col][3][ aK ];

So the function looks fairly similar to the previous one we have just added a bit more complexity.

    public function imageColorFilter( context: CanvasRenderingContext2D, 
                                w: Int, h: Int, filta: Array<Array<Array<Float>>> )
    {
        var imageData = context.getImageData( 0, 0, w, h );
        var dataIn: CanvasPixelArray = imageData.data;
        var arr: Array<Int> = new Array<Int>();
        var count = 0;
        for( i in 0...w )
        {
            for( j in 0...h )
            {
                
                // if 1 pixel in so that 3x3 grid...
                var ks: Array<Int> = new Array<Int>();
                if( ( i > 0 ) && ( i < w - 1 ) && ( j > 0 ) && ( j < h - 1 ) )
                {
                    ks.push( count - w - 1 );
                    ks.push( count - w );
                    ks.push( count - w + 1 );
                    ks.push( count - 1 );
                    ks.push( count );
                    ks.push( count + 1 );
                    ks.push( count + w - 1 );
                    ks.push( count + w );
                    ks.push( count + w + 1 );
                }
                
                for( col in 0...4 )
                {
                    var out = 0.0;
                    var k: Int;
                    
                    if( ks.length > 1 )
                    {
                        for( aK in 0...9 )
                        {   
                            k = (ks[ aK ])*4 + 0;
                            out += (dataIn[ k ]) * filta[col][0][ aK ];
                            k = (ks[ aK ])*4 + 1;
                            out += (dataIn[ k ]) * filta[col][1][ aK ];
                            k = (ks[ aK ])*4 + 2;
                            out += (dataIn[ k ]) * filta[col][2][ aK ];
                            k = (ks[ aK ])*4 + 3;
                            out += (dataIn[ k ]) * filta[col][3][ aK ];
                        }
                        k = count*4 + col;
                        arr[ k ] = Math.round( out );
                    } 
                    else 
                    { // on edges just use value as is...
                        k = count*4 + col;
                        arr[ k ] = dataIn[ k ];
                    }
                    count++;
                }
            }
        }
        
        count = 0;
        for( i in 0...(w*h) )
        {
            
            for( col in 0...4 )
            {
                var k = i*4 + col;
                dataIn[ k ] = arr[ k ];
            }
            
        }
        context.putImageData( imageData, 0, 0 );
    }

Lets try this...
    function copyAccross( e: js.Event ): Void
    {
            surface.drawImage( image, 0, 0, 250, 250 );
        var one = [ .0, .0, .0, .0, 1, .0, .0, .0, .0];
        var zero = [ .0, .0, .0, .0, .0, .0, .0, .0, .0 ];
        var tenth20 = [ .0, .0, .0, .0, 2.0, .0, .0, .0, .0 ];
        var tenth2 = [ .0, .0, .0, .0, .2, .0, .0, .0, .0 ];
        imageColorFilter( surface, 250, 250
                                   , [ [zero,tenth20,tenth2,zero]
                                   ,   [edge,zero,zero,zero]
                                   ,   [zero,zero,edge,zero]
                                   ,   [zero,zero,zero,one]]
                            );
    }

Threshold color manipulation


Ok surpisingly enough javascript seems to run fine with all these complex array structures so as a last experiment lets try adding something abit like the tool in photoshop that allows you to change shadows highlights and midground separately. Well we add a lower and upper thresholds and just add three times as many matrices so we can add test the current pixel value and apply a matrix depending on it's current threshold.
    public function imageFilterThreshold( context: CanvasRenderingContext2D
                                , w: Int, h: Int
                                , filta: Array<Array<Array<Array<Float>>>>
                                , lowThresh: Int, hiThresh: Int )
    {
        var imageData = context.getImageData( 0, 0, w, h );
        var dataIn: CanvasPixelArray = imageData.data;
        var arr: Array<Int> = new Array<Int>();
        var count = 0;
        for( i in 0...w )
        {
            for( j in 0...h )
            {
                
                // if 1 pixel in so that 3x3 grid...
                var ks: Array<Int> = new Array<Int>();
                if( ( i > 0 ) && ( i < w - 1 ) && ( j > 0 ) && ( j < h - 1 ) )
                {
                    ks.push( count - w - 1 );
                    ks.push( count - w );
                    ks.push( count - w + 1 );
                    ks.push( count - 1 );
                    ks.push( count );
                    ks.push( count + 1 );
                    ks.push( count + w - 1 );
                    ks.push( count + w );
                    ks.push( count + w + 1 );
                }
                
                for( col in 0...4 )
                {
                    var out = 0.0;
                    var k: Int;
                    
                    if( ks.length > 1 )
                    {
                        for( aK in 0...9 )
                        {   
                            
                            k = (ks[ aK ])*4 + 0;
                            var val = dataIn[ k ];
                            if( lowThresh > val )
                            {
                                out += val * filta[col][0][0][ aK ];
                            } else if( hiThresh < val )
                            {
                                out += val * filta[col][2][0][ aK ];
                            } else
                            {
                               out += val * filta[col][1][0][ aK ]; 
                            }
                            k = (ks[ aK ])*4 + 1;
                            var val = dataIn[ k ];
                            if( lowThresh > val )
                            {
                                out += val * filta[col][0][1][ aK ];
                            } else if( hiThresh < val )
                            {
                                out += val * filta[col][2][1][ aK ];
                            } else
                            {
                               out += val * filta[col][1][1][ aK ]; 
                            }
                            k = (ks[ aK ])*4 + 2;
                            var val = dataIn[ k ];
                            if( lowThresh > val )
                            {
                                out += val * filta[col][0][2][ aK ];
                            } else if( hiThresh < val )
                            {
                                out += val * filta[col][2][2][ aK ];
                            } else
                            {
                               out += val * filta[col][1][2][ aK ]; 
                            }
                            k = (ks[ aK ])*4 + 3;
                            var val = dataIn[ k ];
                            if( lowThresh > val )
                            {
                                out += val * filta[col][0][3][ aK ];
                            } else if( hiThresh < val )
                            {
                                out += val * filta[col][2][3][ aK ];
                            } else
                            {
                               out += val * filta[col][1][3][ aK ]; 
                            }
                        }
                        k = count*4 + col;
                        arr[ k ] = Math.round( out );
                    } 
                    else 
                    { // on edges just use value as is...
                        k = count*4 + col;
                        arr[ k ] = dataIn[ k ];
                    }
                    
                }
        count++;
            }
        }
        

        for( i in 0...(w*h) )
        {

            for( col in 0...4 )
            {
                var k = i*4 + col;
                dataIn[ k ] = arr[ k ];
            }
            
        }
        context.putImageData( imageData, 0, 0 );
    }

And we can test that...
    function copyAccross( e: js.Event ): Void
    {
            surface.drawImage( image, 0, 0, 250, 250 );
        var one = [ .0, .0, .0, .0, 1, .0, .0, .0, .0];
        var zero = [ .0, .0, .0, .0, .0, .0, .0, .0, .0 ];
        var tenth9 = [ .0, .0, .0, .0, .9, .0, .0, .0, .0 ];
        var tenth11 = [ .0, .0, .0, .0, 1.1, .0, .0, .0, .0 ];
            var low = tenth11;
            var mid = one;
            var hi = tenth9;
        imageFilterThreshold( surface, 250, 250
                            , [[[zero,low,zero,zero],[zero,mid,zero,zero],[zero,hi,zero,zero]]
                            , [[low,zero,zero,zero],[mid,zero,zero,zero],[hi,zero,zero,zero]]
                            , [[zero,zero,low,zero],[zero,zero,mid,zero],[zero,zero,hi,zero]]
                            , [[zero,zero,zero,one],[one,zero,zero,one],[half,zero,zero,one]]]
                                                , 50, 200 );
    }

Next you can try tweening filters.. well that's probably what I want to try next, obviously manipulating pixels like this is often not the fastest approach there are probably better ways, but hopefully direct canvas pixel manipulation is no longer a daunting task.

TODO: Add Images to tutorial? Put some code on SVN?? Simplify the color examples to use an RGBA matrix instead.

version #15654, modified 2012-11-06 17:17:33 by JLM