Detecting square in image

Question

At a school we are preparing artwork which we have scanned and want automatically crop to the correct size. The kids (attempt) to draw within a rectangle:

I want to detect the inner rectangle borders, so I have applied a few filters with accord.net:

var newImage = new Bitmap(@"C:\Temp\temp.jpg");
var g = Graphics.FromImage(newImage);
var pen = new Pen(Color.Purple, 10);

var grayScaleFilter = new Grayscale(1, 0, 0);
var image = grayScaleFilter.Apply(newImage);
image.Save(@"C:\temp\grey.jpg");

var skewChecker = new DocumentSkewChecker();
var angle = skewChecker.GetSkewAngle(image);
var rotationFilter = new RotateBilinear(-angle);
rotationFilter.FillColor = Color.White;
var rotatedImage = rotationFilter.Apply(image);
rotatedImage.Save(@"C:\Temp\rotated.jpg");

var thresholdFilter = new IterativeThreshold(10, 128);
thresholdFilter.ApplyInPlace(rotatedImage);
rotatedImage.Save(@"C:\temp\threshold.jpg");

var invertFilter = new Invert();
invertFilter.ApplyInPlace(rotatedImage);
rotatedImage.Save(@"C:\temp\inverted.jpg");

var bc = new BlobCounter
{
    BackgroundThreshold = Color.Black,
    FilterBlobs = true,
    MinWidth = 1000,
    MinHeight = 1000
};

bc.ProcessImage(rotatedImage);
foreach (var rect in bc.GetObjectsRectangles())
{
    g.DrawRectangle(pen, rect);
}

newImage.Save(@"C:\Temp\test.jpg");

This produces the following inverted image that the BlobCounter uses as input:

But the result of the blobcounter isn't super accurate, the purple lines indicate what the BC has detected.

Would there be a better alternative to the BlobCounter in accord.net or are there other C# library better suited for this kind of computer vision?

Is the border always of the same color? The "cropping area" is always rectangular (i.e., has always four corners), right? — stateMachine
– stateMachine, Commented Oct 12, 2020 at 6:01
Correct, I even put a rotate correction on it's so it's almost a perfectly straight rectangle. Each piece is scanned in the same resolution. — Roger Far
– Roger Far, Commented Oct 12, 2020 at 15:14

TheGeneral · Accepted Answer · 2020-10-12 06:05:52Z

2

Here is a simple solution while I was bored on my lunch break.

Basically it just scans all the dimensions from outside to inside for a given color threshold (black), then takes the most prominent result.

Given

[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static unsafe bool IsValid(int* scan0Ptr, int x, int y,int stride, double thresh)
{
   var c = *(scan0Ptr + x + y * stride);
   var r = ((c >> 16) & 255);
   var g = ((c >> 8) & 255);
   var b = ((c >> 0) & 255);

   // compare it against the threshold
   return r * r + g * g + b * b < thresh;
}

private static int GetBest(IEnumerable<int> array)
   => array.Where(x => x != 0)
      .GroupBy(i => i)
      .OrderByDescending(grp => grp.Count())
      .Select(grp => grp.Key)
      .First();

Example

private static unsafe Rectangle ConvertImage(string path, Color source,  double threshold)
{

   var thresh = threshold * threshold;

   using var bmp = new Bitmap(path);

   // lock the array for direct access
   var bitmapData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height), ImageLockMode.ReadWrite, PixelFormat.Format32bppPArgb);
   int left, top, bottom, right;

   try
   {
      // get the pointer
      var scan0Ptr = (int*)bitmapData.Scan0;    
      // get the stride
      var stride = bitmapData.Stride / 4;

      var array = new int[bmp.Height];

      for (var y = 0; y < bmp.Height; y++)
      for (var x = 0; x < bmp.Width; x++)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[y] = x;
            break;
         }

      left = GetBest(array);

      array = new int[bmp.Height];

      for (var y = 0; y < bmp.Height; y++)
      for (var x = bmp.Width-1; x > 0; x--)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[y] = x;
            break;
         }

      right = GetBest(array);

      array = new int[bmp.Width];

      for (var x = 0; x < bmp.Width; x++)
      for (var y = 0; y < bmp.Height; y++)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[x] = y;
            break;
         }

      top = GetBest(array);

      array = new int[bmp.Width];

      for (var x = 0; x < bmp.Width; x++)
      for (var y = bmp.Height-1; y > 0; y--)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[x] = y;
            break;
         }

      bottom = GetBest(array);


   }
   finally
   {
      // unlock the bitmap
      bmp.UnlockBits(bitmapData);
   }

   return new Rectangle(left,top,right-left,bottom-top);

}

Usage

var fileName = @"D:\7548p.jpg";

var rect = ConvertImage(fileName, Color.Black, 50);

using var src = new Bitmap(fileName);
using var target = new Bitmap(rect.Width, rect.Height);
using var g = Graphics.FromImage(target);

g.DrawImage(src, new Rectangle(0, 0, target.Width, target.Height), rect, GraphicsUnit.Pixel); 

target.Save(@"D:\Test.Bmp");

Output

_{Notes :

This is not meant to be bulletproof or the best solution. Just a fast simple one.
There are many approaches to this, even machine learning ones that are likely better and more robust.
There is a lot of code repetition here, basically I just copied, pasted and tweaked for each side
I have just picked an arbitrary threshold that seems to work. Play with it
Getting the most common occurrence for the side is likely not the best approach, maybe you would want to bucket the results.
You could probably sanity limit the amount a side needs to scan in.}

edited Oct 12, 2020 at 6:05

answered Oct 12, 2020 at 4:47

TheGeneral

82k9 gold badges111 silver badges151 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

stateMachine Over a year ago

Good solution! N i c e

Roger Far Over a year ago

That works well, except unfortunately under the image there is a part where students write their name (cut that off to keep the example simple), and it thinks that that is part of the rectangle because it has a line to write on. Ideally I should add a function to check either the thickness of the line or see how uninterrupted it is.

TheGeneral Over a year ago

@RogerFar yeah there are many ways you can customise it, and in the end it will come down to what works for you. Anyway good luck

Roger Far Over a year ago

In the end I only had to make a small modification: I looked for the best value from the middle out, that way I'm also capturing the inner box at the same time! I ran it on a few test samples and it works great!

TheGeneral Over a year ago

@RogerFar ahh ok that sounds like a win, glad it helped. Also nice work with the inner out, seems like it will work well

Collectives™ on Stack Overflow

Detecting square in image

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related