4

From pictures of tools on a sheet of paper, I'm asked to find their outline contour to vectorize them.

I'm a total beginner in computer-vision-related problems and the only thing I thought about was OpenCV and edge detection.

The result is better than what I've imagined, this is still very unreliable, especially if the source picture isn't "perfect".

I took 2 photographies of a wrench they gave me.

After playing around with opencv bindings for node, I get this:

good wrench contour

Then, I've tried with the less-good picture:

enter image description here

That's totally inexploitable. I can get something a little better by changing the Canny thresold, but that must be automatized (given that the picture is relatively correct).

So I've got a few questions:

  • Am I taking the right approach? Is GrabCut better for this? A combination of Grabcut and Canny edge detection? I still need vertices at the end, but I feel that GrabCut does what I want too.
  • The borders are rough and have certain errors. I can augment approxPolyDP's multiplier, but without a loss of precision on good parts.
  • Related to the above point, I'm thinking of integrating Savitzky-Golay algorithm to smooth the outline, instead of polygon simplification with approxPolyDP. Is it a good idea?
  • Normally, the line of the outer border must form a simple, cuttable block. Is there a way in OpenCL to avoid that line to do impossible things, like passing on itself? - Or, simply, detect the problem? Those configurations are, of course, impossible but happen when the detection is failed (like in the second pic).
  • I'm searching a way to do automatic Canny thresold calculation, since I must tweak it manually for each image. Do you have a good example for that?
  • I noticed that converting the image to grayscale before edge detection sometimes deteriorates the result, and sometimes makes it better. Which one should I choose? (tools can be of any color btw!)

here is the source for my tests:

const cv = require('opencv');

const lowThresh = 90;
const highThresh = 90;
const nIters = 1;

const GRAY   = [120, 120, 120];
const WHITE = [255, 255, 255];

cv.readImage('./files/viv1.jpg', function(err, im) {
  if (err) throw err;

  width = im.width()
  height = im.height()
  if (width < 1 || height < 1) throw new Error('Image has no size');

  const out = new cv.Matrix(height, width);
  im.convertGrayscale();
  im_canny = im.copy();
  im_canny.canny(lowThresh, highThresh);
  im_canny.dilate(nIters);

  contours = im_canny.findContours();

  let maxArea = 0;
  let biggestContour;

  for (i = 0; i < contours.size(); i++) {
    const area = contours.area(i);
    if (area > maxArea) {
      maxArea = area;
      biggestContour = i;
    }

    out.drawContour(contours, i, GRAY);
  }

  const arcLength = contours.arcLength(biggestContour, true);
  contours.approxPolyDP(biggestContour, 0.001 * arcLength, true);

  out.drawContour(contours, biggestContour, WHITE, 5);

  out.save('./tmp/out.png');
  console.log('Image saved to ./tmp/out.png');
});
Improve this question
5
  • Check algorithms for non-uniform background removal Commented Jun 28, 2017 at 16:21
  • @AnderBiguri are you talking about GrabCut ? I'm more and more thinking that it may be the answer. I think i'll use my Canny-based solution to determine the GrabCut's object boundaries. Then apply GrabCut, that will also allow corrections by the user if needed (using masking). And then detect the edges again from GrabCut's output (since the background will be removed, it will work superbly, like when I take a detoured image on Google). Looks viable. To determine the scale of the object, I think I'll use ArUco markers. Commented Jun 28, 2017 at 17:21
  • How is this system intended to work (process wise)? Why I ask - lighting is a critical component of a machine vision system and in your images it's... well, it doesn't look ideal. The first photo is fairly decent, although you still see circular gradient bright in the middle. The second one is quite terrible. You could quite likely save yourself a lot of work or possible failure if you could fix that (assuming it's feasible). Commented Jun 28, 2017 at 19:56
  • Yes, the second pic was voluntarily bad to test the worst-case scenario. But at the end, users will receive sheets of paper for the background and instructions, so this won't be probably as bad as this. I just want a system that will be as solid as possible because users will take these pic at home, etc, where light conditions won't be controlled. Thanks for the resource you shared! Commented Jun 29, 2017 at 9:07
  • Consider moving your flash gun further away if possible (and zooming your lens). Another option, if you are on a budget, is to put a sheet of baking paper (greaseproof paper) over the flash to diffuse it, or surround your tools with white paper on all sides to bounce flashlight in from all angles. Commented Jul 31, 2017 at 21:40

2 Answers 2

Reset to default
1

You'll need to add some pre-processing to clean up the image. Because you have a large variation in intensities in the image because of shadow, poor lighting, high shine on tools, etc you should equalize the image. This will help you get a better response in the regions that are currently poorly lit or have high shine.

Here's an opencv tutorial on Histogram equalization in C++: http://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/histogram_equalization/histogram_equalization.html

Hope this helps

EDIT: You can have an automatic threshold based on some loss function(?). For eg: If you know that the tool will be completely captured in the frame, you know that you should get a high value at every column from x = 10 to x = 800(say). You could then keep reducing the threshold until you get a high value at every column from x = 10 to x = 800. This is a very naive way of doing it, but its an interesting experiment, I think, since you are generating the images yourself and have control over object placement.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thanks! Yes, I think that can greatly help. However I think that an automatically-determined threshold for Canny may do the same job in this case, while being simpler. But I think I'll change my approach and add GrabCut into the pipeline (see my comment on the question)
@MorganTouvereyQuilling My quick glance through grabcut shows that it is a semi-automatic process. I was under the impression you needed an automatic approach. I've edited the answer to show a very naive way of setting an automatic threshold
You're right, I've not been precise-enough: the process must be as automatized as possible. But user interaction is possible because every treated picture will be checked by an human and there must be no errors. I was thinking that the user would have to take pics until the edge detection was perfect, but that will be much easier for these users with GrabCut. Anyway, thanks for your help, maybe it will help at some point, and as @Dan says above, light is important so, I give you an upvote since I'm pretty sure that histogram equalization can help in lots of situations.
1

You might also try running your images through an adaptive threshold first. This type of binarization is fairly adept at segmenting foreground and background in cases like this, even with inconsistent lighting/shadows (which seems to be the issue in your second example above). Adathresh will require some parameter fine-tuning, but once the entire tool is segmented from the background, Canny edge detection should produce more consistent results.

As for the roughness in your contours, you could try setting your findContours mode to one of the CV_CHAIN_APPROX methods described here.

Improve this answer

1 Comment

Thanks @DCSmith, I've tested adaptative threshold after you mentioned it, and it can give quite interesting results on white sheet of paper. Put it has flaws too (incl. increased sensibility to shadows), so I would say that in my case, I get equivalent results with direct edge detection. I give you an upvote too since this approach can also give very good (I mean, excellent) results when the background is clearly distinct. I did not chose yet between this and direct edge detection, and will accept the answer if I finally choose this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.