Hi guys, I hope you don�t mind me dropping in on this thread, as I will soon have to perform a similar task. My task is to see if Polygon A is completely within polygon B
The issues are similar, so here is my take on your problem.
Use a brut force algorithm to check if any intersections occur with the all sides of the triangles.
If no intersection occur, there are two possible situations:
1) One triangle is completely inside the other
2) Or one triangle is completely outside the other
So to check if one is inside the other:
Choose an arbitrary end point of one of the triangle sides and test if that point is inside or outside the other triangle. This could be accomplished by firing a ray from this point and checking the parity of the number of intersections, or compute winding numbers.
So now:
If there are any intersections, collision has occurred
If there are no intersections, but any arbitrary end point from a triangle is inside the other, collision has occurred.
Comments appreciated