Unity: Raycast in Editor Mode does not work

Question

I am trying to render the trajectory path of the movement of the ball in my game, all in Editor mode.

I believe that this will help my team in creating levels, as it will require less testing while prototyping.

Here is my code:

using UnityEngine;
using UnityEditor;
using System.Collections;

namespace smthng {
    [ExecuteInEditMode]
    public class TrajectoryGenerator : MonoBehaviour
    {

        public int distance; //max distance for beam to travel.
        public LineRenderer lineRenderer; //the linerenderer that acts as trajectory path
        int limit = 100; // max reflections
        Vector2 curRot, curPos;


        void OnEnable()
        {

            StartCoroutine(drawLine());
        }

        IEnumerator drawLine()
        {
            int vertex = 1; //How many line segments are there
            bool isActive = true; //Is the reflecting loop active?

            Transform _transform = gameObject.transform;

            curPos = _transform.position; //starting position of the Ray
            curRot = _transform.right; //starting direction of the Ray


            lineRenderer.SetVertexCount(1);
            lineRenderer.SetPosition(0, curPos);

            while (isActive)
            {
                vertex++;
                //add ne vertex so the line can go in other direction
                lineRenderer.SetVertexCount(vertex); 

                //raycast to check for colliders
                RaycastHit2D hit = Physics2D.Raycast(curPos, curRot, Mathf.Infinity, LayerMask.NameToLayer(Constants.LayerHited));
                Debug.DrawRay(curPos, curRot, Color.black, 20);
                if (hit.collider != null)
                {
                    Debug.Log("bounce");
                    //position the last vertex where the hit occured
                    lineRenderer.SetPosition(vertex - 1, hit.point);
                    //update current position and direction;
                    curPos = hit.point;
                    curRot = Vector2.Reflect(curRot, hit.normal);
                }
                else
                {
                    Debug.Log("no bounce");
                    isActive = false;
                    lineRenderer.SetPosition(vertex - 1, curPos + 100 * curRot);

                }
                if (vertex > limit)
                {
                    isActive = false;
                }
                yield return null;
            }
        }
    }
}

The problem is, I have never even saw the Debug.Log("bounce"); appear in the console.

In other words, the raycast never detects anything.

I read dozen of articles for this, many pointing that collider detection is impossible in editor mode.

Is this true? Or I have bug that I am not detecting.

Let me know if you need more info, but I believe this is it.

Thanks in advance,

Answer 1

Layer mask is different than Layer index. Layer mask is a binary number consists of zeros and ones, where each 1 represents one included layer index. In other words ith 1 in Layer mask tells unity to include Layer index #i.

For example if layer mask should include layer#0 and layer#2, it would be equal to ...000000101 = 5

So you must change the line

RaycastHit2D hit = Physics2D.Raycast(curPos, curRot, Mathf.Infinity, LayerMask.NameToLayer(Constants.LayerHited));

to

RaycastHit2D hit = Physics2D.Raycast(curPos, curRot, Mathf.Infinity, 1 << LayerMask.NameToLayer(Constants.LayerHited));

Additional math info

<< operator shifts a LHS by RHS bits to the left.

LHS: left hand side

RHS: right hand side

e.g. when I write 1 << 3 it means take 1 and shift it 3 bits to the left. which gives 00001 << 3 = 01000

Layer mask works bitwise, you have to specify which layers should be masked and which layers should be ignored by simply setting(1) or resetting(0) their corresponding binary digits in the layer mask.

If I set my layer mask to 7 then unity will convert it to binary (00000111) and masks first three layers and ignores all other layers.

If I set my layer mask to 8 then unity will convert it to binary (00001000) and masks 4th layers and ignores all other layers.

You can read more about it here and here