Hi, I have completed the course(Reay enjoyed it BTW) and I am now expanding upon it I am now having some trouble with using Raycast which has brought me to find that none of the ones we do in the course really work as described. According to the documentation the Layermask is used to EXCLUDE a layer from detection not INCLUDE, further experimentation leads me to bellieve that niether is absolutely correct and how Raycast works is more complicated than first seems. Could someone please clarify why in this lecture we put the obstacesLayerMask as a parameter which according to the docs would make these objects IGNORED which is the opposite of what we want.
A LayerMask actually defines the layers that are included in the RayTrace, not the ones that excluded.
Under the hood, a LayerMask is really just a 32 bit integer value, but whatās happening is that each bit within the integer represents a single layer within the 32 layers available. Bit 0 represents layer 0, bit 1 represents layer 1, etc. When a layer is tested, the system does a simple calculation. This is how the layer mask field can represent all, none, or a selection of layers. If Layer 0, 2, and 3 are selected, for example, then the LayerMask would be
1<<0 (1) +
1<<2 (4) +
1<<3 (8) = 13
A single layer value that we set on our GameObjects is a value with that layer shifted by the layer index, so if the GameObjectās layer was 3, then itsā integer value would be
1<<3 (8).
The way a Mask works is actually very simple with these two factors in mindā¦ the Physics system does a bitwise AND operation (&) and checks to see if the result of that calculation is greater than zero
if((layerMask & layer) > 0) hit=true;
Fortunately, we donāt have to worry about any of the actual math under the hood, as long as weāre assigning the LayerMask in the inspector. All we need to do is set the LayerMask for the layers we want to include in the Physics.Raycast.
1 Like
An issue I had, which may or may not be relevant, was incorrectly assuming the int value of a layer set in Unity was the same as the int layerMask value used for a raycast in C#.
Thereās a very short write-up here if you want to convert a Layer Mask to a Layer Index. SimonHildebrandt.com - Unity Tip: Converting a LayerMask to a Layer index
Thatās a common mistake. In fact, I made exactly that mistake coding in a rush yesterday.
If the layer is layer 7, the layer mask value would be 1<<7
A little off topic, but I came across a post somewhere (maybe stackoverflow) many years ago that said that you shouldnāt check if a flag is greater than 0. I canāt quite remember the reason why but iirc it made sense at the time. You should rather check if it matches the original value
if ((layerMask & layer) == layer) hit = true;
Sadly I canāt recall the exact details - I think it had something to do with possibly having a valid flag with value 0 - I just know Iāve been doing it like this ever since.
This was generally for demonstration purposes. Unity itself handles the actual layermask mathematics.
That almost certainly isnāt the reasonā¦ it may be that the direct comparison to layer is faster than >0 (doubtful), but if the result of layerMask & layer can only be one of 2 valuesā¦ 1<<layer or 0. 0 always means no flags were matched.
Yeah, I did a google now and if you have a valid flag with a 0 value (which is a little strange, I know) it will always be false, even if the mask is also 0. The sample I saw had a āNoneā flag set to 0 and if the layerMask is āNoneā and the flag is āNoneā then (0 & 0) > 0 is false instead of the true one would expect. But I think the point is moot. Having a 0 flag is a bit silly
If a LayerMask is showing as None in the inspector, itās value isā¦ 0. (not to be confused with the 0th bit, which has a value of 1 when true) Thatās just what Unity is showing in the inspector for our convenience. That effectively means any & operation will yield 0 regardless of the layer.
Iām trying to think of a use case where I wouldnāt want the result of either an empty LayerMask or a Layer set to None to be interpreted as anything but false. If you were testing for exclusion, you would just negate the answer (flipping the bits). Iām probably missing something in the same way that I donāt fully understand Avacodo Toast. 
Well, thatās probably enough āinto the weedsā on this one, LOL. Fortunately, we donāt need to worry too much about how the physics engine does the actual comparison, just that the bit being set means you include that layer in the result set.
Thanks I appreciate the replies.
This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.