Help with Raycasting and selecting the closest available enemy

Hi there,

I’ve completed Nathan’s 3rd Person Combat course and was wanting to expand the enemy targeting code so that if an enemy is behind a wall they’re not able to be targeted.

I’m trying to do this by using a foreach loop and Raycasting each of the potential targets that have entered our sphere collider (and therefore been added to our List of targets), and only getting the distance of those the ray touches, while discarding any that have a wall between the player and the enemy.

I’ve managed to get a fairly broken version of this up and running, but am wondering if anyone has any ideas on where exactly I’m going wrong, as I’m running into the following issues:


Figure 1) When I hit the targeting key in this position, everything works as expected, and we target the closest available target that’s not blocked by a wall


Figure 2) Again, when I hit the targeting key in this position, everything works as expected, and we target the closest available target that’s not blocked by a wall


Figure 3) Here things start to go wrong. We’ve got an available target to the East/SouthEast of the player, but we don’t seem to aim at them. Hitting the aim key does nothing until I move closer to the target, despite the raycast range being well above that distance.


Figure 4) In this position we target the wrong enemy, as they should be blocked by a wall, and therefore unavailable. In this situation we’d ideally be targeting the enemy to the south.

Here is the code in my Targeter script. It should all be quite similar to the code in the course, as I’m just adapting that for my own needs, but the relevant method is SelectTargetClosestToPlayer(), which is being called when the OnTarget event is triggered over in the FreeLookState.

public class Targeter : MonoBehaviour
{
    [SerializeField] PlayerStateMachine playerStateMachine;
    [SerializeField] List<Target> targets = new List<Target>();
    [SerializeField] LayerMask obstructionLayerMask;
    [SerializeField] LayerMask enemyLayerMask;
    public Target CurrentTarget { get; private set; }

    [SerializeField] CinemachineTargetGroup cinemachineTargetGroup;

    void OnTriggerEnter(Collider other)
    {
        Target target = other.GetComponent<Target>();

        if (target == null) { return; }
            
        targets.Add(target);
        target.OnDestroyed += RemoveTarget;
    }

    void OnTriggerExit(Collider other)
    {
        if (!other.TryGetComponent<Target>(out Target target)) { return; }

        RemoveTarget(target);
    }


    // WRITE CODE THAT WILL RAYCAST EACH POTENTIAL TARGET AND GRAB THE DISTANCE OF
    // ALL ENEMIES THAT AREN'T BEHIND A WALL
    // DOESN'T WORK RIGHT NOW

    public bool SelectTargetClosestToPlayer()
    {
        if (targets.Count == 0) { return false; }

        Target closestTarget = null;
        float closestTargetDistance = Mathf.Infinity;

        foreach (Target target in targets)
        {
            Debug.DrawRay(transform.position, target.transform.position - transform.position, Color.red, 1f);

            Vector3 TargetDistanceFromPlayer = target.transform.position - transform.position;
            RaycastHit hit;

            if (!Physics.Raycast(transform.position, target.transform.position - transform.position.normalized, out hit, playerStateMachine.RangeOfVision, obstructionLayerMask) && TargetDistanceFromPlayer.sqrMagnitude < closestTargetDistance)
            {
                closestTarget = target;
                closestTargetDistance = TargetDistanceFromPlayer.sqrMagnitude;
            }
        }

        if (closestTarget == null) { return false; }

        CurrentTarget = closestTarget;
        cinemachineTargetGroup.AddMember(CurrentTarget.transform, 1.0f, 2.0f);

        return true;
    }


    public void Cancel()
    {
        if (CurrentTarget == null) { return; }

        cinemachineTargetGroup.RemoveMember(CurrentTarget.transform);
        CurrentTarget = null;
    }

    void RemoveTarget(Target target)
    {
        if (CurrentTarget == target)
        {
            cinemachineTargetGroup.RemoveMember(CurrentTarget.transform);
            CurrentTarget = null;
        }

        target.OnDestroyed -= RemoveTarget;
        targets.Remove(target);
    }
}

I’m assuming that it’s still just targeting the closest enemy, although Figure 3 would be particularly confusing in this case.

Have been trying to solve this for days but there’s obviously something key that I’m not understanding about Raycasting or cycling through lists.

Any help would be massively appreciated!

Thanks in advance

Josh

Figures 1 and 2 is just picking the closest enemy, so it really shows nothing.

Figure 3 could be (and I don’t know what your range is) outside the RangeOfVision and the correct target is not within range. Be aware that the RangeOfVision and the size of the detection sphere should ideally be the same. Otherwise enemies could enter the sphere, but be outside the range of the raycast. I think in that scenario, the enemy would be treated as ‘not behind a wall’ if the wall is also outside the RangeOfVision. What I mean is that if the enemy is in the detection sphere and 2.5 units from the player and the wall is 2 units from the player, but the RangeOfVision is 1.5 units, it will be able to use that enemy as a potential target because the ray never hit the wall.

Figure 4 is a little confusing.

I’m sure you’ve checked this but I can’t see if the walls have colliders, or if they are on a layer that is included in the obstructionLayerMask. You may want to double-check those.

This won’t solve the problem, but I would also swap the distance check and raycast around. This would then only check for a wall if the enemy is closer than a previous ‘close’ enemy. Raycasts take longer than just comparing two values and there’s no need to check for a wall if the enemy’s not going to be the closer one, anyway.

I also think you may (and I could be grasping at straws here) have a slight problem in your raycast.

Physics.Raycast(transform.position, target.transform.position - transform.position.normalized, out hit, playerStateMachine.RangeOfVision, obstructionLayerMask)

Your direction is a little strange - at least to me. You are normalising the player position only. Try wrapping the direction in parenthesis and then normalising it

Physics.Raycast(transform.position, (target.transform.position - transform.position).normalized, out hit, playerStateMachine.RangeOfVision, obstructionLayerMask)

Edit
Thought about it a bit more and I do think the direction-normalisation is what’s causing the issue. It will completely change the direction the ray is being cast in.

Hi Bixarrio,

Thanks so much for taking the time to respond.

You’re absolutely right, the missing parentheses were causing some issues. I’ve now added those in so my raycast code looks like this:

    public bool SelectTargetClosestToPlayer()
    {
        if (targets.Count == 0) { return false; }

        Target closestTarget = null;
        float closestTargetDistance = Mathf.Infinity;

        foreach (Target target in targets)
        {
            Debug.DrawRay(transform.position, (target.transform.position - transform.position), Color.red, 1f);

            Vector3 TargetDistanceFromPlayer = target.transform.position - transform.position;
            RaycastHit hit;

            if (!Physics.Raycast(transform.position, (target.transform.position - transform.position), out hit, playerStateMachine.RangeOfVision, obstructionLayerMask) && TargetDistanceFromPlayer.sqrMagnitude < closestTargetDistance)
            {
                closestTarget = target;
                closestTargetDistance = TargetDistanceFromPlayer.sqrMagnitude;
            }
        }

        if (closestTarget == null) { return false; }

        CurrentTarget = closestTarget;
        cinemachineTargetGroup.AddMember(CurrentTarget.transform, 1.0f, 2.0f);

        return true;
    }

This is providing much more stable behaviour, and the player can now only target the enemy when the line of sight is left unbroken, including in that problem spot from Figure 3 above

Ex1

Unfortunately, it only ever seems to be targeting the first enemy in the list, even if other enemies are closer.

The RangeOfVision variable I’m using is the same for both the sphere collider and the raycast length, and the walls all have the correct meshes. Have checked all the Collider Layers and they’re correct as well.

So I guess I need to find out why the list of potential enemies isn’t being refreshed and reordered depending on distance. It must be something to do with not collecting the various distances before I raycast, as you’ve mentioned.

Confusingly, when I’m below the left-most enemy it does actually seem to work correctly, like so:

But then if I head around the back of them it won’t enter the TargetingState at all, as though it can’t find any targets.

Will keep digging and report back if I make any more headway

Thanks again for your help, much appreciated

Perhaps change this to

float targetDistanceFromPlayer = Vector3.Distance(transform.position, target.transform.position);

What you are doing is to create a new vector with a position equal to the distance between the two characters. That is, if a is at (10, 12) and b is at (13, 4) you create a vector that is positioned at (a - b) = (-3, 8). The sqrMagnitude of that would be the square distance from (0,0) to (-3,8). I guess it’s probably correct.

Hi Bixarrio,

Thanks again for your response. A float makes much more sense here than a Vector3, you’re right. Have incorporated your code.

Spent yesterday going round and round in circles with it, trying different things with varying degrees of success. Sometimes it would sort of work but then develop blind spots, sometimes it wouldn’t enter the TargetingState at all, despite there being a list of available targets in the Targeter.

I put a Debug.DrawRay() line into various parts of the code to see what was going on, and the rays seemed to be casting out at strange angles depending on where I was placing the raycasting code. It’s almost like the target positions weren’t update relative to the character or something? Although every forum I looked at seemed to use basically identical code to mine.

I’ve still not got to the bottom of what exactly was happening wih the Raycast to be honest, BUT I have got it working! I just replaced Physics.Raycast() with Physics.Linecast() and for some reason that seems to work completely as intended. My method now looks like this:

    public bool SelectClosestAvailableTarget()
    {
        if (targets.Count == 0) { return false; }

        Target closestTarget = null;
        float closestTargetDistance = Mathf.Infinity;

        foreach (Target target in targets)
        {
            float targetDistanceFromPlayer = Vector3.Distance(transform.position, target.transform.position);

            if (targetDistanceFromPlayer < closestTargetDistance)
            {
                if (!Physics.Linecast(transform.position, target.transform.position))
                {
                    closestTarget = target;
                    closestTargetDistance = targetDistanceFromPlayer;
                }
            }
        }

        if (closestTarget == null) { return false; }

        CurrentTarget = closestTarget;
        cinemachineTargetGroup.AddMember(CurrentTarget.transform, 1.0f, 2.0f);

        return true;
    }

I need to add a few more obstacles to my prototype and test it more thoroughly just to make absolutely sure, but it seems to be doing exactly what I want it to do, switching to the nearest enemy only if there isn’t an obstacle in the way.

You can even add a layermask and pass data out like you would with the Raycast, so I can always incorporate those later if I need something more complex.

Thanks for helping me work through it, appreciate your time!

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.

Privacy & Terms