If it was soaring, it does so with it's wings in a bit of a V. Other birds of prey have their wings out flat.
Often it's a case of knowing which birds will be found in which habitat. Is it marshland? Probably a Marsh Harrier. Is it heathland? Probably a Hen Harrier. Also whether they are common in the part of the country. E.G. Puffins are extremely unlikely in Leicestershire.
That said birds have wings and can end up where they don't habitually go, so it's not a hard and fast rule.