Bottom line here is that this is about use of the land you own. You are under no obligation to let anyone use it and you have no obligation to take a political stand in order to say thank you but I have changed my mind about you using my land.
You have taken more steps than some to help this organization see a change is needed. In the end, you are not a member of the Scouts. You don't have a child in Scouts. You have shared your grievances with both the church that endorses the group and the leadership in the organization. They are the ones that must push change.
I think you have gone above what you needed to do. You could have just said no about land use without explanation. Talking to the guy may put you or your land in harm's way. No reason in the world that you have to have a direction confrontation with someone over an issue that needs to be solved by the organization that gave him a position of power.