We have $100,000 on him in whole life. yeah yeah, I know, I need to change it. But somehow, I'm scared to try

don't know why. Maybe because last time we went in to our Farm Bureau agent he not only convinced me to not drop the whole life, but up it as well (I canceled the extra the next day.)
We have $100,000 on him through my work and $500,000 on me through my work as well.
Should we have independent policies AND work policies? In case of job changes, etc?