I find it depends on the person - some are able to be fine with a few cups of coffee daily, others simply not.
I've never been a coffee or tea drinker - I've usually only found my caffeine in Diet Coke ... When I started doing lots of research on caffeine and diabetics, I too decided to try dropping it.
And yes, my numbers improved slightly when I switched to caffeine-free Diet Coke. But it took close to a week before it was a noticeable trend for me. So don't expect better numbers the day after quitting... =)
I think it's something we need to test individually - give up or dramatically-reduce caffeine intake for a couple weeks... if there's a noticeable improvement trend in your numbers - you'll know what's best for you.