Preheat oven to 350 degrees. Spray or line a 9x5 inch loaf pan, set aside.
In a small mixing bowl combine all of the ingredients for the cream cheese swirl. Using a hand mixer blend until smooth, set aside.
In the bowl of a stand mixer, cream the butter, sugar, cream cheese, vanilla and lemon zest until light and fluffy, about 2-3 minutes. With mixing speed on low, add eggs one at a time until incorporated.
In separate mixing bowl, whisk together the flour, baking powder, baking soda and salt. With mixing speed on low, gradually alternate adding the dry ingredients and milk, mixing just until no flour pockets remain. Remove the bowl from the stand and fold in the fruit.
Pour half of the batter for the bread into the prepared loaf pan. Then pour the cream cheese "swirl" on top. You can either leave it as-is or carefully swirl it with a knife. Pour the remaining batter over top and sprinkle with coarse sugar if desired.
Place in the oven and bake for 50 to 60 minutes or until the center of the loaf is set. You may need to tent the loaf with foil if the top is getting too brown (around the 40-50 minutes). Remove from the oven, allow to cool in the pan for 30 minutes, then turn onto a wire rack to cool completely.