Preheat oven to 350 degrees. Butter or spray a deep 9x9-inch baking dish, fill with bread, set aside.
In a large mixing bowl, whisk together the pumpkin puree, brown sugar, granulated sugar, eggs, vanilla, pumpkin pie spice, cinnamon, salt, and nutmeg. Add the milk and cream, whisk to combine.
Pour the mixture over the bread, making sure all of the bread is coated. The baking dish will be very full, but this will create a tall, thick slice of bread pudding.
Place in the oven and bake for 45-50 minutes or until fully set. Tent with foil during the last 15 minutes if the top browns too quickly.
Remove from the oven and allow to cool for 20 minutes before slicing and serving.
Serve warm, garnish with candied pecans, caramel sauce, whipped cream or vanilla ice cream, if desired.