Make the dough: In the bowl of a stand mixer fitted with a paddle attachment, add the warm milk and water; stir in the yeast and let sit for 5 minutes.
Add the sugar, egg yolk, vanilla extract and salt to the yeast mixture; mix on low speed to combine ingredients.
Add 3 cups of the flour and continue to mix on low until flour is incorporated. Add pieces of softened butter, one tablespoon at a time, until it becomes incorporated. At this point switch to the dough hook attachment. Mix on medium-low speed while adding in enough of the remaining flour, about 1/2 cup at at time, to get a soft and slightly tacky dough. Knead until the dough cleans the bottom of the bowl and forms a nice soft ball, about 8-10 minutes.
Spray a clean bowl with non-stick spray. Place dough into the bowl and spray top of dough with non-stick spray. Covered bowl with a damp towel, plastic wrap or shower cap. Let dough rise in a warm place until double in size, about 60-75 minutes.
Make the filling: Add egg yolk and vanilla to a larger bowl; whisk together. Add brown sugar and apple pie spice; stir to combine. Add caramel bits and apple pieces; stir until well coated. Set aside.
Prepare pans: Coat two 8x4-inch loaf pans with nonstick spray or butter.
Form loaves: Turn dough out onto a lightly floured work surface. Divide the dough into 2 equal pieces; set one piece aside. cover with damp towel. Roll dough out into a 10x15-inch rectangle. Divide the filling in half and spread half evenly over the rolled out dough, leaving a 1-inch border on all sides. From the short side, roll the dough up tightly (like a cinnamon roll) into an even shaped log, pinching seam to seal. Cut the log down the center, long ways. With the cut sides facing upwards, gently twist the 2 pieces together to create a twisted loaf, being careful to keep the filling from falling out. Place loaf into prepared pans with the cut sides facing upwards; cover with plastic wrap. Repeat shaping with other dough piece.
Let loaves rise until slightly domed over top of pan, about 45-60 minutes.
Preheat oven to 350°F.
Bake for 55-65 minutes or until an internal temperature of 210°F. (If you are using a dark or coated metal pan, lower temperature to 325°F for the same duration.)
Turn out the loaves onto a cooling rack; cool for at least an hour before glazing.
Prepare vanilla bean glaze: In a bowl, whisk all ingredients together until well combined. Drizzle half of the glaze over each cooled loaf.