Preheat your oven to 350°F (175°C) and line a muffin tin with paper liners.
In a large mixing bowl, whisk together the flour, granulated sugar, baking powder, salt, and ground cinnamon.
In another bowl, combine the melted butter, milk, eggs, and vanilla extract. Mix well until fully combined.
Pour the wet ingredients into the dry ingredients and stir until just combined. Do not overmix; a few lumps are okay.
In a small bowl, mix together the brown sugar, ground cinnamon, and softened butter to create the cinnamon filling.
Spoon half of the muffin batter into the prepared muffin cups, filling each about halfway. Add a teaspoon of the cinnamon filling on top of the batter in each cup, then cover with the remaining muffin batter.
Bake in the preheated oven for 18-20 minutes, or until a toothpick inserted into the center comes out clean.
While the muffins are baking, prepare the glaze by whisking together the powdered sugar, milk, and vanilla extract in a small bowl until smooth.
Once the muffins are done, let them cool in the pan for 5 minutes before transferring them to a wire rack. Drizzle the glaze over the warm muffins before serving.