In a medium saucepan over medium heat, combine peaches, granulated sugar, brown sugar, cinnamon, nutmeg, cornstarch, lemon juice, and vanilla extract.
Stir continuously until the mixture thickens and becomes syrupy, about 5 to 7 minutes.
Remove the filling from heat and let it cool slightly.
Lay an egg roll wrapper on a clean surface in a diamond shape.
Place 2 tablespoons of the peach filling in the center of the wrapper.
Fold the bottom corner up over the filling, fold in the sides, then roll tightly.
Seal the edge with water to close the wrapper.
Repeat the process with remaining wrappers and filling.
Heat about 2 inches of vegetable oil in a deep skillet to 350°F.
Fry the egg rolls in batches, turning occasionally, until golden brown and crispy, 2 to 3 minutes per side.
Transfer egg rolls to a paper towel-lined plate to drain excess oil.
Dust with powdered sugar if desired and serve warm with vanilla ice cream.