In a bowl, cream butter and sugar. Add eggs; mix well. Add sweet potatoes, milk and orange peel; mix well. Combine flour, allspice and nutmeg; add to creamed mixture. Mix just until combined. Pour into greased muffin tins. Bake 350 degrees for 20 minutes or until bread tests done. Cool in pan before removing to a wire rack.
In bowl combine spread ingredients; beat until smooth. Serve with bread.