In a medium saucepan combine brown sugar, sugar, condensed milk and butter. Heat to 250 to 265 degrees F (121 to 129 degrees C), or until a small amount of syrup dropped into cold water forms a rigid ball.
Pour into an 8x8 inch pan. Cool until firm and cut into squares.