Literally, the edge of town is where the town ends and the country (or an adjoining town) begins: it is a line. Outskirts are the parts of town at or near the edge, they are an area. We often use 'the edge' to refer to the outskirts. The terms can be used quite vaguely - you might refer to small settlements just outside the town as 'outskirts', this is quite OK.
By the way, you might live at or near the edge, you live in the outskirts.