Worst case scenario: I have a friend who had an addition on the back of his house he felt was too small. He tore it off, and built all the way to the back property line, including incorporating the back alley easement behind the house. It was a really stupid thing to do, because not only did L & I make him tear down the new addition, but he couldn't even rebuild what he had torn down. To this day he has never rebuilt anything there, and now he has a nice, and sort of largish, backyard.