In this exchange with Fox's Greta Van Susteren, Virginia's Attorney General - Ken Cuccinelli - argues just that. Early last year, one U.S. District Court actually ruled this way. Judge Roger Vinson first found the Individual mandate Unconstitutional and ruled that because there is no severability clause - which allows part of the law to go and part of it to stay - the entire law had to be thrown out.
This decision shouldn't rest on one person but because SCOTUS already has four left wing ideologues on the bench, including one - Elena Kagan - who helped craft the Obamacare monstrosity and refused to recuse herself, it probably will. If it's unconstitutional, it should be a 9-0 ruling but ideology trumps just about everything these days.
If this law is struck down in its entirety, the election of Scott Brown - which was thought to be rendered irrelevant when this bill passed - could actually prove to be one of the greatest Tea Party victories in Obama's first term. His election after the passing of Ted Kennedy took away the Democrats' filibuster-proof majority and they couldn't take the bill up for another vote to make it more ironclad. Instead, they went for the reconciliation process, which required a vote of 51 instead of 60.
In short, it's quite possible that if Kennedy had not expired, the Supreme Court could have a much more difficult time ruling Obamacare unconstitutional.
Perhaps this video might help explain why...