Indeed. That is the major hurdle in space faring.
There is also another major concern that most scientists who study these things tend to forget...
...The substances, minerals or objects from some exoplanets might not be stable in Earths environment and we have no idea how they would react to exposure to our Atmosphere.
For all we know we could have Cats Cradle/Ice-Nine type situation on our hands. Or worse yet, the substance might decide to act like Anti-Matter and suddenly combust/detonate or burn off our planets atmosphere.
These challenges are why we are a LONG ways off from true interstellar (or god forbid, Intergalactic) travel.