I think you're misunderstanding how every point in space can be seen as a vector
if you have a point (5, 3, 6), you can represent it as a vector (5, 3, 6) from the origin (0, 0, 0)
a point on it's own doesn't have a direction, if you're claiming it's a vector it's often with respect to something, whatever or wherever that may be (either the origin or some other point way out somewhere)
and I do often see points in space as vectors, unlike what jetz says
it really depends on the context
but yeah, if you're subtracting vector B from vector A, just adding the inverse of B works well