Remove dependency on dfs-datastores

[rubanm]

This change copies over the VersionedStore code from dfs-datastores, and drops our dependency on dfs-datastores. This helps remove some dependencies for the upgrade path to Cascading 3.

https://github.com/nathanmarz/dfs-datastores/blob/master/dfs-datastores/src/main/java/com/backtype/hadoop/datastores/VersionedStore.java

Pending:
Test runs on some real jobs using VersionedKeyValSource.

Open question:

This change currently drops the PailSource.scala file altogether. Should we:

1. copy over the PailTap code from dfs-datastores to a scalding-pail module -- quite a few files to copy over and own in that case, to support one PailSource class in scalding.
2. drop PailSource altogether -- no current usages at Twitter, but likely being used externally. There's also an open PR Allow constructing a PailSource from a PailSpec #1424

2 would be ideal but we'll need to provide a good way for users to migrate away.

[johnynek]

+1 to this.

For Pail fans, they can import the code into their project or migrate to: https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/PartitionedTextLine.scala (since we assume there are few, if any).

[ianoc]

This looks pretty good to me, it would be great to 'scalaize' it, would be shorter and easier to read. But i think we could keep that for future clean up work as a noobie issue really

[rubanm]

Test runs done. Also fixed a bug in the store cleanup method (it was double-adding datasets with both version suffix and success files).

[johnynek]

👍 will wait for Ian to merge