How to install, use and upgrade dataserv-client for Storj Test B

Development 3rd Party
1

A short how-to for those interested in participating in Test B of the Storj Project.

###What does Test B do?

  • See http://blog.storj.io/post/118706366258/driveshare-test-group-b-plan for the official explanation
  • Simply speaking Test B tests the system’s ability to work with a large (up to 1PB) amount of client data. Because it would take a huge amount (well, at least 1PB) of network traffic to create 1PB of data the way it’s going to be done in production, the current version of client-side software creates random data (the build command). In real life Storj won’t work like that (data will be uploaded by “consumers”), so this part is deliberately simplified. Server receives client status which includes the address and number of shards (128MB large binary files) the client has.

###Notes

  • At the moment (v1.3.0, see a list of available releases here) the test app doesn’t earn any awards - those will likely be available in v1.4.0 when authentication is added to the app, so until such time you probably don’t want to dedicate a large amount of space to the test. 128M (1 “shard” or the height of 1) is enough.
  • Both now and later, do not “farm” (or “mine”) from an address that does not belong to you (an example of such address is your Deposit address at a crypto-exchange). You want to have a bitcoin address (with 10K+ SJCX) that belongs to you. If you have a Countewallet wallet, pick one address and buy SJCX (http://coinmarketcap.com/assets/storjcoin-x/) from it using a vending machine (or buy SJCX at an exchange and withdraw it to your Counterwallet address).

###Windows

  • Install Python 3.4 (download the latest & greatest from python.org)
  • Open the CLI and install dataserv-client (c:\python34\scripts\pip.exe install dataserv-client)
  • Now you should be able to run the client like this:
c:\python34\python.exe c:\python34\Scripts\dataserv-client --help
usage: dataserv-client [-h] [--address ADDRESS] [--url URL]
                       [--max_size MAX_SIZE] [--store_path STORE_PATH]
                       [--debug]
                       <command> ...

Dataserve client command-line interface.

optional arguments:
  -h, --help            show this help message and exit
  --address ADDRESS     Required bitcoin address.
  --url URL             Url of the farmer (default: http://104.236.104.117).
  --max_size MAX_SIZE   Maximum data size in bytes. (default: 1073741824).
  --store_path STORE_PATH
                        Storage path. (default:
                        C:\Users\IBM_ADMIN\.storj\store).
  --debug               Show debug information.

commands:
  <command>
    version             Print version number.
    register            Register a bitcoin address with farmer.
    ping                Ping farmer with given address.
    poll                Continuously ping farmer with given address.
    build               Fill the farmer with data up to their max.

###Linux (Ubuntu 14.04)

  • You should have all the packages required, but in case pip3 isn’t there, install it with sudo install python3-pip python3-dev)
  • Install dataserv-client: sudo pip3 install dataserv-client
  • Run the app: dataserv-client --help
  • These commands should work on all Ubuntu/Debian variants and architectures

###How to configure and run

  • Register the bitcoin address using dataserv-client --address YOUR_ADDRESS register (this is required just once for the address; if you need to repeat the procedure on another system or directory later on, skip the registration step if you’re using the same address)
  • Create a directory which will hold your shards (it has to have enough space), such as e:\storj (Win) or /storj on Linux
  • Build the capacity required for test: dataserv-client --address YOUR_ADDRESS --max_size 134217728 --store_path e:\storj --debug build (the figure is in bytes, v1.4.0 should support user friendly values such as 128M)
  • Once that is done, you can start notifying the server with your status: dataserv-client --address YOUR_ADDRESSS --debug poll
  • In v1.3.0, dataserv-client starts polling immediately after you issue the build command, but stops after the command completes. This behavior may change in v1.4.0, but until then build must be followed by poll.

###How to update dataserv-client

The process is the same as installation just instead of install (see above), type update, e.g. on Linux
run sudo pip3 update dataserv-client
Then verify with:
dataserv-client version (Lin)
On Windows, the commands are the same, but issue them by prepending c:\python34\python.exe c:\python34\scripts\ to dataserv-client, as explained in the Windows section above.

###Monitoring
At the moment, the current status is visible at http://104.236.104.117/api/online. Example:
1EgxmPLMvi4RB5nh5HYxDaWxFFpLTwHmjf | Last Seen: 10 second(s) | Height: 4023

Height 4023 means this address is “farming” about 500 GB (4023 * 128 MB) for the network.

Again, keep in mind that the current client, v1.3.0, allows “cheating” because anyone can “register” another person’s SJCX address as well as report a fake amount of shards. But that isn’t so relevant because as mentioned above in real life data will be uploaded by real users’s computers to real users storage, whereas for the purpose of this test users generate data by themselves.

The only interesting consequence of this temporary ability to cheat in Test B is that the award fund which was supposed to be divided among test participants who reached the milestones of 100 and 1024 TB (0.1 and 1 PB, respectively) was not distributed yet. It is currently expected that “snapshots” of the participants’ storage contribution will be recorded after v1.4.0 is released. On the one hand, this is good for those who start late, but on the other those who have 1-2 TB probably won’t see meaningful rewards because there’s already 350 TB of (largely legit) capacity on the network.

In any case, you should be able to see your amount of shards as soon as you correctly execute the register, build and poll command and receive some token amount of SJCX once v1.4.0 is available and you upgrade to it. This how-to won’t be updated with the details about that - please follow Storj blog and community updates because they’re working very fast.

###Help!
If you need help with Storj software (such as dataserv-client), join the Storj Community slack at http://storjcommunity.slack.com. If you need help with Counterwallet, you’ll probably find the answer on this forum. The most common issue related to Counterparty is about wallet downtime which is sometimes caused by Bitcoin blockchain reorganizations (short answer: retry in 90 mins; longer answer: see Support section).

2

Version 2.x is out and it attempts to solve a Test-specific problem of cheaters (as a reminder, because test data is self-generated to save on network bandwidth, it was possible to falsely report the amount of data stored) and the tempo of reward payout was slowed down till v2 comes out. It also makes it easier to continue interrupted builds.

How does v2 work compared to v1?

It autogenerates an encrypted local wallet which is used to sign the payout address (which is probably the one people used during crowdsourcing) and stores that in the user’s .storj directory (e.g. /home/user/.storj).

To install:

  • uninstall previous version if you had 1.x: sudo pip3 uninstall dataserv-client
  • install the new version (either binary (especially on Windows), or use pip if you know how): sudo pip3 install dataserv-client
  • Both Linux and Windows users may need to use elevated prompt (or sudo) to upgrade, depending on what user you are and so on

After you’re done:

c:\python34\python.exe c:\python34\scripts\dataserv-client version
2.0.3

To get started, one has to set his payout address (again, this is probably the address from which you joined crowdsourcing, wtih 10K (or more) of unspent SJCX.

Set this address as payout address to make sure it’s stored in your config file:

$ dataserv-client config --set_payout_address 1PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP
{
  "master_secret": "ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss==",
  "version": "2.0.3",
  "wallet": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "payout_address": "1PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP"
}

You may want to backup this wallet info (or the entire configuration file) to a secure location!

By default dataserv-cleint config file is stored in your home directory (like /home/user/.storj) and if you want to work with multiple files, you can specify one with dataserv-client --config_path=/home/user/.storj/config1.json .... build. This might make it possible to run two instances of dataserv-client on single system by referring to two different configuration files. But it’s not officially supported at the moment so your mileage may vary. On Linux and Windows you could place your config files anywhere, like c:\config1.json and c:\config2.json if that makes it easier to edit them. But in that case remember to always use --config_path= in all dataserv-client commands, otherwise your dataserv-client commands may recreate and/or update config file in the default location!

Next, register (yes, again for v2, but note the URL is different, because this is server v2) :

$ dataserv-client --url http://status.driveshare.org:5000 register
Address 1LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL registered on 
http://status.driveshare.org:5000 with payout address 
1PPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPP.

NOTE: The registered address is the address you’ll look up on the status page (1L....), not your payout address!

Now decide your storage allocation (thankfully, you don’t have to use bytes any more, 1T = 1TB)…

$ dataserv-client --max_size 1T --store_path /storj-store --debug --url http://status.driveshare.org:5000 build
Resuming from height 100

As you can see from the example above, the other good news is that resuming is now instant.

Once you’re done building you can stop the build command and run dataserv-client poll.

c:\python34\python.exe c:\python34\scripts\dataserv-client poll

If you are using a config file in custom location, you’d have to inform the client where it is: dataserv-client --config_path=c:\config2.json poll. Use dataserv-client poll --help to display other options.
Do not run poll prematurely (while building is still going on) since that is not necessary (the build action performs polling as well).

To check your status, go to the status page and look up your “lookup” (not crowdsourcing/payout) address:
http://status.driveshare.org:5000/api/online

Status is updated every 25 “shards” (25 * 128 MB) so if your client builds at a slower rate, you may need to wait for a few minutes before you can see the latest status.

###Support

3

this will help as i will be setting this up tonight at some point. thanks!

4

This is all very interesting but how does a non-programmer get involved? I have several TB I could use for this to earn some coin

5

Obviously that’s not something we can’t know, but you can join the Storj Slack - http://slack.storj.io/ - to find out.
I don’t think they’re exactly close to making earnings flow, but maybe there are testing bonuses and such.

6

I am unable to install dataserv-client. I get the following error:

error: [Errno 2] No such file or directory: 'c:\\python351\\python35.zip\\lib2to3\\Grammar.txt'

Command “python setup.py egg_info” failed with error code 1 in C:\Users\eazyi\AppData\Local\Temp\pip-build-l27q0d0h\partialhash\

7

I haven’t used a recent version, maybe you need to pip install 2to3, but I would suggest you check out the Storj Slack, I’m sure they have the latest instructions there.

These instructions are for an early beta release from few months ago - a lot has changed from then.