progress.json does not exist

[jontxu70]

I'm trying to install cusf-standalone-predictor but I can't.

I've installed all python packages, pydap, pydap, numpy, simple-json, python-cjson
python-demjson, python-simplejson, python-turbojson , python-anyjson ...

Compiled OK pred file with cmake . && make

Changed config.php file

When I access to the index.php page, it starts but does not finish.

Generating py_log and scenario.ini :
preds/9206471c0b6b8a43a8bcb187bd41994844a2f1da/py_log
preds/9206471c0b6b8a43a8bcb187bd41994844a2f1da/scenario.ini

progress.json does not exist.

Using Debian

Linux 2.6.32-5-xen-amd64 x86_64 GNU/Linux
Using Squeeze

Any idea?

Thanks in advance

[jonsowman]

Hi,

progress.json should be created by predict.py when it is invoked by the PHP backend system. Firstly I would check permissions are OK (though they look to be).

Then su to the user under which the predictor runs and try running predict.py manually, providing appropriate command line arguments.

If this fails, please post the resulting traceback here.

Jon

[jontxu70]

I copied and modified the file py_log to run from the prompt as root:

sh predict/preds/1679da091e2b11beba156b02c66f32a55b31f4d4/py_log2

---

Traceback (most recent call last):
File "predict.py", line 20, in
statsd.init_statsd({'STATSD_BUCKET_PREFIX': 'habhub.predictor'})
AttributeError: 'module' object has no attribute 'init_statsd'

---

File content py_log2: (a line)

python predict.py --cd=/home/www/html/predict/ --fork --alarm --redirect=predict/preds/1679da091e2b11beba156b02c66f32a55b31f4d4/py_log
-v --latdelta=3 --londelta=3 -p1 -f5 -t 1348588980 --lat=52 --lon=0 1679da091e2b11beba156b02c66f32a55b31f4d4

Without arguments:

Traceback (most recent call last):
File "predict.py", line 20, in
statsd.init_statsd({'STATSD_BUCKET_PREFIX': 'habhub.predictor'})
AttributeError: 'module' object has no attribute 'init_statsd'

I have statsd installed and using Python 2.6.6

Any idea?

thanks

[jonsowman]

Nominate @danielrichman, since he implemented the stats stuff in the predictor.

Jon

[adamgreig]

You'll need to make sure it's the right version of statsd. Try this:

$ python
$ import statsd
$ help(statsd)

Scroll right to the bottom:

VERSION
    1.0.2

[danielrichman]

N.B.: This is the statsd client used in habitat/the predictor http://pypi.python.org/pypi/statsd-client (rather than 'statsd')

[jontxu70]

VERSION
1.0.0

How I can force an update to 1.0.2?

thanks

[adamgreig]

Try:

$ sudo pip install --upgrade statsd-client

[jontxu70]

pip - command not found

[adamgreig]

If you have easy_install, try:

$ sudo easy_install --upgrade statsd-client

If not, you could install pip (a python package manager):

$ sudo apt-get install python-pip
$ sudo pip install --upgrade statsd-client

I don't know what statsd you installed or how you installed it exactly but you might want to remove it first.

[adamgreig]

Actually one easy option is to just remove everything about statsd from predict.py: it's an entirely optional feature that will do absolutely nothing if you don't have a statsd server and graphite installed.

[jontxu70]

i've upgraded stats to 1.0.2 and I've remoded all about statsd.

Executing py_log2 again:

---

ERROR: Uncaught exception
Traceback (most recent call last):
File "predict.py", line 587, in
main()
File "predict.py", line 158, in main
detach_process(options.redirect)
File "predict.py", line 569, in detach_process
out_fd = os.open(redirect, os.O_WRONLY | os.O_APPEND)
OSError: [Errno 2] No such file or directory: 'predict/preds/07f7d85501b383816c56554714057ebde694964a/py_log'
Traceback (most recent call last):
File "predict.py", line 587, in
main()
File "predict.py", line 158, in main
detach_process(options.redirect)
File "predict.py", line 569, in detach_process
out_fd = os.open(redirect, os.O_WRONLY | os.O_APPEND)
OSError: [Errno 2] No such file or directory: 'predict/preds/07f7d85501b383816c56554714057ebde694964a/py_log'

[adamgreig]

Sounds like it could be a file permissions error again. Check the 'predict/preds' path exists and that PHP was able to create the folder for that UUID.

[jontxu70]

chmod -R 777 predict

result:

Traceback (most recent call last):
File "/home/www/html/predict/predict.py", line 587, in
main()
File "/home/www/html/predict/predict.py", line 158, in main
detach_process(options.redirect)
File "/home/www/html/predict/predict.py", line 569, in detach_process
out_fd = os.open(redirect, os.O_WRONLY | os.O_APPEND)
OSError: [Errno 2] No such file or directory: 'predict/preds/7963a693b4306a3c2e7886e971dbf05768dbd3ab/py_log'

[danielrichman]

Make sure your current working directory is the root of the git repository when you run that command. Alternatively, add --preds to tell it where to find predict/preds (i.e., the default is ./predict/preds)

Have you tried running a prediction from the web interface since fixing the stats stuff?

[jontxu70]

Running a prediction from the web:

Server says: downloaded 10% of GFS files
Server says: downloaded 10% of GFS files
Server says: downloaded 10% of GFS files
Server says: downloaded 10% of GFS files
Server says: downloaded 10% of GFS files
Server says: downloaded 10% of GFS files
There was an error in running the prediction: Unhandled exception: Traceback (most recent call last): File "/home/www/html/predict.py", line 587, in main() File "/home/www/html/predict.py", line 265, in main time_to_find + datetime.timedelta(hours=options.future)) File "/home/www/html/predict.py", line 373, in write_file assert len(selection.shape) == 3 AssertionError

[danielrichman]

We've found the NOAA servers to be a bit temperamental - it sounds like this might be a problem with the wind data you're getting. Try deleting /tmp/pydap-cache/ and running the prediction again - or try a different prediction (different launch site or time) to see if you still get the same error.

[jontxu70]

I get the same error with different predictions and deleting /tmp/pydap-cache/* files.

[jontxu70]

py_log info:

INFO: Looking for latest dataset which covers Mon Oct 1 18:35:00 2012
INFO: Found good dataset at http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20120927/gfs_12z.
INFO: Found appropriate dataset:
INFO: Start time: Thu Sep 27 12:00:00 2012 (POSIX 1348747200)
INFO: End time: Fri Oct 5 12:00:00 2012 (POSIX 1349438400)
INFO: Latitude: -90.0 -> 90.0
INFO: Longitude: 0.0 -> 359.0
INFO: Downloading data in window (lat, lon) = (52.0 +/- 3.0, 0.0 +/- 3.0).
INFO: Downloading from Mon Oct 1 18:00:00 2012 to Mon Oct 1 21:00:00 2012.
INFO: Downloading data for Mon Oct 1 18:00:00 2012.
INFO: Processing variable 'hgtprs' with shape (65, 26, 181, 360)...
INFO: Downloaded data has shape (1, 26, 7, 360)...
ERROR: Uncaught exception
Traceback (most recent call last):
File "/home/www/html/predict.py", line 587, in
main()
File "/home/www/html/predict.py", line 265, in main
time_to_find + datetime.timedelta(hours=options.future))
File "/home/www/html/predict.py", line 373, in write_file
assert len(selection.shape) == 3
AssertionError
Traceback (most recent call last):
File "/home/www/html/predict.py", line 587, in
main()
File "/home/www/html/predict.py", line 265, in main
time_to_find + datetime.timedelta(hours=options.future))
File "/home/www/html/predict.py", line 373, in write_file
assert len(selection.shape) == 3
AssertionError
INFO: Looking for latest dataset which covers Mon Oct 1 18:35:00 2012

[danielrichman]

Hi. Just had an idea

Please paste the output of

$ pip freeze

[jontxu70]

AddOns==0.6
BytecodeAssembler==0.5.1
DecoratorTools==1.8
Extremes==1.1.1
Genshi==0.6

FIXME: could not find svn URL in dependency_links for this package:

PEAK-Rules==0.5a1.dev-r2600
Paste==1.7.5.1
PasteDeploy==1.5.0
PasteScript==1.7.5
Pydap==3.1.RC1
SymbolType==1.0
TurboJson==1.2.1
anyjson==0.2.3
demjson==1.4
distribute==0.6.14
httplib2==0.7.6
numpy==1.6.2
prioritized-methods==0.2.1
python-cjson==1.0.5
simplejson==2.1.1
statsd==1.0.0
statsd-client==1.0.2
wsgiref==0.1.2

thanks

[danielrichman]

Okay.
Try running

pip install Pydap==3.0.1 numpy==1.5.1

(may require root, depending on how your stuff is set up; add sudo if required).

[jontxu70]

It wors better but flight_path.csv file is empty

0 sep 27 21:37 599a53a75305ae13fb3f32e8da239545c273cbf6/flight_path.csv

i can't see balloon trajectory

The server gave us uuid:
d9ff1be495877c53f6916c57e52fafe0488eadf2
Starting to poll for progress JSON
Server says: downloaded 10% of GFS files
Polling for progress JSON timed out
Increasing AJAX timeout from 500ms to 1000ms
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 70% of GFS files
Server says: downloaded 85% of GFS files
Server says: downloaded 85% of GFS files
Server says: downloaded 100% of GFS files
Server says: the predictor finished running.
Attempting to retrieve flight path from server
Server gave a prediction run timestamp of 1348778392
Server said it used the gfs20120927_12z GFS model
Got JSON response from server for flight path, parsing...
Flight data parsed, creating map plot...
Clearing previous map trace

thanks!

[jontxu70]

It is working fine now!!! :-)

Changed predict.py, line 276:

subprocess.call([pred_binary, '-i/var/www/cusf-standalone-predictor/gfs/'...

Changed to:
subprocess.call([pred_binary, '-i/home/www/html/gfs/', ...

A million thanks to all for your help.

[jontxu70]

I'm trying the other package. rjw57 / cusf-landing-prediction

I've installed all scripts and created all directories, i'm getting an error executing ./scripts/hourly-predictions-cronjob.sh

INFO:root:Scanning data directory /opt/landing-prediction-data/gfs/
Traceback (most recent call last):
File "/opt/cusf-landing-prediction/scripts/run-hourly-predictions.py", line 226, in
main()
File "/opt/cusf-landing-prediction/scripts/run-hourly-predictions.py", line 179, in main
(end_stamp, model) = get_latest_timestamp()
File "/opt/cusf-landing-prediction/scripts/run-hourly-predictions.py", line 172, in get_latest_timestamp
models.append({ "date": header[5], "time": header[6] })
IndexError: list index out of range

Any idea?

thanks in advance

[kkm2206]

I do all posted above but i get that error:

i am at a loss :-(

ERROR: Uncaught exception
Traceback (most recent call last):
File "./predict.py", line 587, in
main()
File "/usr/local/lib/python2.7/dist-packages/statsd.py", line 141, in f
return func(_args, *_kw)
File "./predict.py", line 158, in main
detach_process(options.redirect)
File "./predict.py", line 569, in detach_process
out_fd = os.open(redirect, os.O_WRONLY | os.O_APPEND)
OSError: [Errno 2] No such file or directory: 'predict/preds/f3cf2942341d69f8e5adde822162a1f737440368/py_log'
Traceback (most recent call last):
File "./predict.py", line 587, in
main()
File "/usr/local/lib/python2.7/dist-packages/statsd.py", line 141, in f
return func(_args, *_kw)
File "./predict.py", line 158, in main
detach_process(options.redirect)
File "./predict.py", line 569, in detach_process
out_fd = os.open(redirect, os.O_WRONLY | os.O_APPEND)
OSError: [Errno 2] No such file or directory: 'predict/preds/f3cf2942341d69f8e5adde822162a1f737440368/py_log'